git.kernel.org's Scraper Tax: More Cycles Than All Legitimate Traffic Combined
Konstantin Ryabitsev reports git.kernel.org spends more CPU rendering pages for abusive crawlers than on all legitimate access combined, including every git clone.
What it is
A writeup by Konstantin Ryabitsev, maintainer of git.kernel.org, the official Linux kernel Git host, describing the current scale of automated crawler traffic hitting the site.
What it does
It quantifies the load: across five geo-distributed nodes, 14 CPU cores run continuously just rendering commit pages for scrapers, exceeding the CPU spent on all other legitimate access types combined, including actual git clone operations.
Why it matters
Any team running a public git host, docs site, or code browser should expect scraper traffic to dominate infrastructure cost long before it shows up as a security incident, which changes the calculus on rate limiting and bot defenses for anything AI-training-adjacent.
How to use it
If you operate a public code-hosting or documentation site, budget capacity for crawler load as a first-class cost, not an edge case, and consider the mitigations Ryabitsev describes at the source post before scrapers force a redesign.