git.kernel.org burns more CPU on scrapers than on clones

Fourteen CPU cores, spread across five geo distributed nodes, doing nothing but turning git commits into HTML pages for automated crawlers. That is the figure given by Konstantin Ryabitsev, who runs the kernel.org infrastructure, in a piece titled Creepy crawlies that Simon Willison picked up on 7 September on his weblog.

His own summary is even more uncomfortable:

> We spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones.

We are talking about git.kernel.org, the official repository of the Linux kernel. Not a personal blog with two visits a month.

Why a git server is the worst possible place for this

The crawlable surface of a large repository is, for practical purposes, infinite. Every commit has its page, every commit has its diff, every file has a view for each revision in which it changed, and on top of that come ranges, branch comparisons and blame views. The URL space grows combinatorially while the genuinely useful content does not grow at the same rate.

Each of those pages is also expensive. Rendering a diff means reading objects from the repository and computing the difference on the spot. And since almost none of them is requested twice, caching barely helps: the crawler visits a URL, drops it and moves to the next one. A static site absorbs that traffic without noticing; a git forge pays for it in CPU.

Not a kernel only problem

Willison notes in his entry that the issue worries him because of Datasette, his tool for publishing databases as browsable websites, which also generates a huge number of crawlable pages. The pattern repeats in anything that derives many URLs from a database: forges, wikis, catalogues, internal search, document archives. Willison himself found the piece via Hacker News.

The point is not that crawlers exist. They always have, and there is a reasonable tacit deal with them: you index me, I get visitors. What has changed is the volume and the asymmetry. The cost of generating the page falls on whoever serves it; the benefit goes to whoever ingests it.

What can be done without breaking the open web

Three lines of action that do work:

1. Make the expensive view cheaper. Precomputing diffs, caching the most hammered routes aggressively or serving a reduced version to anonymous traffic cuts the bill without closing anything off.
2. Offer the bulk up front. If someone wants the full history, a clone, a dump or a data endpoint is cheaper for everyone than two hundred thousand HTML requests. Publishing the cheap route and signposting it well diverts traffic away from the expensive one.
3. Add friction where it belongs. Rate limiting by ASN and putting a proof of work challenge in front of costly routes, with options such as Anubis that several free software projects have adopted, filters out the bulk of the abuse without asking humans to register.

And if you build agents, this concerns you too

This is where the story lands squarely on anyone working with Claude. An agent browsing third party sites is, technically, one more crawler. The hygiene rules are the same ones written down twenty years ago: identify the user agent, respect robots.txt, limit concurrency, cache what you have already seen and prefer the API or the feed over the HTML page when they exist.

On the publisher side there is a useful reading too. Exposing an MCP server or a documented endpoint over your own data is cheaper than being scraped blindly. A client that can fetch what it needs in one structured call stops walking ten thousand pages to find it.

There is no crisis of the web here, but there is a bill someone is quietly paying, and much of it lands on public infrastructure kept alive by donations. It seems reasonable to us that whoever builds on other people's data should carry the share of the cost they generate, starting with the simplest part: asking for less, and asking better.