crawl-census-client

# crawl-census-client

**Ask before you fetch.** A drop-in client that stops your crawler spending requests on doors
that are shut, and stops it routing around content someone is trying to sell.

Reading robots.txt answers one question and hides two others. Measured across **23,482 domains**
by [Crawl Census](https://crawlcensus.com):

- **2,874 domains permit AI agents in robots.txt and then refuse those same agents at the
  network edge.** A parser sees permission; the fetch returns 403. You pay for the round trip
  and get nothing.
- **208 domains answer an AI user agent with `HTTP 402 Payment Required`.** That is a price,
  not a refusal. Treating it as a block walks away from content the operator wants to sell you.
  Retrying around it takes something they are charging for.

No dependencies. No key required.

## MCP server

The same measurement is exposed as a remote MCP server, so an agent can ask before it fetches
rather than after it fails. Listed in the
[official MCP registry](https://registry.modelcontextprotocol.io) as
`io.github.taylorsmithgg/crawl-census`.

```json
{ "mcpServers": { "crawl-census": { "url": "https://crawlcensus.com/mcp" } } }
```

| Tool | Answers |
|---|---|
| `crawl_preflight` | will these domains serve my agent, refuse it, or charge it? |
| `agent_profile` | what does this census publish about my crawler, and how do I correct it? |
| `census_facts` | the headline findings as dated records with denominators and citation lines |
| `site_report` | the stored audit for one domain |
| `scan_site` | measure a domain now |
| `census_stats` | corpus-level totals |

No authentication for read tools. Streamable HTTP.

## Install

```bash
npm i github:taylorsmithgg/crawl-census-client
pip install git+https://github.com/taylorsmithgg/crawl-census-client
```

## Use

```js
import { politeFetch } from "crawl-census-client";

const r = await politeFetch("https://example.com/", { agent: "gptbot" });
if (r.skipped) console.log(r.verdict, r.reason);   // disallow | refuse | pay
else           process(await r.response.text());
```

```python
from crawl_census import polite_fetch

r = polite_fetch("https://example.com/", agent="gptbot")
if r.skipped:
    print(r.verdict, r.reason)
else:
    process(r.body)
```

Skipping is returned, not raised. It is the normal outcome for a large share of the web, and a
crawl loop should be able to count skips without a try/except around every URL.

## Split a queue before crawling it

One call per 1,000 domains instead of one per host:

```js
const { crawl, skip, pay, unknown } = await partition(urls, { agent: "gptbot" });
```

```python
p = partition(urls, agent="gptbot")
p.crawl, p.skip, p.pay, p.unknown
```

## Or just take the file

For a fetcher that only needs a deny list in memory, skip the per-domain calls entirely:

```bash
curl https://crawlcensus.com/agents/gptbot/blocklist.txt   # one domain per line, commented header
```

```js
const sync = await syncBlocklist("gptbot");   // full list once
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // then deltas only, a few hundred bytes
```

```python
sync = BlocklistSync("gptbot")
if host in sync: skip()
sync.refresh()          # {'added': 3, 'removed': 1, 'size': 3310, 'cursor': ...}
```

The delta feed is `https://crawlcensus.com/agents/<agent>/changes.json?since=<unix>` and each
response carries `next_since`, so a long-running crawler stays current on a few hundred bytes
an hour instead of re-downloading the list.

That file covers **robots.txt only**. Edge refusal and HTTP 402 are per-request behaviours and
still need `preflight` or `politeFetch`.

## What a crawl costs the census

Measured, not asserted. Twenty hosts fetched concurrently used to cost twenty preflight calls
carrying one domain each; the same host requested three times at once cost three, because the
cache only helps after the first lookup resolves. The anonymous allowance is 240 calls an hour,
so a crawler hit its ceiling at 240 hosts when one call covers twenty-five.

`politeFetch` now shares work automatically: lookups issued in the same tick leave as one
batched call, and concurrent lookups for the same host await a single request.

| pattern | before | now |
|---|---|---|
| 20 hosts, concurrent | 20 calls | 1 call of 20 |
| 1 host, 3 URLs, concurrent | 3 calls | 1 call |
| 60 hosts, concurrent | 60 calls | 3 calls (25 / 25 / 10) |
| `partition` then fetch | 2 calls | 1 call |

`batchSize` defaults to 25, the per-call cap without a key. Raise it with a Pro or Data key.
`batchWaitMs` widens the coalescing window for concurrency that arrives in waves rather than
all at once; the default of zero flushes on the next tick.

## Paying, when an origin quotes a price

A `pay` verdict carries the amount when the origin named one:

```js
const r = await politeFetch(url, { agent: "claudebot" });
if (r.verdict === "pay") console.log(r.price);   // "USD 0.5", or null if none was quoted
```

Two things worth knowing. Most origins answering HTTP 402 name no amount at all, so `price`
is usually null and the arrangement has to be made out of band. And pricing is per crawler:
across the measured corpus, 78 of 213 charging origins charge some agents and serve others
free, so ask with your own token rather than assuming a domain on the list will charge you.

## Two kinds of unknown

`partition` splits a work queue into `crawl`, `pay`, `skip`, `unknown` and `undecidable`.

The last two look alike and are not. `unknown` means the census has not measured that domain
yet: submit it and the next pass gets a real verdict. `undecidable` means the site's robots.txt
disallows CrawlCensusBot, so this census will never measure it — retrying is guaranteed waste,
and a loop that resubmits its unknowns each pass would resubmit those forever. The server marks
the difference with a `measurable` boolean; read that, never the `reason` prose.

```js
const p = await partition(urls, { agent: "gptbot" });
await Promise.all(p.crawl.map(politeFetchOne));
if (p.unmeasured.length) await submitUnmeasured(p, { agent: "gptbot" });
// p.undecidable: read their robots.txt yourself. Asking us again cannot help.
```

Submission is a separate call on purpose. A library that quietly POSTs during what reads as a
lookup is a bad citizen, and you should choose when your queue positions are spent.

## Skipping everything that will not serve you

A deny list is the smaller half. Measured against the live census, a crawler that skips only
robots disallows still spends around 2,900 requests a pass on domains that permit it in
robots.txt and refuse it at the edge, or that answer HTTP 402 — for PerplexityBot that set is
larger than its deny list. Those fetches return nothing and cost a round trip each.

```js
const skip = await syncSkipList("gptbot");
if (skip.has(host)) continue;        // disallowed, refused at the edge, or priced
skip.why(host);                      // "disallow" | "pay" | "refuse" | null
setInterval(() => skip.refresh(), 3600_000);
```

| agent | deny list | also skippable | total |
|---|---|---|---|
| GPTBot | 3,542 | 2,944 | 6,486 |
| ClaudeBot | 3,169 | 3,194 | 6,363 |
| PerplexityBot | 1,128 | 3,658 | 4,786 |

The three sets are kept apart internally, so a change moves the one it belongs to. `why()`
follows the same precedence as preflight: a disallow outranks a price, because a price is not
permission.

## Keeping a deny list current

`syncBlocklist` / `BlocklistSync` download the list once, then apply only what changed.

The list is served with the exact position in the change feed it was built at, in an
`x-cursor` header and a `# cursor:` comment. The clients read it and resume from there, so
there is no gap between the snapshot and the first poll, and no reliance on your clock being
in step with the server's. Polling by second cannot express a position inside a second, and a
crawl batch writes dozens of events into one, so a second-granularity resume can drop the
remainder of it: measured live, resuming after the first of three same-second changes
recovered both siblings by cursor and neither by second.

```js
const sync = await syncBlocklist("gptbot");   // cursor comes from the list itself
if (sync.blocked.has(host)) skip();
setInterval(() => sync.refresh(), 3600_000);  // a few hundred bytes per poll
```

`refresh()` applies only robots transitions to the list, because that is what the list is made
of. Edge refusals, new prices and llms.txt changes come back in `other` for you to act on
separately — an earlier version deleted those domains from the deny list, so a crawler resumed
fetching exactly what had just started refusing it.

## Verdicts

The authoritative definition of each verdict — what it means, what it obliges a crawler to do,
and whether asking again could change it — is published as data at
[`/api/v1/verdicts`](https://crawlcensus.com/api/v1/verdicts). The list below is a summary; if
the two ever disagree, the endpoint is right and this file is stale.

`politeFetch` skips `disallow`, `refuse` and `pay` by default, which is the endpoint's derived
`do_not_fetch` set. The copy here is deliberate — a crawl loop should not need a network call to
decide — and a test compares the two so it cannot drift unnoticed.

| Verdict | Meaning | Default behaviour |
|---|---|---|
| `allow` | robots.txt permits this agent, and a live request carrying its user agent was served | fetch |
| `disallow` | robots.txt forbids this agent at the site root | skip |
| `refuse` | robots.txt permits it; the edge refused it anyway. The allowance is not real | skip |
| `pay` | the origin answered HTTP 402. It will serve this agent on commercial terms | skip |
| `unknown` | not measured recently enough to answer | fetch |

`onPay: "fetch"` (`on_pay="fetch"`) overrides the paywall default. It is an explicit opt-in and
is recorded on the result as `paidRouteOverridden` so it shows up in your logs.

## It degrades, it does not fail

If the census is unrea