Claude and Claude Code Do Not Crawl the Web the Same Way

Anthropic does not run one crawler, it runs several, and each has a different job: ClaudeBot collects content, Claude-User fetches pages when someone asks for them inside a conversation, and Claude-SearchBot feeds the search index. If your server logs group all of that under a single Claude label, you are mixing behaviours that differ in frequency, volume and intent.

Search Engine Journal published an analysis with data on those differences on Monday, and it lands on an uncomfortable conclusion for anyone running a site: conversational Claude and Claude Code do not reach the web by the same route, so rules written with one in mind do not describe what the other does.

Two clients, two routes

The difference starts with the goal. Chat Claude searches in order to answer: it goes through a search layer, receives results and then fetches the pages it needs. Claude Code barely searches at all; it goes straight for a specific URL because someone pasted a documentation link or because the agent found the reference in the code.

The fetching is not the same either. The WebFetch tool in Claude Code converts the page to markdown, caches the result for fifteen minutes per URL, upgrades HTTP to HTTPS, and does not follow cross host redirects: it returns them to the model to decide. It fails on authenticated URLs. Small details that explain quite a few oddities seen from the server side.

Volume behaves differently too. The fifteen minute cache limits immediate repeats, but a long development session against the same documentation leaves a burst pattern against technical paths that looks nothing like the isolated visit generated by a conversational search on a product or pricing page.

Why it matters if you run a site

First, robots.txt. A generic rule against Claude does not distinguish between training, indexing and user requested retrieval, and those three things have different commercial consequences. Blocking training crawls is a legitimate decision; also blocking on demand retrieval means disappearing from answers when someone asks about your product by name.

Second, the WAF. Many bot management rules from Cloudflare and other providers act on user agent reputation, not on the policy you wrote. It is common to find sites that believe they allow access and return 403 to half the agents.

Third, rendering. If markdown conversion is the first step, everything painted client side with JavaScript does not exist. A site with its content behind hydration can look perfect in a browser and return an almost empty page to an agent. This is not new in SEO, but with agents it hurts faster because there is no second rendering pass.

What we would do about it

1. Split the user agents in log analysis. ClaudeBot, Claude-User and Claude-SearchBot in separate columns, not in a single bucket.
2. Decide per agent rather than as a block, and write it into robots.txt with explicit per user agent rules.
3. Check the real response with curl while changing the user agent, before assuming the configuration works.
4. Serve critical content from the server. If it depends on hydration, for an agent it is not there.
5. Watch redirect chains on canonical URLs. Every cross host hop costs an agent turn and some agents will not take it.

Who this is useful for

For SEO and GEO teams trying to measure presence in AI answers while still using one bucket for all Anthropic traffic. And for anyone building agents: understanding how Claude Code fetches explains why one set of docs reads perfectly and another comes back empty.

The underlying point is that AI traffic is no longer a useful category. Inside it there is training, indexing, live answering and development agents, and each deserves its own policy. This looks like a good moment to review robots.txt carefully, before a default plugin setting ends up making the decision for you.