Claude Code's creator says token burn is the wrong way to measure AI success

Token burn has become the AI industry's favorite number: providers boasting about trillions of tokens served, companies measuring internal adoption by their teams' monthly consumption, and dashboards treating the inference bill as if it were a productivity indicator. Boris Cherny, the Anthropic engineer who created Claude Code, has just called that habit into question: as reported by Business Insider on July 17, there is a better way to measure AI success than token burn.

The nuance matters because of who is saying it. Claude Code is now the flagship agentic tool of the Anthropic ecosystem, with skills, subagents, hooks and MCP servers orbiting around it, and few people have seen as much real usage data as its creator. When that voice points out that the industry is looking at the wrong number, it pays to listen.

Where the token obsession comes from

The token is the billing unit of language models, not a unit of value. Aggregate consumption going up says a lot about spend and very little about outcomes. A misconfigured agent that retries the same task five times burns five times more tokens than a well tuned one, and produces exactly the same pull request. By the token burn yardstick, the clumsy agent looks five times more successful.

The habit has a simple explanation: it is the easiest number to get. Any billing console provides it out of the box, it grows almost by default month after month, and it looks good on a board slide. Measuring outcomes, on the other hand, requires instrumenting the actual workflow: which tasks came in, which came out finished, and how much it cost to review them. There is also an obvious incentive worth naming: for model providers, tokens consumed are literally their revenue line. That this same figure has crept in as a success metric inside the companies paying the bill is much harder to justify, and it recalls Goodhart's old law: when a measure becomes a target, it ceases to be a good measure.

Measure finished work, not fuel

The thesis Business Insider attributes to Cherny points in that direction: AI success is measured by what the work produces, not by the compute it consumes. In practice, the metrics already used by the most mature teams all point the same way:

Tasks closed end to end without human rescue intervention.
Merged pull requests that survive review and generate no rework.
Time from ticket opened to change shipped to production.
Usage retention: how many engineers come back to the tool every week by their own choice.

None of this is exotic. Claude Code itself exposes usage telemetry via OpenTelemetry and a /cost command to check each session's spend, so crossing consumption with outcomes is a matter of discipline, not technology.

Who this matters to

For a CTO justifying an AI budget to leadership, the difference is substantial: reporting tokens is reporting cost, and presenting it as an achievement invites the awkward question of what was obtained in return. For platform teams deploying internal agents, changing the metric changes the incentives: if consumption is rewarded, nobody optimizes prompts or contexts; if outcomes are rewarded, efficiency shows up on its own. And for those building services on top of Claude (agencies, consultancies, integrators), the message works almost as a style guide: an adoption report that leads with tokens consumed describes activity; one that leads with tasks solved describes value.

Our take

At ElephantPink we have been applying this principle for months in Claude integrations for clients: our reports show tasks closed and hours recovered, and token consumption goes in the cost section, which is where it belongs. Having the creator of Claude Code publicly push in the same direction helps the conversation about AI's real return finally mature.