AgentLens: judging code agents by their trajectory, not just the final result

Most benchmarks for coding agents reduce each run to a single bit: the task passed or it did not. That criticism is not ours, it comes from the authors of AgentLens, a paper published on July 9 on arXiv that proposes evaluating exactly the opposite: the agent's entire trajectory, from the first instruction to the final message.

The starting point is an observation that is hard to argue with. Whoever uses a coding agent day to day does not experience a binary outcome but the whole journey: whether the agent follows instructions or reinterprets them in its own way, how it uses its tools, whether it verifies its own work before calling it done, how it recovers when it makes mistakes and how it communicates along the way. Two agents can solve the same task and deliver opposite experiences.

How AgentLens evaluates

The system combines two mechanisms. Where an objective check exists (tests, formal verification), it applies it. Where it does not, it relies on trajectory reviews written by an LLM and side by side comparisons between runs. The result is that every run produces a readable explanation of why the score is what it is, instead of a bare number.

The authors stress that this turns the benchmark into something more than a leaderboard. They use it themselves for three things: diagnosing model behavior, comparing successive versions of their own agent and catching product regressions in a nightly evaluation pipeline. The code is released as open source on GitHub.

Why the single bit falls short

Metrics based on the percentage of solved tasks have been useful for comparing models, but they hide differences that matter in real use. One agent can pass the task after touching twenty files nobody asked it to touch. Another can solve it just as well while asking for confirmation before deleting something delicate. On a classic leaderboard both score the same; for the developer who suffers or enjoys them, they are nothing alike.

For teams building products on top of agents, the nightly pipeline argument is probably the most practical one in the paper: behavioral regressions (an agent that suddenly verifies less, or overuses a tool) do not show up in success rates until they are already serious. A trajectory review catches them earlier.

Reasonable caveats

Reviews written by LLMs inherit the biases of the model that writes them, and side by side comparisons have well known issues, such as sensitivity to presentation order. Pairing them with formal verification where possible mitigates part of the risk, but does not remove it: the subjective part of the benchmark is only as reliable as its judge. Open code helps here, because it lets you audit the evaluation prompts instead of trusting the grade.

Who this is useful for: teams building coding agents who want to compare versions of their own, model evaluators who need something finer than pass@1, and product teams that want continuous evaluation in place. For the end user of tools like Claude Code the effect is indirect but real: if vendors adopt trajectory evaluations, agents that fail gracefully will stop scoring the same as agents that just fail.

At EP we evaluate agents often, and the pass or fail verdict never tells the whole story. A benchmark that explains the why behind every grade is worth more as a diagnostic tool than as a ranking, and that seems to be exactly the use its authors are giving it.