Beyond accuracy: what to measure when a benchmark saturates

When an agent benchmark hits the accuracy ceiling, the usual reaction is to retire it and replace it with a harder version. A paper just published on arXiv argues that this habit wastes valuable information. According to Life After Benchmark Saturation: A Case Study of CORE-Bench, looking only at correctness ignores six other dimensions of an agent's performance that remain measurable even after accuracy has saturated.

The case study is CORE-Bench Hard, a benchmark that measures the computational reproducibility of scientific code: given the materials of a paper, the agent has to reproduce its results. It is a demanding and, above all, verifiable task, which makes it a good test bed for the paper's thesis.

The six dimensions accuracy hides

The authors list six aspects that a single accuracy number leaves out. The first is construct validity, that is, whether the agent solves the task for the right reason or through shortcuts. The second is out of distribution generalizability: what happens when the task moves away from the examples seen. The other four are efficiency (how much it costs to reach the answer), reliability (whether the result repeats), the weight of the model versus the scaffold around it, and the uplift from human and agent collaboration.

The underlying idea is that two agents with the same accuracy can be very different. One may be right while spending three times the tokens, fail at the first context change, or depend entirely on a carefully tuned scaffold. None of that shows up in the accuracy figure.

The shortcut problem

One of the most useful findings concerns construct validity. When testing more capable agents on CORE-Bench Hard, the authors detect shortcuts and threats to measurement that went unnoticed with weaker agents: the system appears to solve the task, but does so by exploiting cues from the benchmark itself rather than genuinely reproducing the code. It is an uncomfortable reminder that a high score does not always mean what we think.

From there, the work introduces two artifacts. CORE-Bench v1.1 is a corrected version of the benchmark, cleaned of some of those shortcuts. CORE-Bench OOD is an out of distribution task suite designed to check whether what was learned holds in new scenarios.

What a saturated benchmark still measures

The second set of results is the one that gives the paper its title. Even with accuracy saturated, CORE-Bench v1.1 remains useful for measuring efficiency, reliability, model performance and scaffold performance. Put another way: the benchmark is not exhausted, we were simply reading a single variable from it.

That distinction between model and scaffold matters to anyone building agents. Much of the recent improvement comes not from a different model but from the scaffolding around it (tools, memory, verification loops). Separating both contributions tells you where to invest.

Finally, the authors run a small scale randomized experiment to estimate the contribution of human and agent collaboration. It is a first step, with the caution its size demands, but it points to a dimension almost no benchmark captures: how much the result improves when there is a human in the loop.

Who it is useful for

The paper mainly matters to those who evaluate agents seriously: research teams, benchmark authors and product leads who need to decide between models. Its practical message is that retiring a benchmark as soon as it saturates throws away an instrument that still measures things.

At ElephantPink, when we compare agent configurations, accuracy is rarely the deciding figure: cost per task and consistency across runs are. That a paper formalizes that instinct and proposes how to measure it strikes us as more valuable than another leaderboard with a tenth of a point more accuracy.