RealtimeBench: when an agent's correct answer arrives too late

With the game paused, GPT-6.1-sol beats Jev, a faster model. With the game running, the order flips. That result sums up the argument of The World Won't Wait, a paper submitted to arXiv on 14 October by Ningxin Pan, Hanyu Li, Yeqing Shen, Ning Ding, Yehui Tang and Wentao Zhang: in an environment that never stops changing, a correct action that arrives too late is worth the same as a wrong one.

Almost every agent benchmark assumes the world waits while the model thinks. Long reasoning or a chain of tool calls only costs money. The authors propose measuring exactly the opposite.

What RealtimeBench is

RealtimeBench brings together 180 tasks across six game types. What sets it apart from other evaluations is that the game keeps running during inference: if the agent takes too long to decide, the state of the environment is no longer the one it analysed. The evaluations are paired, with the same model in paused mode and in live mode, which makes it possible to isolate how much performance is lost to latency alone.

The case of GPT-6.1-sol versus Jev is the central example. The most capable model under ideal conditions is not necessarily the best one when time matters. The abstract does not give the figures for that comparison, but the pattern is enough to cast doubt on leaderboards that only measure answer quality.

TORSI: think slowly outside, act fast inside

The second contribution is an improvement method that leaves the model weights untouched, called Two-Timescale Offline Recursive Self-Improvement (TORSI). It works with two roles:

A slow teacher studies failures between attempts, tests corrections in paused games and writes reusable instructions.
A fast student plays in real time without calling the teacher; it only carries those instructions.

Updates are not accepted by default: they are adopted only if they improve the validation score. After two rounds, Jev's average test score rises from 13.51 to 24.19 on a 0 to 100 progress scale, with gains in all six game types. The same instructions also raise the average scores of three other models without any retuning.

That last finding may be the most practical. If the instructions the teacher writes work with other models, they look less like a tuning trick and more like distilled knowledge that can be versioned and reused.

Why it matters beyond video games

Games are a convenient test bed, but the problem shows up in any agent operating on something live: infrastructure monitoring, live support, robotics or a CI pipeline that does not stop while the agent decides. In all of those cases, latency is part of quality.

TORSI's architecture closely resembles how many teams already work with Claude Code. A large model such as Opus 4.8 analyses failed sessions and writes a skill or an instructions file; then a subagent running on a faster model such as Haiku 4.5 carries out the task with that context, without escalating at every step. The paper does not evaluate Anthropic models, but it formalises that pattern and adds a piece that is usually missing: an explicit validation criterion before accepting each change to the instructions.

Who it is useful for

For people designing agents for production, the study offers a quantitative argument against always choosing the most powerful model. For evaluators, RealtimeBench offers a metric that static benchmarks do not capture. And for teams maintaining prompt or skill libraries, TORSI suggests a reproducible improvement loop: collect failures, propose changes offline and keep only those that pass validation.

The limitations are worth keeping in mind. This is a preprint without peer review, and a score of 24.19 out of 100 makes it clear the problem is far from solved.

At ElephantPink we apply a similar idea in the custom agents we build: separating the moment of thinking from the moment of acting usually pays off more than looking for a single model that does both. We would like to see more benchmarks that measure with the clock running.