OpenDiscoveryTrace: 558 traces to audit AI scientist agents

A benchmark that only looks at the final output cannot tell a model that reasons from one that guesses well. That is the premise behind OpenDiscoveryTrace, published on arXiv on 11 September: a public dataset of 558 complete AI scientific agent trajectories where what gets stored is not the hypothesis, the code or the paper produced, but every step of the process that led there. The authors frame it as an answer to a specific gap: without the process you cannot audit the methodology or diagnose why an agent fails.

The format is the interesting part. Each step is recorded with nine fields: thoughts, tool calls, observations, errors, revision triggers and self reported confidence, among others. The trajectories cover 124 scientific tasks spread across drug discovery, materials science, genomics and scientific literature analysis. On top of that come 60 live retrieval variant trajectories, that is, with access to external information during execution.

Which models are inside

The split is balanced on purpose. Three high end commercial models, one from Anthropic, one from Google and one from OpenAI in whichever versions were available when the tasks were run, contribute 124 trajectories each and cover every domain and difficulty level. The four open weight models (Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini and Qwen2.5-1.5B) contribute 30 each. That imbalance does not look like an oversight but a budget decision: with seven models and 124 tasks, full coverage would have multiplied the cost without adding much at the small end. Anyone wanting to compare specific providers will have to go to the paper and read the table, because the value of the dataset is not in the ranking.

Why auditing the process matters

The practical value is in diagnosis. With only the final output, a failure is a failure and little else: you cannot tell whether the model picked the wrong tool, misread an observation, ignored an error or ran out of step budget. With the full trace you can separate method errors from execution errors, and compare how each model reacts when something goes wrong halfway through. The revision trigger field is especially telling: it measures whether the agent corrects itself when the evidence contradicts it, or pushes on with its initial hypothesis.

One detail is worth not skipping: self reported confidence. Asking a model to score its own certainty at every step produces a noisy signal, but logged across hundreds of trajectories it enables something a final result never could: checking whether that confidence rises right before errors or genuinely tracks the correct answers. It is a cheap calibration to measure once the trace exists.

The pilot analysis shipped with the dataset rests on 363 LLM judged trajectories, a method with known biases that the authors themselves present as preliminary. It is worth reading for what it is: a first pass over material published precisely so that others can analyse it better and with different metrics.

Who it is useful for

For anyone evaluating agents, scientific or otherwise, the value lies more in the format than in the content. A nine field per step trace is a reasonable template for instrumenting any agent of your own: if you log thought, call, observation, error and confidence, you already have enough to debug when something breaks in production. In Claude Code that instrumentation can be built with PostToolUse hooks dumping every call to a file, without touching the agent logic or adding dependencies.

We care less about which model comes out ahead than about the underlying idea: measuring agents by outcome alone means measuring luck together with method, and afterwards there is no way to know which of the two you rewarded. Publishing full traces is uncomfortable because it exposes the ugly paths, and that is exactly why it is the kind of work worth seeing more often.