Silent failures: when an agent's tool only half answers

A tool call that returns a 200 status code, well-formed JSON and no error message can still be failing. That is the core idea of a paper published this week on arXiv that audits 15 scientific tools integrated into ToolUniverse, along with their API and tool documentation, looking for a type of error that almost no benchmark measures.

The authors call it a silent failure: the invocation appears to have worked, but some of the information or functionality the tool offers through its API or wrapper is missing or incomplete, and neither the agent nor the user receives any warning. The agent keeps reasoning over partial data as if it were complete.

What the study measures

Most evaluations of agentic systems focus on whether the task gets completed. The paper, titled Silent Failures in Agent-Tool Interaction: An Audit of ToolUniverse, asks a different question: what happens at the boundary between the agent and the tool, specifically in biology workflows, an area where, according to the authors, research is still limited.

To answer it they develop an audit mechanism and organise the analysis around seven failure loci, that is, seven points in the chain where information loss can originate. The abstract stresses an important nuance: ToolUniverse is the experimental environment, not the object of study. The problem is not specific to that project but to the layer that wraps third-party APIs so a model can use them.

How to recognise a silent failure

Without going into the paper's specific cases, the pattern will be familiar to anyone who has written a connector:

An API paginates results and the wrapper only returns the first page, with no indication that there are more.
The API documentation offers filters or fields that the wrapper does not expose, so the agent never learns they exist.
An error in the upstream service turns into an empty list, which the agent reads as an absence of results.
A long field is truncated to save tokens and the response does not flag it.

In all these cases the agent receives a syntactically correct response and nothing in the flow prompts it to be suspicious. In biology the consequences are direct: a search for variants, interactions or publications that returns half the results leads to conclusions that look solid and are not.

Why it matters beyond the lab

Although the study focuses on scientific tools, the lesson applies to any MCP server that wraps an external API. MCP settles how a tool is described and invoked, but the fidelity of what it returns depends on whoever writes the wrapper. The specification already includes useful pieces, such as the `isError` flag in tool results or structured output schemas, although they only help if the server uses them honestly.

Based on this diagnosis, these are the practices we recommend to anyone building connectors:

Periodically compare the wrapper's output with that of the original API, rather than just checking that it responds.
Explicitly state when a response is partial: total results, pagination cursor, omitted fields.
Distinguish in the response between "no data" and "data could not be retrieved".
Review the wrapper's coverage against the API documentation every time the API changes.

In Claude Code, a `PostToolUse` hook can also inspect a tool's response before the agent moves on, a good place for completeness checks in sensitive workflows.

Who it is useful for

It is relevant to bioinformatics and R&D teams that already chain agents to public databases and to those who build or maintain MCP servers. It is also relevant to anyone who evaluates agents only by success rate, which is perhaps the group that most needs to read it: a task marked as completed with incomplete data counts as a success in almost any benchmark.

We find the paper valuable precisely because it is not about models but about plumbing. Many of the errors we see in production agents originate in that layer, and auditing wrappers with the same rigour as prompts is still a pending task in many projects.