How prompt formatting shifts LLM leaderboard rankings

Changing only the wrapper around a prompt, without touching the question or the model, can move accuracy enough to flip a leaderboard. That is the uncomfortable conclusion of an arXiv study that analyzes 140,000 OpenRouter generations across 7 question answering tasks, 5 wrapper families and 4 instruct models between 7B and 72B parameters.

The wrapper is everything surrounding the question: the system instructions, the format example, the tags asking for the answer in JSON or between brackets. Two wrappers that differ only in that presentation layer should give almost identical results. The paper Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking shows that this is not the case, and measures it with a protocol that keeps the token count constant so the comparison is clean.

Two new metrics

The authors propose two indices. The Format Sensitivity Index (FSI) is the accuracy range caused by the choice of wrapper: how much a model's score rises or falls just by reformatting the request. The Parseability Sensitivity Index (PSI) measures the equivalent range in parseability, that is, how often the answer can be reliably extracted. The distinction matters because a correct answer the parser cannot read counts as a failure in any automated benchmark.

The central finding: mean FSI varies by more than 30 times across models, and that variation is largely explained by compliance failures. In other words, the models that score worst when the format changes do not reason worse, they stop following the requested schema and their answers become impossible to parse.

Why it matters

A fixed effects regression confirms the point: parseability remains a strong predictor of accuracy even after controlling for task, model and wrapper. That is, much of what a leaderboard attributes to model intelligence is actually the ability to stick to a format. If a ranking publishes accuracy without reporting format variance or compliance rates, the authors argue the figure is statistically fragile and can lead to ordering the models wrong.

For anyone reading benchmarks, the lesson is to read the fine print: a model topping a table with one wrapper can drop several places with another equally reasonable one. For anyone deploying LLMs in production, especially with structured output (JSON, tool calls, strict schemas), the warning is more direct: format robustness is a model property worth measuring apart from raw accuracy, because it is what decides whether the system holds up in the real world.

Who it is useful for

The work matters for three profiles. For those building evaluations who want reproducible numbers, who now have two concrete metrics to report alongside accuracy. For product teams that depend on structured outputs and need to choose a model by its parsing reliability, not just by its position in a ranking. And for anyone integrating models via MCP or tools, where an answer that does not respect the schema breaks the whole chain.

It matches what we see building integrations: much of what looks like reasoning failure is actually formatting failure, and it gets fixed by pinning down the schema well and validating the output before trusting it. Someone putting a number on that effect with 140,000 generations strikes us as more useful than another accuracy table with no context. The open question is whether these indices will be adopted in public leaderboards or remain a niche best practice.