A paper argues we should define reasoning before measuring it

A position paper posted to arXiv on 14 August makes an uncomfortable claim for almost anyone publishing evaluations: the generative AI community has not converged on an operational definition of "reasoning", and without one the construct validity of any benchmark claiming to measure it goes unverified. The paper, titled Position: Reasoning is a Learnable Rule-Based Process, gets straight to the point: it is not that the results are false, it is that you cannot show what they measure.

The argument has a historical side. Automated reasoning was for decades the territory of symbolic AI and formal logic, with a fairly strict verification tradition: reasoning is valid if the conclusion follows from the premises, and sound if the premises are also true. The authors argue that recent progress, built almost entirely on deep probabilistic generative models, arrived while implicitly rejecting that earlier treatment, and that the resulting ambiguity is in fact addressable.

Construct validity, in plain terms

A benchmark has construct validity when there is evidence that it measures the concept it claims to measure and not some correlated one. With accuracy on maths or logic problems, what gets recorded is whether the final answer matches the expected one. A model can reach that answer through an invalid chain, through memorisation of the training set, or through a statistical shortcut that happens to work in that particular question format. The metric does not tell the three cases apart, and without an operational definition there is no way to decide which of them counts as reasoning.

What the paper proposes

It contributes two things. First, a set of operational definitions synthesised from the literature, positioning valid and sound reasoning as a learnable rule based process: rules, because it allows step by step auditing; learnable, because it does not require hand writing a full symbolic system again. Second, a checklist of best practices for communicating AI reasoning research. That second part has the most immediate practical value, because it forces you to state which definition you are using before showing a number.

Why it matters if you build agents

It concerns anyone evaluating models, but also anyone running agents in production. When you delegate a task to a subagent and only check whether the final output is correct, you are running a domestic version of the same mistake. A task that succeeds with faulty intermediate reasoning will fail again as soon as one input changes, and by then you will have built on top of it.

We have seen this with pipelines that passed tests for weeks and broke on the first case outside the distribution of the examples. The checklist works well as a template for writing internal evaluations: define what counts as a valid step, log the chain and not just the result, and separate correct answers backed by correct reasoning from correct answers reached by luck. You do not need formal logic for that, only to stop treating the final answer as the single signal.

The limits

It is worth remembering what this piece is and is not. It is a position paper: no new benchmark, no empirical results, and its central thesis is arguable by design. Anyone who holds that LLM reasoning is an emergent phenomenon that resists reduction to explicit rules has plenty of room to disagree, and that debate has been open since well before these models existed. A definition also does nothing to fix a test set leaked into training.

Our view: the definitional part will fuel a long debate and probably settle nothing. The checklist, by contrast, can be applied tomorrow and costs little. If the only thing that survives from this paper is the habit of saying what you mean by reasoning before publishing the number, it will have been worth it.