RL versus SFT: what changes inside a reasoning model

A model tuned with reinforcement learning and one tuned with supervision can give the same correct answer while having reached it through different internal architectures. That is the claim in a preprint uploaded to arXiv on 31 July (arXiv:2607.26119v1), which looks for the mechanistic basis of an already familiar observation: reasoning models trained with RL perform better on mathematical tasks than their SFT trained counterparts.

The question the authors ask is not whether RL works better, something benchmarks have been repeating for over a year, but which internal difference explains it. To answer it they use two methods that support each other.

Linear probes over hidden states

The first trains linear probes on layer by layer hidden states to predict whether the final answer will be correct. It is a very simple classifier: if it succeeds, that information is already encoded in a linearly separable way inside the activations. According to the paper, the probes reach higher accuracy on RL trained models than on SFT ones. Translated: the RL model carries a cleaner signal about whether it is on the right track, and that signal can be read without sophisticated tooling.

Mean ablation and layer hierarchy

The second method replaces a layer's activations with their mean and measures how much performance degrades. This is where the most interesting part of the work shows up: in RL models the deeper layers become progressively more critical, while in SFT models importance is spread uniformly across the network. Put differently, RL does not add a capability on top of what was already there; it reorganises where the work happens. The authors describe this as an emergent hierarchical architecture, which fits the intuition that long reasoning needs distinct stages of compute.

What the abstract leaves out

Expectations should come down on three points. The public abstract ends by discussing token count variability under repeated sampling, a way of measuring whether the model allocates compute according to problem difficulty, and it cuts off right as the result arrives: all we read is that they observe higher variability in some RL tuned models. Second, probing is not explaining. A linear probe predicting correctness shows the information is present, not that the model uses it causally to decide. And third, this is a version 1 preprint with no peer review, and the summary does not specify which model families or which problem sets enter the comparison. That last point matters quite a lot, because interpretability results travel badly across architectures.

Who this is useful for

For teams doing post-training on open weights there is a direct application. If correctness is linearly readable from hidden states, a probe trained once is far cheaper than calling a second model as a judge to estimate confidence. It is the kind of signal that helps decide whether to retry, whether to escalate to a larger model or whether to ask for human review, and it costs a fraction of one output token. The catch is obvious: you need access to the activations, so anyone consuming closed models over an API is left out. For that case the paper works mainly as an explanatory frame for why reasoning variants behave differently and not just better.

Our reading

We like that the work commits to a structural hypothesis instead of adding another benchmark table, and the depth ablation is the result we find hardest to explain by chance. That said, until we see the full PDF with the specific models and data, we would treat it as a solid lead rather than a conclusion. The full text is on arXiv.