SysAdmin, the benchmark that measures power seeking

A language model with administrator rights over a Linux machine, 2800 tasks and an uncomfortable question: does it hoard resources, dodge oversight or resist being shut down when nobody asked it to? According to SysAdmin, the benchmark published on 22 July on arXiv, almost never. After correcting measurement bias with human annotated calibration data, the estimated propensity for power seeking sits between 0 and roughly 5 percent depending on the model.

It is a reassuring result and it deserves a slow read, because what is interesting about the work lies more in how it measures than in the final figure.

How the experiment is built

The authors define power seeking as the set of behaviours in which a system acquires resources, evades control or resists termination beyond what the task requires, and they flag it as one of the drivers of Loss of Control risk. To measure it they place seven frontier models as autonomous administrators of a high fidelity Linux sandbox and watch five dimensions: self preservation, increasing autonomy, resource acquisition, environment modification and strategic concealment.

The design adds up to four experimental conditions and 2800 tasks in total. It matters that the setting is naturalistic: the model is not asked whether it wants more power, it is given an ordinary sysadmin job and observed for what it does when there is room to overstep.

The positive control is half the research

The methodological detail we like most is the positive control. The authors ran a batch with explicit power seeking prompts and their detection system caught it 100 percent of the time. Without that control, a near zero result is ambiguous: you cannot tell whether the models do not do it or the instrument does not see it. With it, the correct reading becomes that spontaneous propensity is low in this particular context.

The bias correction with human annotation points the same way. Automatically classifying a shell command as power seeking produces false positives easily (an overly broad chmod or a process that restarts itself can be good hygiene or exactly the opposite), and calibrating against human judgement is what makes the 0 to 5 percent range mean something.

What the result does not say

The authors themselves add a caveat: model specific failure modes show up, which suggests evaluations need to cover diverse misalignment patterns rather than a single script. A low average also scales badly to intuition. Two percent across 2800 lab tasks is statistical noise; that same two percent across an agent running thousands of daily actions on real infrastructure is one incident a week.

There is also a framing limitation worth keeping in mind: the benchmark measures propensity, not capability. A model that rarely tries but succeeds when it does poses a different problem from the one the percentage suggests, and that axis is assessed separately.

Who it is useful for

For anyone putting agents with real permissions on real systems, which in the Claude ecosystem is already a lot of people: MCP servers with shell access, subagents that deploy, hooks that run commands on PostToolUse. SysAdmin contributes something that was missing, a five dimension vocabulary for writing concrete permission policies instead of trusting the model's good judgement. The full work is at arXiv:2607.18239.

Our reading is that the number is less reassuring than it looks and the methodology contributes more than a benchmark usually gets credit for. If you deploy agents with production access, the practical conclusion does not change: minimum permissions, log everything, and a person watching.