skill-eval

# /skill-eval

Author one behavioral probe for one skill, at the cheapest tier that can still
separate the arms, and report the verdict honestly. A probe measures
**behavior-change** — did loading the skill change what the agent *did* — never
quality-uplift. This skill authors and tiers probes. `scripts/probe-skill.sh`
runs them.

**Insight:** when a probe returns INERT because the control arm already aces the
scenario, the measurement failed, not the skill. Weakening the producer is one
escape and it costs realism. The cheaper escape is to **plant the defect**: build
a scenario containing exactly one flaw the discipline catches and a skim does
not, then grade whether the agent acted on it. Signal you manufacture is signal
you can reproduce.

**The failure mode this exists to prevent:** a skill catalog whose tier badges are
editorial. A skill nobody measured is a skill nobody can defend, and re-running a
saturated scenario at a lower effort level produces more rows in the ledger
without producing more knowledge.

## Modes

| Trigger phrases | Mode | Entry point |
|---|---|---|
| "measure this skill", "does this skill do anything" | author tier 1 (quiz probe) | `evals/skill-probes/<id>/` |
| "the control arm aces it", "harden this scenario" | author tier 2 (seeded-defect probe) | [`references/seeding.md`](references/seeding.md) |
| "the probe came back INERT" | diagnose headroom | gate `skill.probe-headroom` |
| "run the probes" | run a tier | `scripts/probe-skill.sh` |

## Inputs

Required: the skill slug, and one sentence naming the **action** the skill should
cause — a tool call made, an artifact written, a question raised, a sequence
followed. If the sentence names a belief instead of an action ("understands
that…", "considers…"), stop: that is not probeable, and rewriting it as an action
is the actual work.

Optional: an existing probe id to harden.

**Non-goals.** This skill does not score output quality, rank skills, claim a
skill is good, or gate a release. It does not run `claude -p`. It does not
generalize from N=2 — small N is directional and every artifact it produces says
so.

## The two tiers

| | Tier 1 — quiz | Tier 2 — seeded task |
|---|---|---|
| Scenario | asks the agent a question about a situation | hands the agent work containing a planted defect |
| Grades | which answer it gave | whether it acted on the defect |
| Saturates | fast — frontier models answer doctrine questions correctly unaided | slowly — skimming is a real failure mode at every altitude |
| Cost | low | higher (real task, longer transcript) |
| Use when | the skill's whole content is a decision rule | tier 1 saturated, or the skill's value is *noticing* |

Historical tier-1 groups saturate repeatedly at both `xhigh` and `low` effort —
harder quizzes did not fix it (`validate-not-proven-v2` re-saturated). Run the
`skill.probe-headroom` gate for the live classification; do not trust a
hardcoded count. Tier 2 is the escape, because it
changes what is being measured from *knowing the rule* to *applying it while busy*.

## Procedure

1. **State the action.** One sentence, an observable act. Reject beliefs.
2. **Pick the tier.** Start at tier 1 unless a prior probe for this skill is
   saturated; then go straight to tier 2.
3. **Build the scenario.** For tier 2, seed exactly one forcing defect using the
   rules in [`references/seeding.md`](references/seeding.md). One defect for a
   floor probe; N defects for a band probe.
4. **Write the discriminator.** Deterministic, over one transcript. Exit `0`
   present, `1` absent, `2` infra. It checks the **act**, never a mention — a
   discriminator that greps for a word the prelude contains measures the prelude.
5. **Calibrate on replay** against committed fixtures before spending a live run.
6. **Run both arms** at two effort levels. Same scenario, same reps; the
   declared `treatment_source` is the only variable — `canonical-skill` (the
   exact SKILL.md bytes; the only mode the coverage gate counts) or
   `injected-prelude` (prelude-only evidence, never skill coverage).
7. **Pre-screen headroom before believing the verdict** — gate
   `skill.probe-headroom`. A verdict over a saturated scenario is void.
8. **Record the outcome and stop.** Pre-screen passed (`SEPARATED`/`FLOOR`):
   append exactly one ledger row in `evals/skill-probes/LEDGER.md`. `SATURATED`:
   append nothing to the ledger — note the scenario's retirement in the RUNBOOK.

```bash
# Calibrate deterministically against committed fixtures.
bash scripts/probe-skill.sh --probe <id> --replay

# Live A/B (codex exec — the sanctioned headless path).
bash scripts/probe-skill.sh --probe <id> --live --capture --reps 3 --output out.json

# Is the verdict trustworthy, or did the control arm ace it?
bash scripts/check-skill-probe-headroom.sh
```

### Floor and band

- **Floor probe** — one seeded defect, assert the agent acted **at least once**.
  Catches the total no-op: the review that produced a polished report naming
  nothing.
- **Band probe** — N seeded defects, assert findings land in **[N-1, N+2]**.
  The lower bound catches rubber-stamping; the upper bound catches spray, where
  an agent lists every conceivable concern and is credited for the one that
  happened to be planted. A probe with only a floor rewards noise.

### Saturation rule — owned by the gate, not by this skill

The rule is **deterministic, so it does not live here.** Gate
`skill.probe-headroom` owns it: a scenario is **SATURATED** when the control arm
scores **≥ 0.75 at two or more effort levels** with at least 2 usable control
reps each. The rule, its thresholds, and its exit codes are
`cli/internal/probeheadroom`; the gate script is
[`scripts/check-skill-probe-headroom.sh`](../../scripts/check-skill-probe-headroom.sh)
and the helper it drives is `cli/cmd/probe-headroom`. Do not restate the
thresholds in a probe package or re-derive them by hand — read the gate's answer.
This skill's job is what to DO with that answer:

- **SATU