verdict

# Verdict

[![ci](https://github.com/ArtJack/verdict/actions/workflows/ci.yml/badge.svg)](https://github.com/ArtJack/verdict/actions/workflows/ci.yml)
[![verdict on itself](https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fraw.githubusercontent.com%2FArtJack%2Fverdict%2Fmain%2F.qa%2Fstate.json&query=%24.verdict&label=verdict%20on%20itself&color=blue)](.qa/reports/INDEX.md)
[![eval 8/8 seeded defects](https://img.shields.io/badge/eval-8%2F8_seeded_defects-brightgreen)](eval/README.md#published-results)
[![pinned rules 165/165 killed](https://img.shields.io/badge/pinned_rules-165%2F165_killed-brightgreen)](eval/README.md#suite-fault-detection-power--mutation-testing-on-ourselves)
[![PyPI](https://img.shields.io/pypi/v/verdict-qa-mcp?label=verdict-qa-mcp&color=blue)](https://pypi.org/project/verdict-qa-mcp/)
[![Claude Code plugin](https://img.shields.io/badge/Claude_Code-plugin-6E56CF)](#install)
[![license MIT](https://img.shields.io/github/license/ArtJack/verdict)](LICENSE)

**Your test suite is green. Verdict found a defect that had lived 4,595 days.**

Verdict is a QA agent that does not fix, does not flatter, and does not forget. It measures
before it judges — the harness runs your gates, hashes every line a finding cites, re-runs
the guarding test at the old commit and the new one — and it keeps a memory: every run is a
delta against the last, findings age, regressions rank first, and the tester's own misses
are published beside its hits. The contract it runs under is immutable and hashed into every
verdict; what it learns lives beside the contract, dated and auditable, and never edits it.
The number above is real: `FilePerms` in a 4k★ Python library could not revoke a permission
bit since 2014-02-07, and every one of its 625 tests was green the day Verdict filed it —
[the run, and the misses, are in the ledger](eval/README.md#runs-on-strangers-repositories).

```
/plugin marketplace add ArtJack/verdict     # Claude Code
/plugin install verdict@verdict
/verdict:run
```
```
npx skills add ArtJack/verdict              # every other coding agent
```

Most AI "QA agents" are a paragraph of enthusiasm with a checklist. They audit your repo
from scratch every time, re-report the same 20 findings until you stop reading, call flaky
tests "failures", call stale tests "failures", and end with "LGTM! 🎉".

Verdict is a Claude Code plugin built the way QA is actually practiced:

- **It remembers.** A state file carries the baseline. Every finding gets a stable ID and
  an age; every run reports `NEW / STILL_OPEN / RESOLVED / REGRESSED` — regressions ranked
  first, always.
- **A red test means something.** Every failure is classified — `REAL_DEFECT`,
  `STALE_EXPECTATION` (which needs a citation proving the change was intended),
  `BRITTLE_TEST`, `ENVIRONMENT`, or `FLAKY` (confirmed by re-runs, quarantined **with an
  expiry**). The classification most likely to excuse a regression carries the highest
  evidence bar.
- **A verdict you can defend.** Every run ends in exactly one of
  `pass | pass with risks | blocked | fail` — an open Blocker forces `fail`, `blocked` is a
  legitimate outcome, and a `pass` always names what was *not* tested.
- **It never fixes your code.** There is no `Edit` tool, a hook confines its writes to the
  QA root, and a strict-mode Bash guard closes the shell's write channels — a tester that
  patches what it judges isn't independent. The guard is a heuristic, not a sandbox, and
  [the README says so](#the-read-only-guarantee-honestly-stated).
- **It is tested, and it tests itself.** A scored eval suite with the misses published, a
  signed run history the model cannot forge, a track record the tester cannot edit — and a
  nightly audit of its own releases: 69 findings filed against itself so far, every one
  fixed in a tagged release with the defect pinned as a mutant the suite must kill.

![A Verdict delta run: verdict first, REGRESSED findings ranked on top, a flake quarantined with an expiry, and the gate's exit codes keeping "never ran" apart from "said no"](docs/demo.svg)

**Who pays for the model?** You do, with the Claude subscription you already have: the
plugin runs inside your own session, nothing routes through anyone else, and everything
below the model — the state, the gate, the MCP server, the eval scorer — is stdlib Python
that runs for free. Works on Python, TypeScript, Go, or anything with a test runner; the
[eval fixtures](eval/) cover Python and TypeScript.

**Read next:** [Install](#install) · [What installs, and when it runs](#what-installs-and-when-it-runs) ·
[Quickstart](#quickstart) · [Why another QA agent](#why-another-qa-agent) ·
[The tested tester](#the-tested-tester) · [CI gate](#ci-gate-prs-on-the-testers-memory) ·
[Accepting a risk](#accepting-a-risk--the-maintainers-pen) · [FAQ](#faq)

## Install

```
/plugin marketplace add ArtJack/verdict
/plugin install verdict@verdict
```

Any other coding agent — Cursor, Codex, OpenCode and the rest of the
[agent skills](https://skills.sh) ecosystem — gets the same doctrine as five skills, and the
same harness as a pip package:

```
npx skills add ArtJack/verdict        # release risk · verify a fix · flaky triage · root cause · spec review
pip install verdict-qa-mcp            # verdict-facts, verdict-finalize, verdict-gate, verdict-accept, verdict-answer
```

[![skills.sh](https://skills.sh/b/ArtJack/verdict)](https://skills.sh/ArtJack/verdict)

The skills restate the contract for an agent that cannot run the `verdict` agent; the
hooks that enforce the read-only guarantee exist only in Claude Code, so there the
guarantee is the agent's own discipline plus the harness's refusals. `AGENTS.md` and
`llms.txt` at the repository root are for agents that read before they act.

**Python 3.9 or newer**, whatever your `python3` resolves to — the hooks and the
fact harness are stdlib-only and are invoked by that name, which on a stock Mac is
`/usr/bin/python3` (3.9). The optional MCP server is a pip install and needs 3.10+,
which is what `requires-python` in `pyproject.toml` refers to. The floor is tested:
a module that would fail to import on 3.9 fails CI instead
([tests/test_interpreter_floor.py](tests/test_interpreter_floor.py)) — because the
failure it prevents was silent. The Bash guard once raised on import there while the
write guard beside it kept working, so a strict session looked armed with half its
controls missing.

## What installs, and when it runs

Installing a plugin means letting its code run in your sessions, so here is exactly
what this one does — measured from `hooks/hooks.json`, not summarised from memory.
Six hook registrations; each starts a `python3` (tens of milliseconds) when its
event fires:

| Event | Fires on | Script | Silent when |
|---|---|---|---|
| `PreToolUse` | `Write`/`Edit`/`MultiEdit`/`NotebookEdit` | write-scope guard | always, unless `VERDICT_STRICT=1` or the caller is the verdict agent itself |
| `PreToolUse` | `Bash` | bash-scope guard | always, unless `VERDICT_STRICT=1` |
| `PostToolUse` | `Write`/`Edit`/`MultiEdit` | state validator | unless the written file is literally named `state.json` |
| `Stop` / `SubagentStop` | end of turn | run-contract check | unless a QA run *in this session* left hand-written state — and it blocks **at most once**, never loops |
| `SessionStart` | session open | findings banner | unless the repository has QA state |

Every hook **fails open**: malformed input, missing files, or an exception mean
exit 0 and silence — a broken hook must never brick a session. `VERDICT_STRICT=1`
is what arms the scope guards, and you set it only for dedicated QA sessions
(headless, CI, the nightly); in ordinary interactive work the guards are no-ops.

**Prefer not to install globally?** Everything works per-repository: copy
[agents/verdict.md](agents/verdict.md) into `<repo>/.claude/agents/` and the
[hooks/hooks.json](hooks/hooks.json) entries into `<repo>/.claude/settings.json`,
with `${CLAUDE_PLUGIN_ROOT}` replaced by a checkout path. That is exactly how the
eval harness provisions its scratch projects — [eval/run_eval.py](eval/run_eval.py)
is the reference implementation.

## Quickstart

```
/verdict:run                          # first run: profile + isolation rules + baseline
/verdict:run the payment retry change   # every later run: a delta against the stored state
```

Every later run is a delta against the stored state. A repeat run returns something like:

```text
VERDICT: fail

Scope: 2c67f47..b4e2943 (4 commits, 16 files) · run 4 (delta)
Isolation check: pass (no .env present; no live service touched)

Findings — REGRESSED first:
REGRESSED  PRICER-F-002  Critical/P0  round_cents uses banker's rounding again (pricer.py:17)
                         resolved 08-19, reintroduced by b4e2943 — this forces the verdict
NEW        PRICER-F-007  Major/P1     quarantine graveyard: test_listable_at_floor_exactly
                         skipped 114 days with no expiry — it is the test that would catch F-001
STILL_OPEN PRICER-F-001  Critical/P0  age 6d  is_listable rejects a price exactly at the floor
FLAKY      test_bulk_discount_applies — fails 3/6 runs with no code change; quarantined
           until 2026-09-07, excluded from this verdict, listed until re-evaluated

Delta gates: tests 213 → 213 · duration +0.4% · coverage on changed files: no decrease
Release blockers: PRICER-F-002 (regressed), PRICER-F-001
Not tested: concurrency under parallel checkout — no harness present
Fix order: 1) F-002  2) F-001 (unskip its test first, watch it fail red)  3) F-007 expiry
Artifact: .qa/reports/2026-08-24-pricer-review.md
```

## Why another QA agent

| | Typical `qa-expert.md` | Verdict |
|---|---|---|
| Remembers the last run | no — every run is a fresh audit | state file; `NEW/STILL_OPEN/RESOLVED/REGRESSED` with ages |
| Flaky tests | "keep flakes under 1%" (prose) | quarantine ledger with mandatory expiry; flakes excluded from the verdict, never from the report |
| Relea