verdict-mcp

# verdict


https://github.com/user-attachments/assets/62923912-98aa-4246-8bc8-4adf55ae3ff8



[![CI](https://github.com/Dgotlieb/verdict-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/Dgotlieb/verdict-mcp/actions/workflows/ci.yml)
[![PyPI](https://img.shields.io/pypi/v/verdict-mcp)](https://pypi.org/project/verdict-mcp/)
[![Python](https://img.shields.io/pypi/pyversions/verdict-mcp)](https://pypi.org/project/verdict-mcp/)
[![License](https://img.shields.io/badge/license-Apache--2.0-blue)](LICENSE)

**Structured, sandboxed verification feedback for coding agents.**

An [MCP](https://modelcontextprotocol.io) server that replaces your agent's `pytest` shell-outs with something built for the agent inner loop: impact-selected tests, run in an isolated environment, returning **compact typed verdicts** instead of 40,000 tokens of raw runner output — with **failure fingerprints** that tell the agent whether a failure is *its* regression or was broken all along.

```
raw pytest dump:  ~40,000 tokens, unstructured, run un-sandboxed on your machine
verdict:              ~400 tokens, typed JSON, run in a rootless container, with memory
```

## Why

The highest-frequency tool call in agentic coding is verification — and it's the least structured. Agents re-run whole suites when one module changed, burn context parsing ANSI-coded tracebacks, run arbitrary code directly on your machine, and routinely misdiagnose pre-existing breakage as their own regression (then "fix" code that wasn't broken). verdict fixes all four.

## Tools

| Tool | What it does |
|---|---|
| `verify(scope?, base?)` | Selects tests affected by your working-tree diff (static import graph via grimp), runs them via podman/docker with the worktree mounted **read-only**, returns typed failures with fingerprints and a `preexisting` flag |
| `explain_failure(check_id)` | Full traceback for one failure, on demand — bulk never rides in the summary |
| `history(fingerprint)` | First seen / last seen / times seen — regression vs. long-standing breakage |
| `run_checks(["ruff","mypy"])` | Lint and type checks, normalized into the same verdict schema |

Every failure carries a **fingerprint**: a stable hash of the normalized failure signature (volatile tokens — addresses, tmp paths, ids, durations — collapsed). Same logical failure, same fingerprint, across runs and refactors. Fingerprints are what give verdict memory.

## Quickstart

No install step needed — `uvx` fetches it on first use. (Or `uv tool install verdict-mcp` / `pip install verdict-mcp` for a permanent `verdict-mcp` command.)

**Claude Code** — `.mcp.json` in your project root:

```json
{
  "mcpServers": {
    "verdict": {
      "command": "uvx",
      "args": ["verdict-mcp"],
      "env": { "VERDICT_PROJECT": "." }
    }
  }
}
```

**Cursor** — same shape in `.cursor/mcp.json`.

Optional `verdict.toml` in your repo root:

```toml
[project]
packages = ["your_package"]          # for impact selection (auto-guessed if omitted)

[runner]
image = "ghcr.io/you/yourproj-test"  # prebuilt env with your deps
setup_cmd = "pip install -e .[test]" # or install on the fly (runs with network; tests don't)
# prefer = "local"                   # escape hatch if you have no container runtime

[limits]
max_failures = 10
```

Try it without an agent:

```bash
cd examples/demo_project
VERDICT_PROJECT=. verdict-mcp   # then connect any MCP client, or use the MCP inspector
```

## Sandbox posture (v0.1)

Checks run in an ephemeral container (podman preferred, docker fallback): worktree mounted **read-only** at `/src`, copied to a writable `/work` inside the container, `--network=none` for the check run. Your host environment is never mutated by a test run. If `setup_cmd` is configured, that step runs *with* network before the check; prefer a prebuilt image for a tighter posture. No container runtime → explicit `prefer = "local"` fallback runs checks against a temp copy of your worktree (still never in place). See [SECURITY.md](SECURITY.md) for the full threat model and known limitations.

**Troubleshooting:** if a verdict says `container engine 'podman' could not start the check`, run the suggested `podman pull <image>` by hand — the engine's own error is the answer. One known trap on macOS: a `"credsStore": "gcloud"` line in `~/.docker/config.json` makes podman call the gcloud credential helper for *every* registry, including docker.io; an expired gcloud login then breaks all pulls. Fix with `gcloud auth login` or remove that line.

## Honest limitations

- Impact selection uses the **static import graph** — approximate by design. Dynamic imports, fixture-by-name resolution, and data-driven tests can be missed; `verify(scope="all")` is always available and verdict says in `selection_note` whenever it falls back.
- Python/pytest only today, plus ruff/mypy. The adapter interface is small and documented — vitest and `go test -json` adapters are the most-wanted contributions ([CONTRIBUTING.md](CONTRIBUTING.md)).
- Flake detection and coverage-map-based selection are v0.2 ([roadmap](#roadmap)).

## Roadmap

**v0.2:** coverage-based impact maps (precise selection), flake detection via fingerprint alternation, devcontainer.json support, result cache keyed on (tree hash, check, image digest). **Later:** vitest/jest, go test, cargo test adapters; per-repo verdict daemon mode.

## License

Apache-2.0

<!-- mcp-name: io.github.Dgotlieb/verdict-mcp -->