falsepass

# falsepass

<!-- mcp-name: io.github.Super-Eric/falsepass -->

English | [中文](https://github.com/Super-Eric/falsepass/blob/main/README.zh-CN.md)

[![CI](https://github.com/Super-Eric/falsepass/actions/workflows/ci.yml/badge.svg)](https://github.com/Super-Eric/falsepass/actions/workflows/ci.yml)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-blue.svg)](https://github.com/Super-Eric/falsepass/blob/main/LICENSE)
[![PyPI](https://img.shields.io/pypi/v/falsepass)](https://pypi.org/project/falsepass/)
[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue.svg)](https://github.com/Super-Eric/falsepass/blob/main/pyproject.toml)

Probe RL environment verifiers for **false passes** (a known-bad completion is
rewarded) and **false fails** (a known-good completion is not).

Website and docs: **https://super-eric.github.io/falsepass/** ·
write-up: [Your reward function is an assertion](https://super-eric.github.io/falsepass/blog/reward-functions-are-assertions/)

In one public environment, echoing the question back earned full reward on every task:
each question listed all of its candidate answers, and the grader only checked that the
right one appeared somewhere in the reply, so a policy could collect full reward by
listing everything. falsepass finds this kind of hole before training does.

![falsepass demo: a planted grading defect found from two sides](https://raw.githubusercontent.com/Super-Eric/falsepass/main/docs/demo.gif)

An RL environment is a task dataset plus a verifier that turns a model's output into
reward. `falsepass` treats that verifier as the system under test: it calls the
environment's own reward code on completions whose correctness is known in advance and
reports every disagreement. No model, no rollouts, no GPU.

It works with three frameworks, published or still in a local folder:

| Framework | What falsepass runs | `falsepass init …` |
|---|---|---|
| [verifiers](https://github.com/PrimeIntellect-ai/verifiers) (Prime Intellect Environments Hub), all three API generations | the environment's rubric or taskset | `owner/environment`, or `./folder` |
| [Inspect AI](https://inspect.aisi.org.uk) | the task's solver chain with a scripted model, then its scorers | `path/to/task.py`, `./project`, or `inspect_evals/gsm8k` |
| [HUD](https://docs.hud.ai) | the task template's grader, driven like HUD's server | `path/to/env-folder` (add `--answers refs.json` for every probe) |

## Install

Needs [uv](https://docs.astral.sh/uv/) on the PATH (every audit runs in its own
throw-away virtualenv) and Python 3.11 or newer.

```bash
uvx falsepass demo              # try it, nothing to install
uv tool install falsepass       # or keep `falsepass` on the PATH
pip install falsepass           # or into an existing environment
uvx --from git+https://github.com/Super-Eric/falsepass falsepass --version   # the unreleased main branch
```

## Quick start

```bash
falsepass demo                            # audits a bundled toy environment with one planted defect (about a minute)
falsepass init owner/environment          # a Hub environment: package, version and verifiers pin read from the Hub
falsepass init ./my-environment           # or your own, from its folder (pyproject.toml); edits are picked up on every run
falsepass audit my-environment            # probe it; results in audits/my-environment/summary.md
falsepass results my-environment --json   # headline numbers and failing probes, for scripts and agents
```

`demo` writes the toy environment to `./falsepass-demo` and audits it. Its grader pays
when the reference appears anywhere in the answer, and the summary shows that one line
from several sides: the static scan points at it, and the probes that list several
answers (P5), retract the answer (P9) or deny it inside the answer slot (P15) are paid.
A condensed excerpt of the real output (full file: `examples/toy-env/summary.md`):

```
- False-pass ceiling: 1 of 1 (highest reward given to a known-bad completion; via P5 shotgun/all_in_one_slot)
- Lazy-policy reward: 0 (best average reward of one fixed strategy that never uses the reference answer; via P3 empty/empty)
- False-fail rate: 0.108 (93 known-good completions)

| Probe              | Variant                | Expect | Runs | Failed | Score min / mean / max |
| P5 shotgun         | all_in_one_slot        | zero   | 10   | 10     | 1 / 1 / 1              |
| P9 oversized       | correct_then_retracted | zero   | 10   | 10     | 1 / 1 / 1              |
| P15 hedged_in_slot | denied                 | zero   | 10   | 10     | 1 / 1 / 1              |

Static hint: falsepass_toy_env.py:26 reference_substring_match
    return 1.0 if answer in reply else 0.0
    confirmed: P9 failed 10/20, P5 failed 10/30
```

## Inspect AI and HUD

```bash
falsepass init inspect_evals/gsm8k                     # a task from inspect_evals (pinned from PyPI)
falsepass init my_eval.py --task hard_split            # a task file; --task picks one of several
falsepass init ./my-hud-env                            # a HUD environment folder (env.py + tasks.py)
falsepass init ./my-hud-env --answers refs.json        # with reference answers: every probe runs
```

Inspect: the scripted model replaces only the call to `generate()`; prompt templates,
system messages and answer parsing (`multiple_choice()`) are the task's own code. A
solver or scorer that calls a model itself (self-critique, `model_graded_qa`) is flagged
on every row instead of being scored. Tasks that need a sandbox are not supported.

HUD: a task template keeps its expected answer inside the grader, so without
`--answers` only answer-free probes run: an empty reply, the prompt echoed back, and
replies that list every integer from 0 to 1000, every letter, yes/no words or the
prompt's words (A1; it also runs, with every other probe, when answers are given). A
grader that pays those is a substring test in practice. The answers file is JSON: an
object from task slug to answer (`{"add-2-3": "5"}`) or a list in task order
(`["5", "12"]`). To add answers to an existing audit, rerun `init` with `--force`.
`examples/inspect-toy/` and `examples/hud-toy/` each have one planted defect to try.

## For environment authors: check before you publish

Run `falsepass init ./your-env && falsepass audit your-env` locally, or add the GitHub
Action to the environment's repository so every push is checked:

```yaml
# .github/workflows/falsepass.yml
on: [push, pull_request]
jobs:
  falsepass:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: Super-Eric/falsepass@main
        with:
          path: .                         # folder with the environment's pyproject.toml
          fail-on: "ceiling>0.1 errors>0" # fail when a known-bad answer earns over 0.1, or scoring raises
```

The step summary shows the probe table. Gate names: `ceiling` (highest reward any
known-bad completion got), `lazy` (best average reward of one fixed strategy that never
uses the answer), `false_fail` (share of correct answers refused), `errors`. Datasets that
need a login or an API key at load time cannot be audited in CI without that secret.

## For coding agents

- **MCP server.** `falsepass mcp` serves the commands as tools (`falsepass_scan`,
  `_init`, `_audit`, `_audit_status`, `_results`, `_completion`, `_repro`, `_discover`)
  over stdio. Audits run in the background, so a client's tool-call timeout never cuts
  one off; failures come back as tool errors with the reason. Any MCP client starts
  it with `uvx --from "falsepass[mcp]" falsepass mcp`;
  in Claude Code, for example: `claude mcp add falsepass -- uvx --from "falsepass[mcp]" falsepass mcp`.
- **Skill.** [skills/falsepass/SKILL.md](https://github.com/Super-Eric/falsepass/blob/main/skills/falsepass/SKILL.md) is a ready-made audit procedure in the Agent
  Skills format: how to run an audit, how to tell a defect from a design choice, and how
  to write it up for a maintainer. Copy the folder into the agent's skills directory
  (for Claude Code, `~/.claude/skills/`).
- **Plain CLI.** Every command is non-interactive and writes files; `init --json`,
  `results --json` and `scan --json` print JSON; `results --completion ID` prints one
  completion in full; `audit --fail-on ...` exits 1 when a condition is met. The output
  formats are described in [docs/OUTPUT.md](https://github.com/Super-Eric/falsepass/blob/main/docs/OUTPUT.md).

Whatever drives it, the output is a list of leads. An agent should reproduce a lead and
read the reward code before calling it a defect, and should not report an environment as
broken on the strength of a probe table.

## All commands

```bash
falsepass init owner/environment          # set up audits/<name>/ for a Hub environment, or pass a local folder
falsepass audit environment               # no adapter needed for Q&A tasks: the answer format is inferred
falsepass audit environment --search 120  # add mutation search (scorer calls per direction and example)
falsepass audit environment --jobs 1      # score one task at a time (default: up to 8 at once where that is safe)
falsepass audit environment --fail-on 'ceiling>0.1' --fail-on 'errors>0'   # CI gate, exit 1 when met
falsepass results environment [--json]    # the last run: summary.md, or a compact JSON digest
falsepass results environment --completion ID   # one completion in full (IDs are in the JSON digest)
falsepass scan path/to/source [--json]    # static pre-scan only, nothing is installed or run
falsepass discover --days 90 --limit 160  # recently maintained Hub environments and what each needs
falsepass repro environment               # stand-alone script that re-scores the exact failing completions
falsepass rollout environment -n 20       # what a local model's own replies earn (Ollama)
falsepass propose environment -n 5        # a local model writes wrong answers and climbs on the reward
falsepass draft environment               # REPORT.draft.md: candid