autopia

# Autopia

Autopia empirically tests HTTP APIs that ship an OpenAPI spec, discovers how they
**actually** behave, verifies every finding by reproducing it, and serves the result as a
versioned, queryable knowledge base that AI agents consult over MCP, and that a Claude Code
plugin injects into an agent's context at the moment it is about to call one of those APIs.

It is the MVP of the "Discrimen Compendium" described in `docs/compendium-spec.txt`
(the blueprint) and `docs/compendium-stack.txt` (stack choices). Read the spec's glossary
first; the code uses those exact terms. `HANDOFF.txt` holds the running state and decisions.

**Live:** the hosted server is at `https://autopia-mcp.onrender.com/mcp` (per-user keys, see
[Identity](#identity-per-user-keys)); the database is a managed Postgres on Neon; a GitHub
Actions schedule refreshes every target nightly (rules only) and weekly (with the model pass).
The catalogue currently holds 16 APIs, including GitHub REST, the npm registry, PyPI, the
Anthropic models endpoints, the Open-Meteo family, weather.gov, Binance, Frankfurter,
Open Library, PokeAPI and Nager.Date.

## What an agent gets

A **behavior sheet** per API: Markdown generated from verified claims only, ordered by value.
Where the live API contradicts its own spec comes first, then rules the spec never states,
then behavior common to every endpoint, then per endpoint: what responses look like,
error-body shapes, pagination, validation, auth. Every line ends with the trial ids that back
it. Examples of what the sheets record, all reproduced on the live APIs:

- GitHub silently applies `since=not-a-date` on commits as a filter that matches nothing
  (200, empty list), while issues rejects the same value with a 422 that spells out the format.
- GitHub accepts a missing token on public reads but serves the request from the 60/hour
  tier instead of 5000/hour; `per_page` is clamped to 100 (proved by the Link header).
- npm search accepts `size=2500` against a documented maximum of 250, and does not type-check
  its score-weight parameters.
- The Anthropic `/v1/models` cursors must name an existing model id; an unknown id is a 400
  as a cursor but a 404 as a path.

## The pipeline

```
discover -> exercise -> aggregate -> cross-ref -> verify -> write (+ diff)
 spec       TRIALS      rules +      spec +       evidence   CLAIMS
 fetch,     (no LLM)    shape rules  siblings +   check +    + DIFFS
 hash,                  + one LLM    self-        reproduce
 trim                   pass         consistency
```

- **Trials** are pinned requests generated by the **battery** (`src/worker/battery`,
  version 1.3.0), a deterministic catalogue of edge cases: omit required, wrong type, enum
  outside the documented set, null and empty values, numeric boundaries (zero, negative,
  max int, below the minimum, just above and far above the maximum, 2048-character strings),
  duplicate parameters, unknown fields, bad and missing credentials (scheme-preserving), page
  zero, a huge page size, a page beyond the last, four date formats for date-time parameters
  (date only, no timezone, epoch, garbage), wrong content type, trailing slash, HEAD. No
  randomness anywhere, so two runs produce identical trial inputs. Every stored response keeps
  the lengths of its arrays before truncation, so list semantics survive storage.
- **Claims** come from three sources: deterministic **rules** over the trials (free),
  **shape rules** over successful responses (content type, timestamp formats, null-valued
  fields, pagination fields, rate-limit and cache headers), and **one** batched LLM pass for
  open-ended discovery. The LLM output is validated with ajv against
  `src/schemas/claim.schema.json`; invalid objects are discarded. Rules classify what came
  back, not just the status: a 200 on a bad date value is "applied as an impossible filter"
  when the list is empty where the comparison request had rows, "apparently ignored" when the
  counts match, "interpreted" otherwise. A page-size cap is claimed only when more items
  provably exist (a Link `rel="next"`, a total field, or the documented maximum reached).
  Error-shape claims count only probes that sent a bad value, not auth, HEAD or content-type
  probes.
- **Verification** is the product: every claim cites trial ids and the status each showed.
  The evidence check is mechanical; reproduction re-runs the exact inputs and compares a
  normalized response **signature** (status, body kind, JSON shape, error message), never
  raw bytes, so live data can change without breaking reproduction. Non-reproducing claims
  are discarded. Stored claims are always `verified = true`.
- **Divergences** are claims that contradict a documented fact (required flag, type, enum,
  range). The money view. Silence in the spec is never a divergence: a rule the API enforces
  that the spec never states (parameters that must be sent together, implicit defaults) is
  stored under its own kind, `undocumented_constraint`.
- **Claim kinds:** `required`, `type`, `auth`, `pagination`, `error_behavior`,
  `response_shape`, `undocumented_constraint`, `undocumented`, `other`.
- **Sibling comparison.** Tools that share a `category` are compared probe by probe: the
  same rule on the same parameter name with the opposite verdict marks the claim
  `divergence_type = sibling` with a note naming the other tool.
- **Carry-forward.** Rules-only runs re-admit the previous run's LLM claims as candidates,
  remapped to tonight's trials by endpoint and probe, and send them through the same evidence
  check and reproduction. A carried claim is dropped when a fresh LLM claim on the same
  endpoint cites the same probes (the model restates findings in new words), and carried
  claims that duplicate each other keep the more confident one.
- **Diffs** compare claim sets across tool versions by stable `claim_key`. A tool gets a new
  version whenever its spec hash changes; large specs are stored trimmed to the exercised
  paths, with the hash still over the full document.

## Quick start

```bash
npm install
npm run autopia -- run targets/open-meteo-forecast.json     # full pipeline (LLM pass if a key is set)
npm run autopia -- docs open-meteo-forecast                 # the behavior sheet
npm run autopia -- claims open-meteo-forecast --divergences
npm run autopia -- search "is latitude required"
npm test
```

Configuration lives in a local `.env` file (gitignored), read on startup:

| Variable | Purpose |
|---|---|
| `ANTHROPIC_API_KEY` | enables the LLM pass; without it the pipeline runs rules-only and says so. Also the credential for the `anthropic-api` target |
| `AUTOPIA_MODEL` | model for the LLM pass, default `claude-opus-5-5` |
| `DATABASE_URL` | Postgres connection string; unset = embedded PGlite in `./data/pglite` |
| `GITHUB_TOKEN` | credential for the `github-rest` target (a fine-grained, public-read-only token is enough) |
| `AUTOPIA_MCP_TOKEN` | admin token for the HTTP MCP server (per-user keys work without it) |
| `AUTOPIA_MCP_OPEN` | `1` opens `/mcp` without any credential; tests only, never in a deployment |
| `AUTOPIA_MCP_PORT` | HTTP server port, default 4200 |
| `AUTOPIA_LOG` | `debug` for per-trial logs |

Target files reference credentials as `${ENV_VAR}` in headers, so a committed target never
contains a key.

**PGlite caveat:** the default database is an embedded Postgres (PGlite) under `data/pglite`
and only one process can open it at a time. Stop the worker before starting the MCP server
or dashboard. Set `DATABASE_URL` to a real Postgres to lift that limit (same schema, no code
change). `npx tsx scripts/whichdb.ts` says which one a command would use.

## Serving the compendium

```bash
npm run mcp          # MCP server over stdio (register in Claude Code / any MCP client)
npm run mcp:http     # the same server over Streamable HTTP at http://localhost:4200/mcp
npm run dashboard    # read-only dashboard at http://localhost:4100 (local only; not deployed)
```

Register the stdio server with a client as `npx tsx <repo>/src/server/mcp.ts`, or point an
HTTP-capable client at `/mcp` on the HTTP server (`GET /healthz` for liveness). The server
ships an **instructions block** telling an agent when to consult it: before its first request
in a session to a covered host, and after an API call fails or surprises it.

| MCP tool | Use |
|---|---|
| `get_tool_docs` | the behavior sheet; call it before the first request to a covered API. `brief: true` for divergences, undocumented rules, auth, pagination and errors only |
| `search_claims` | the error path: full-text search with host + symptom, e.g. "github per_page 500" |
| `get_divergences` | only the claims where the live API contradicts its spec |
| `get_tool_knowledge` | every verified claim as structured records, paginated, filter by kind |
| `list_tools` | the catalogue, each tool with the **hosts** its trials hit |
| `list_versions`, `get_version_diff` | spec versions, and what changed in behavior between two |
| `request_target`, `get_request_status` | ask for an API that is not covered yet (needs a key) |

The `tool` argument of every tool accepts a tool name, a host, or the URL you are about to
call (`api.github.com`, `https://registry.npmjs.org/-/v1/search`). The compendium is a
directory, not a dump: `get_tool_docs` with a URL or `path` returns **only the endpoint that
path matches** plus a one-line index of the tool's other endpoints, so an agent descends to
what it needs instead of receiving everything. Every sheet ends with a coverage section: which
endpoints of this API are probed, which other APIs exist (by host), what is not probed anywhere
yet, and the tools to fetch more, so a reader of one sheet knows what else it can ask for. Evidence is compact (trial
id, probe, status, request line) unless `verbose=true` asks for response previews. Stored
request inputs are redacted before storage, so no credential can be served as evidence;
`npx tsx scripts/scan-st