validate

# machinegrade validate

Validate AI-generated artifacts against a contract before you act on them:

- **`json_schema`** — validate `artifact` against a JSON Schema (all errors collected).
- **`openapi_response`** — validate a response body against the response schema for a given `path` + `method` + `status` in an OpenAPI spec.
- **`sql`** — check a SQL string for syntax errors in a given dialect.

Every check returns a **verdict**, not an error: `{valid, errors, latency_ms}`,
HTTP 200 whether the artifact is valid or not. Only genuinely wrong requests
(bad key, unsupported type, malformed body, over your limit) get typed HTTP
errors.

Live at **https://api.machinegrade.dev** — free tier, self-service key,
try it in 30 seconds (first example below). Built on Hono; the same
codebase runs on Cloudflare Workers (production) and plain Node (local
dev), and is MIT-licensed if you'd rather self-host.

## Why

Agents that generate JSON, API responses, or SQL need a fast, cheap,
machine-checkable pass/fail before they ship the result — cheaper than a
full LLM-as-judge call, and deterministic.

## Run it locally

```bash
npm install
npm run dev
# machinegrade validate listening on http://localhost:8787
```

## 3 runnable examples

### 1. curl

```bash
# Get an API key (live service — works as-is)
curl -s -X POST https://api.machinegrade.dev/keys \
  -H 'content-type: application/json' \
  -d '{"email": "you@example.com"}'
# => {"key":"sk_..."}

# Validate a JSON artifact against a JSON Schema
curl -s -X POST https://api.machinegrade.dev/v1/validate \
  -H 'content-type: application/json' \
  -H 'X-Api-Key: sk_...' \
  -d '{
    "type": "json_schema",
    "artifact": {"name": "Ada", "age": 30},
    "contract": {
      "schema": {
        "type": "object",
        "required": ["name", "age"],
        "properties": {"name": {"type": "string"}, "age": {"type": "number"}}
      }
    }
  }'
# => {"valid":true,"errors":[],"latency_ms":1}
```

### 2. Python (requests)

```python
import requests

base = "http://localhost:8787"

key = requests.post(f"{base}/keys", json={"email": "you@example.com"}).json()["key"]

resp = requests.post(
    f"{base}/v1/validate",
    headers={"X-Api-Key": key},
    json={
        "type": "sql",
        "artifact": "SELECT id, name FROM users WHERE id = 1",
        "contract": {"dialect": "mysql"},
    },
)
print(resp.status_code, resp.headers.get("X-Calls-Remaining"), resp.json())
```

### 3. MCP config snippet

`mcp/server.ts` exposes a single tool, `validate`, that forwards to
`POST /v1/validate`. Point an MCP-compatible client at it:

```json
{
  "mcpServers": {
    "machinegrade-validate": {
      "command": "npx",
      "args": ["tsx", "mcp/server.ts"],
      "cwd": "/path/to/validate",
      "env": {
        "SANDBOX_URL": "http://localhost:8787",
        "SANDBOX_API_KEY": "sk_..."
      }
    }
  }
}
```

## Connect remotely

The production service also exposes an MCP endpoint directly — no local
process, no npm install — via streamable HTTP at:

```
POST https://api.machinegrade.dev/mcp
```

It's the same single `validate` tool as the stdio adapter above.
`initialize` and `tools/list` work without a key (discovery is
anonymous); `tools/call` requires `X-Api-Key` (issue one via `POST
/keys`, same as the REST API — the free tier and limits are shared).

With Claude Code:

```bash
claude mcp add --transport http validate https://api.machinegrade.dev/mcp --header "X-Api-Key: sk_..."
```

The stdio adapter via npm (`@machinegrade/validate`, see above) remains
available for local/offline use or clients without HTTP transport
support.

## Claude Desktop: one-click install

Download the latest `.mcpb` bundle from
[Releases](https://github.com/machinegrade/validate/releases/latest) and
double-click it. Claude Desktop asks for an API key during install and stores it
in the OS keychain (macOS Keychain, Windows Credential Manager) rather than in a
config file — nothing lands in `claude_desktop_config.json`.

Issue a free key (500 calls/month) first:

```bash
curl -s -X POST https://api.machinegrade.dev/keys \
  -H 'content-type: application/json' \
  -d '{"email": "you@example.com"}'
```

The bundle is a thin stdio client over the hosted API: ~34 KB, zero bundled
dependencies. If you self-host, point the extension's **API base URL** setting at
your own deployment and the tool talks to that instead.

## API

See [`public/openapi.yaml`](./public/openapi.yaml) for the full contract, or
[`/v1/manifest`](http://localhost:8787/v1/manifest) for a machine-readable
summary (types, limits, pricing, error codes) once the service is running.
[`/llms.txt`](./public/llms.txt) is a short pointer for LLM agents.

| Endpoint | In | Out |
|---|---|---|
| `POST /keys` | `{email}` | `{key}` |
| `POST /v1/validate` | header `X-Api-Key`, body `{type, artifact, contract?}` | verdict, header `X-Calls-Remaining` |
| `GET /v1/manifest` | — | capability manifest |
| `GET /stats` | header `X-Admin-Token` | funnel: keys_issued, active_callers, repeat_callers_7d, limit_hits, paid_requests |
| `POST /v1/paid-request` | header `X-Api-Key` | records interest in paid access |
| `GET /openapi.yaml`, `GET /llms.txt` | — | static docs |
| `POST /mcp` | MCP streamable HTTP, header `X-Api-Key` for `tools/call` | see "Connect remotely" above |

## Pricing

- **Free tier:** 500 calls/month per key, 60 calls/minute rate limit.
- **Paid tier:** EUR 0.002/call beyond the free tier — **opens soon**.
  Request paid access via `POST /v1/paid-request` (requires `X-Api-Key`);
  you'll be notified when it's live.

## Errors

Every error is typed JSON — `{code, message, hint, docs_url}` — never a
free-form string:

| Code | HTTP status | When |
|---|---|---|
| `INVALID_KEY` | 401 | `X-Api-Key` missing or unknown |
| `LIMIT_EXCEEDED` | 402 | Free-tier monthly limit (500 calls) exceeded |
| `UNSUPPORTED_TYPE` | 400 | `type` is not `json_schema`, `openapi_response`, or `sql` |
| `MALFORMED_INPUT` | 400 | Request body doesn't match the documented shape |
| `RATE_LIMITED` | 429 | More than 60 calls/minute for a key |

A **verdict** (`{valid, errors, latency_ms}`) is never an error — an
invalid artifact is a normal, expected outcome and returns HTTP 200.

## Sending the artifact

`artifact` must be a JSON **value**, not a JSON-encoded string:

```jsonc
{"type": "json_schema", "artifact": {"name": "Ada"}, "contract": {"schema": {"type": "object"}}}   // correct
{"type": "json_schema", "artifact": "{\"name\": \"Ada\"}", "contract": {"schema": {"type": "object"}}}  // wrong
```

For `type: "sql"` the artifact *is* a string — the statement itself.

Because MCP callers stringify values often enough (and did so through Claude
Desktop until the tool schema declared `artifact`'s types), the service
tolerates the wrong form narrowly: if `artifact` is a string, the type is
`json_schema` or `openapi_response`, and the contract's **top-level schema
declares types that exclude `string`**, the string is JSON-decoded before
validation and the verdict carries `"decoded_from_string": true`. The field is
additive; `{valid, errors, latency_ms}` is unchanged.

It deliberately does not decode otherwise, because a string artifact is often
legitimate:

| Sent | Contract schema | Result |
|---|---|---|
| `"42"` | `{"type": "number"}` | decoded to `42`, `valid: true`, `decoded_from_string: true` |
| `"42"` | `{"type": "string"}` | left alone, `valid: true` |
| `"{\"a\":1}"` | `{"type": "string"}` | left alone, `valid: true` |
| `"{\"a\":1}"` | `{"required": ["a"]}` | left alone — no top-level `type`, intent unknown |

## Storage

`src/storage.ts` defines a `Storage` interface with two implementations:

- `MemoryStorage` — full in-memory implementation, used for `npm run dev`
  and the test suite.
- `D1Storage` — real Cloudflare D1 binding, backed by `schema.sql` (`keys`,
  `events` tables). Used in production; the Workers entry point in
  `src/index.ts` builds it from the `DB` binding on first request.

Apply `schema.sql` to a new D1 database with:

```bash
wrangler d1 execute machinegrade-validate-db --file=schema.sql          # local
wrangler d1 execute machinegrade-validate-db --file=schema.sql --remote # production
```

## Testing

```bash
npm test          # vitest run, in-process via app.request(), MemoryStorage
npm run typecheck # tsc --noEmit
```

Tests cover: key issuance, happy + fail cases for each validator, typed
401/400/402/429 errors, the metering limits (both injectable in tests so
they don't require looping hundreds of real requests), and `/stats` funnel
counts.

## Deploy

The service runs on Cloudflare Workers (Hono + D1 + Workers Static
Assets). To self-host on a fresh Cloudflare account:

```bash
wrangler d1 create machinegrade-validate-db   # copy the returned database_id into wrangler.toml
wrangler d1 execute machinegrade-validate-db --file=schema.sql --remote
wrangler secret put ADMIN_TOKEN
wrangler deploy
```

Then bind a custom domain (e.g. `api.machinegrade.dev`) to the Worker via
the Cloudflare dashboard or `wrangler`. CI can deploy on push to `main` once
`CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID` repo secrets are set and
the `deploy` job in `.github/workflows/ci.yml` is uncommented.

Two things worth knowing about the Workers port:

- `GET /openapi.yaml` and `GET /llms.txt` are served by the `ASSETS` binding
  (`[assets]` in `wrangler.toml`, pointing at `public/`) — Cloudflare serves
  them directly, without invoking the Worker. The routes in `src/index.ts`
  are a fallback for local Node dev/tests, where there's no ASSETS binding.
- The `json_schema` and `openapi_response` validators use
  `@cfworker/json-schema`, not `ajv`: ajv compiles schemas via
  `new Function(...)`, which the Workers runtime disallows, and schemas
  here arrive dynamically per request (from the caller), so they can't be
  precompiled at build time either.

## Status

Early stage, honestly so: this service is live and free-tier usage is real,
and we're measuring whether