mcp-compliance

# @yawlabs/mcp-compliance

[![npm version](https://img.shields.io/npm/v/@yawlabs/mcp-compliance)](https://www.npmjs.com/package/@yawlabs/mcp-compliance)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](https://opensource.org/licenses/MIT)
[![GitHub stars](https://img.shields.io/github/stars/YawLabs/mcp-compliance)](https://github.com/YawLabs/mcp-compliance/stargazers)

**Test any MCP server for spec compliance.** 88-test suite covering transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security against the [MCP specification](https://modelcontextprotocol.io/specification/2025-11-25). Works against **HTTP endpoints** (`https://my-server.com/mcp`) and **stdio servers** (`npx @modelcontextprotocol/server-filesystem /tmp`) alike. CLI, MCP server, and programmatic API.

Built and maintained by [Yaw Labs](https://yaw.sh).

[![Add to Yaw MCP](https://yaw.sh/yaw-mcp-button.svg)](https://yaw.sh/mcp/install?name=mcp-compliance&command=npx&args=-y%2C%40yawlabs%2Fmcp-compliance&description=Test%20any%20MCP%20server%20against%20the%20spec%20-%2088-test%20suite%20with%20letter-grade%20scoring&source=https%3A%2F%2Fgithub.com%2FYawLabs%2Fmcp-compliance)

One click adds this to your local Yaw MCP config so it's available in every Yaw Terminal session. Or install manually below.

## Why this tool?

MCP servers are multiplying fast — but most ship without compliance testing. Broken transport handling, missing error codes, malformed schemas, and silent capability violations are common. Hand-rolling test scripts is tedious and incomplete.

This tool solves that:

- **88 tests across 8 categories** — transport, lifecycle, tools, resources, prompts, error handling, schema validation, and security. No gaps. (HTTP runs all 85 transport-applicable tests; stdio runs ~75 — HTTP-specific tests like CORS, TLS, session headers, and rate limiting are gated out.)
- **Capability-driven** — tests adapt to what the server declares. If it says it supports tools, tool tests become required. No false failures for features the server doesn't claim.
- **Graded scoring** — A-F letter grade with a weighted score (required tests 70%, optional 30%). One number to communicate compliance.
- **CI-ready** — `--strict` mode exits with code 1 on required test failures. Drop it into any pipeline.
- **Spec-referenced** — every test links to the exact section of the MCP specification it validates. No ambiguity about what's being tested or why.
- **Three interfaces** — CLI for humans, MCP server for AI assistants, programmatic API for integration.
- **Published methodology** — the [testing methodology](./COMPLIANCE_RUBRIC.md) and [rule catalog](./mcp-compliance-rules.json) are open (CC BY 4.0) so anyone can build compatible tooling or fork the rules.

## Quick start

**Remote HTTP server:**

```bash
npx @yawlabs/mcp-compliance@latest test https://my-server.com/mcp
```

**Local stdio server** (the vast majority of MCP servers on npm):

```bash
# Pass the command directly, Inspector-style
npx @yawlabs/mcp-compliance@latest test npx @modelcontextprotocol/server-filesystem /tmp

# Or a local build
npx @yawlabs/mcp-compliance@latest test node ./dist/server.js

# With env vars
npx @yawlabs/mcp-compliance@latest test -E GITHUB_TOKEN=$GITHUB_TOKEN -- npx @modelcontextprotocol/server-github
```

**Install globally:**

```bash
npm install -g @yawlabs/mcp-compliance
mcp-compliance test https://my-server.com/mcp
```

That's it. You'll get a colored terminal report with a letter grade (A-F), per-test pass/fail, and a compliance score.

## CLI usage

### HTTP targets

```bash
# Terminal output with colors and grade
mcp-compliance test https://my-server.com/mcp

# JSON / SARIF for scripting + GitHub Code Scanning
mcp-compliance test https://my-server.com/mcp --format json
mcp-compliance test https://my-server.com/mcp --format sarif > compliance.sarif

# Strict mode for CI — exits 1 on required-test failure
mcp-compliance test https://my-server.com/mcp --strict

# Auth (shorthand or full header)
mcp-compliance test https://my-server.com/mcp --auth "Bearer tok123"
mcp-compliance test https://my-server.com/mcp -H "X-Api-Key: abc"

# Focus the run
mcp-compliance test https://my-server.com/mcp --only transport,lifecycle
mcp-compliance test https://my-server.com/mcp --skip prompts,resources
mcp-compliance test https://my-server.com/mcp --verbose
```

### stdio targets

Pass the command and its args as positional arguments (MCP Inspector-style). Use `--` to disambiguate when the target needs flags that collide with ours.

```bash
# npm-distributed stdio servers
mcp-compliance test npx -y @modelcontextprotocol/server-filesystem /tmp
mcp-compliance test uvx mcp-server-git

# Local build
mcp-compliance test node ./dist/server.js

# With env vars (repeatable -E, or --env-file)
mcp-compliance test -E API_KEY=secret -E REGION=us-east-1 -- npx my-server
mcp-compliance test --env-file .env -- node ./server.js

# Set working directory
mcp-compliance test --cwd ./services/mcp -- node ./dist/server.js

# Target uses a flag that collides with ours — use `--` to separate
mcp-compliance test --verbose -- node ./server.js --verbose
```

On Windows, `npx` and other `.cmd` shims are handled automatically by spawning through the shell.

### Options

| Option | Applies to | Description |
|--------|-----------|-------------|
| `--format <format>` | both | Output format: `terminal`, `json`, `sarif`, `github`, or `markdown` (default: `terminal`) |
| `--config <path>` | both | Load defaults from a config file (default: `mcp-compliance.config.json` in cwd) |
| `--output <file>` | both | Write a local SVG badge to the given path after the run |
| `--list` | both | Print test IDs that would run given current filters, then exit (no connection) |
| `--transport <kind>` | both | Filter by `http` or `stdio` (only used with `--list` when no target is provided) |
| `--strict` | both | Exit with code 1 on any required test failure (for CI) |
| `--min-grade <grade>` | both | Exit with code 1 if grade is below this threshold (`A`–`F`) |
| `-H, --header <h>` | HTTP | Add header to all requests, format `"Key: Value"` (repeatable) |
| `--auth <token>` | HTTP | Shorthand for `-H "Authorization: <token>"` |
| `-E, --env <var>` | stdio | Set env var for stdio command, format `"KEY=VALUE"` (repeatable) |
| `--env-file <path>` | stdio | Load env vars from a file (one `KEY=VALUE` per line) |
| `--cwd <dir>` | stdio | Working directory for the stdio command |
| `--timeout <ms>` | both | Request timeout in milliseconds (default: `15000`) |
| `--preflight-timeout <ms>` | HTTP | Preflight connectivity check timeout (HTTP only) |
| `--retries <n>` | both | Number of retries for failed tests (default: `0`) |
| `--only <items>` | both | Only run tests matching these categories or test IDs (comma-separated) |
| `--skip <items>` | both | Skip tests matching these categories or test IDs (comma-separated) |
| `--concurrency <n>` | both | Max parallel-safe tests in flight (default: `1`; raising reduces wall time but can perturb timing-sensitive servers) |
| `--verbose` | both | Print each test result as it runs (also forwards stdio stderr) |

### CI integration

**GitHub Action** (drop into any `.github/workflows/*.yml`):

```yaml
- uses: YawLabs/mcp-compliance@v0
  with:
    target: 'node ./dist/server.js'   # or a URL like https://my-server.com/mcp
    format: github                     # ::error / ::warning annotations on the PR
    strict: 'true'                     # exit non-zero if any required test fails
    min-grade: 'A'                     # also exit if grade slips
```

**Manual CLI invocation:**

```bash
# GitHub Actions: emits ::error / ::warning annotations inline on the PR
mcp-compliance test https://my-server.com/mcp --format github --strict

# Slack/Linear/PR comment: drop the body straight into a comment
mcp-compliance test https://my-server.com/mcp --format markdown > report.md

# HTML report (self-contained, share anywhere — issue comments, S3, GitHub Pages)
mcp-compliance test https://my-server.com/mcp --format html > report.html

# Block release if grade slips below B
mcp-compliance test https://my-server.com/mcp --min-grade B

# Preview which tests will run before connecting (handy for --only/--skip authoring)
mcp-compliance test --list --transport stdio --skip security

# Diff two runs — exit 1 if anything that was passing is now failing
mcp-compliance test https://my-server.com/mcp --format json > current.json
mcp-compliance diff baseline.json current.json

# Watch mode for stdio dev loop — re-runs on file changes in cwd
mcp-compliance test --watch -- node ./dist/server.js

# Latency benchmark
mcp-compliance benchmark -- node ./dist/server.js -r 200 -c 4
```

**Docker:**

```bash
docker run --rm ghcr.io/yawlabs/mcp-compliance test https://my-server.com/mcp
```

### Scaffold a config

```bash
mcp-compliance init
```

Interactive prompts walk you through transport (http/stdio), command/url, env vars, timeout, and strict mode — then write a `mcp-compliance.config.json` you can commit.

### Config file

Check in a `mcp-compliance.config.json` so CI and your dev loop can run `mcp-compliance test` with no arguments. Supported locations (searched in order): `mcp-compliance.config.json`, `.mcp-compliancerc.json`, `.mcp-compliancerc`, and the `"mcp-compliance"` field of `package.json`. Pass `--config <path>` to load an explicit file.

**HTTP:**

```json
{
  "target": {
    "type": "http",
    "url": "https://my-server.com/mcp",
    "headers": { "Authorization": "Bearer tok123" }
  },
  "timeout": 20000,
  "strict": true
}
```

**stdio:**

```json
{
  "target": {
    "type": "stdio",
    "command": "node",
    "args": ["./dist/server.js"],
    "env": { "LOG_LEVEL": "error" }
  },
  "skip": ["security"],
  "strict": true
}
```

Precedence: CLI flags > config file > defaults. Any field can be overridden on the command line.

### Local SVG badge

Write a local SVG reflec