okareo-mcp

# Okareo MCP Server

The Okareo MCP server exposes Okareo's evaluation capabilities as MCP tools, allowing AI coding assistants to create and manage scenarios, register models, run evaluations, and execute multi-turn simulations directly from your editor.

For detailed documentation, see the [Okareo MCP docs](https://docs.okareo.com/docs/mcp/introduction).

## Prerequisites

- An Okareo account at [app.okareo.com](https://app.okareo.com)
- A copilot that supports MCP servers (Claude Code, Cursor, or VS Code)
- Python 3.10–3.12 (only for the local install modes; not needed for remote)

---

## Remote MCP (hosted) — recommended

The fastest way to get started. No local install, no Python, no API key in `.mcp.json`. Browser sign-in handles auth on first connect.

### Recommended (OAuth — Claude Code, Claude Desktop, Cursor, VS Code 1.101+)

Add to your copilot's MCP config (typically `.mcp.json`):

```json
{
  "mcpServers": {
    "okareo": {
      "type": "http",
      "url": "https://tools.okareo.com/mcp"
    }
  }
}
```

Reload the copilot. It will open a browser to Okareo sign-in once; thereafter the copilot stores the token itself. Your `.mcp.json` contains no secrets.

### Fallback (Bearer header — older clients or headless / CI)

For clients that haven't shipped the MCP OAuth flow yet, paste your API key as a bearer header. Prefer the env-var form over an inline literal:

```json
{
  "mcpServers": {
    "okareo": {
      "type": "http",
      "url": "https://tools.okareo.com/mcp",
      "headers": {
        "Authorization": "Bearer ${env:OKAREO_API_KEY}"
      }
    }
  }
}
```

### Working across multiple Okareo organizations

If you have access to more than one Okareo account, simply `Clear Authentication` and then re-authenticate with the account you want to work in. You can always ask which account your are in and have acccess to.

---

## Local install (alternative)

Run the MCP server on your own machine. Useful for offline / airgapped environments and for development.

> Prefer the **[Remote MCP](#remote-mcp-hosted--recommended)** section above unless you have a specific reason to install locally (airgapped, custom build, development on this repo). The remote endpoint requires no install and stays current automatically.

For multi-org users, the remote endpoint also exposes [tenant management tools](docs/remote-mcp.md#tenants--working-across-multiple-okareo-organizations) (`list_tenants`, `switch_tenant`).

### Step 1: Set Your API Key

```bash
export OKAREO_API_KEY="your-api-key"
```

Add this to your `~/.zshrc` or `~/.bash_profile` for persistence.

### Step 2: Configure Your Copilot

#### Claude Code

Add to `.mcp.json`:

```json
{
  "mcpServers": {
    "okareo": {
      "command": "uvx",
      "args": ["okareo-mcp"],
      "env": {
        "OKAREO_API_KEY": "${OKAREO_API_KEY}"
      }
    }
  }
}
```

No pre-install needed — `uvx` handles it automatically.

#### Cursor

Add to `.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "okareo": {
      "command": "uvx",
      "args": ["okareo-mcp"]
    }
  }
}
```

Cursor inherits `OKAREO_API_KEY` from your shell environment.

#### Alternative: pip install

If you don't have `uv` installed:

```bash
pip install okareo-mcp
```

Then use `"command": "okareo-mcp"` instead of `"command": "uvx"` with `"args": ["okareo-mcp"]`.

---

## Configuration Reference

| Variable | Default | Description |
|----------|---------|-------------|
| `OKAREO_API_KEY` | *(required)* | Your Okareo API key |
| `OKAREO_BASE_URL` | `https://api.okareo.com` | Override for on-prem Okareo backend |
| `TRANSPORT` | `stdio` | Transport: `stdio` (local) or `sse` (Docker) |
| `PORT` | `8000` | Port for SSE transport |

---

## Available Tools

### Scenarios

| Tool | Description |
|------|-------------|
| `save_scenario` | Save a named scenario from rows of input/result data (idempotent) |
| `list_scenarios` | List all scenarios in the project with names, IDs, and row counts |
| `get_scenario` | Retrieve a scenario's metadata and all data rows by name or ID |
| `create_scenario_version` | Create a new version of an existing scenario with updated data |
| `preview_delete_scenario` | Preview what will be deleted before removing a scenario |
| `delete_scenario` | Permanently delete a scenario and all related test data |

### Generation Models

| Tool | Description |
|------|-------------|
| `list_available_llms` | Browse available LLMs from the Okareo registry |
| `register_generation_model` | Register a generation model for testing by selecting an LLM from the registry |
| `list_generation_models` | List all registered generation models in the project |
| `get_generation_model` | Read detailed information about a registered generation model |
| `update_generation_model` | Change the LLM a registered generation model points to |
| `delete_generation_model` | Remove a registered generation model and all its related test data |

### Tests & Checks

| Tool | Description |
|------|-------------|
| `list_checks` | List available quality checks (built-in and custom) for evaluating model outputs |
| `run_test` | Run a quality test that evaluates a model against a scenario using specified checks |
| `list_test_runs` | List past test runs with optional filters (model, scenario, simulation-only) |
| `get_test_run_results` | Load detailed per-row results of a test run or simulation by ID or name |
| `get_conversation_transcript` | Retrieve the full conversation transcript for a single data point |
| `reevaluate_test_run` | Re-score a completed test run against a (possibly different) set of checks |
| `create_or_update_check` | Create or update a quality check by name — model-based, code-based, or audio (upsert) |
| `generate_check` | Generate a check from a natural-language description, then save it |
| `get_check` | Retrieve a check's full configuration, including its prompt template or code |
| `delete_check` | Permanently delete a check by name |
| `calibrate_check` | Dry-run a draft check against a finished test run — per-row verdicts and the arguments the check received, saving nothing |

### Simulations (Multi-Turn)

| Tool | Description |
|------|-------------|
| `create_or_update_target` | Create or update a Target — generation model, custom endpoint, or voice (OpenAI, Deepgram, Twilio) |
| `get_target` | Retrieve a Target's configuration by name (all types) |
| `list_targets` | List all simulation targets (voice and custom_endpoint) in the project |
| `delete_target` | Remove a simulation target and all its related test data |
| `create_or_update_driver` | Define a simulated user persona that will interact with your target |
| `get_driver` | Retrieve a Driver's full configuration including the persona prompt |
| `list_drivers` | List all Driver personas in the project |
| `list_driver_voices` | Discover the voices, voice profiles, and languages available for voice drivers |
| `run_simulation` | Run a multi-turn conversation evaluation (or rerun a previous one with overrides) |
| `list_simulations` | List past simulation runs with optional filters (target, scenario, limit) |

### Voice Monitoring

| Tool | Description |
|------|-------------|
| `ingest_conversations` | Submit completed voice conversations to Okareo for monitoring |
| `connect_voice_integration` | Connect a voice provider so its traffic flows into Okareo monitoring |
| `list_voice_integrations` | List the voice provider integrations in your project |
| `get_voice_integration` | Retrieve a voice provider integration by ID, including its status |
| `update_voice_integration` | Update a voice provider integration's metadata |
| `rotate_voice_integration_secret` | Rotate a voice provider integration's secrets |
| `delete_voice_integration` | Delete a voice provider integration by ID |
| `get_voice_webhook_url` | Get the inbound webhook endpoint for a voice provider |

### Projects

| Tool | Description |
|------|-------------|
| `list_projects` | List every project you can work in — id, name, tags, archive state — with the active one marked |
| `select_project` | Choose the project to work in for this conversation |
| `create_project` | Create a new project |
| `update_project` | Rename a project, or replace its tags |
| `archive_project` | Hide a project from the project picker. Reversible; nothing is deleted |
| `unarchive_project` | Restore an archived project to the picker |
| `clone_project` | Copy an existing project's scenarios into a new project (dry-run first) |

> Okareo has no project **delete** — in the app, the API, or here. Archiving is
> the removal, and it only hides the project from the picker: its scenarios,
> runs, and dashboards stay intact and every tool still works against it.

### Analytics & Dashboards

| Tool | Description |
|------|-------------|
| `query_analytics` | Query Okareo's product analytics to understand evaluation trends |
| `list_dashboards` | List the analytics dashboards in your project |
| `get_dashboard` | Retrieve a dashboard's full configuration by name |
| `save_dashboard` | Create or update an analytics dashboard by name (upsert) |
| `reorder_dashboards` | Set the display order of dashboards |
| `delete_dashboard` | Delete a dashboard by name |

### Tenant Management (remote MCP only)

| Tool | Description |
|------|-------------|
| `list_tenants` | List every Okareo organization you have access to in this MCP session |
| `switch_tenant` | Change which Okareo organization subsequent tool calls operate against |

### Documentation & Templates

| Tool | Description |
|------|-------------|
| `get_docs` | Query the Okareo documentation system for conceptual or user-legible explanations |
| `get_templates` | Retrieve prompt templates for common Okareo patterns (works offline) |
| `get_reps_baseline` | Serve REPS agent-evaluation baseline material (scenario banks, drivers, checks, eval configs) from the latest tagged okareo-tools release |

---

## Troubleshooting

| Symptom | Cause | Fix