agent-observability

# agent-observability

> Open source agent observability — see what your agents did, why they failed, and what it cost. Runs locally. No cloud required.

## Quick Start

One command. No manual config. No copy-paste.

```bash
npx agent-obs@latest setup
```

This auto-detects your AI agent (opencode, Claude Code, Cursor) and configures everything:
- Adds agent-obs to your MCP config
- Adds the self-reporting instruction
- Tells you what to do next

Then restart your agent. Every action will now self-report.

```bash
agent-obs dashboard    # http://localhost:9400 — see your sessions appear
```

### What you'll see

After running a task in your agent, the dashboard shows:
- Every tool call with duration and status
- A-F session grade (how efficient was your agent?)
- Cost per session in tokens and dollars
- Full audit trail of what was read, edited, and executed

### Manual setup (if `setup` can't detect your agent)

**opencode:** Add to `opencode.json`:

```json
{
  "mcp": {
    "agent-obs": {
      "command": "npx",
      "args": ["-y", "agent-obs@latest", "server"],
      "enabled": true
    }
  }
}
```

**Claude Code:** Create `.mcp.json`:

```json
{
  "mcpServers": {
    "agent-obs": {
      "command": "npx",
      "args": ["-y", "agent-obs@latest", "server"]
    }
  }
}
```

**Cursor:** Add to Cursor Settings > MCP:

```json
{
  "mcpServers": {
    "agent-obs": {
      "command": "npx",
      "args": ["-y", "agent-obs@latest", "server"]
    }
  }
}
```

See [Agent-Specific Setup](#agent-specific-setup) below for detailed per-platform instructions including self-reporting directives.

> **Note:** All data is written to a local SQLite database in `~/.agent-observability/`. No data leaves your machine.

### Proxy mode (fallback)

For agents that can't self-report, wrap any MCP server and every tool invocation gets traced automatically:

```bash
agent-obs proxy --desc "fix login bug" -- npx @modelcontextprotocol/server-filesystem /tmp
```

Proxy mode captures only MCP tool calls (~30% of typical agent actions). Prefer the `setup` / MCP server approach above.

## Agent-Specific Setup

### Claude Code

1. **MCP Config** — Create `.mcp.json` in your project root:

```json
{
  "mcpServers": {
    "agent-obs": {
      "command": "npx",
      "args": ["-y", "agent-obs@latest", "server"]
    }
  }
}
```

2. **Self-reporting instruction** — Create `.claude/instructions.md` (or reference the existing SKILL.md under `.claude/skills/agent-obs/SKILL.md`):

```markdown
After every tool call, report to agent-obs's log_tool_call.
Start each task with start_session. End with end_session.
```

3. **Verify** — Run `agent-obs dashboard`, open http://localhost:9400, and look for your session after the agent completes a task.

### Cursor

1. **MCP Config** — Add to Cursor's MCP settings (`~/.cursor/mcp.json`, or Settings → MCP → Add new global MCP server):

```json
{
  "mcpServers": {
    "agent-obs": {
      "command": "npx",
      "args": ["-y", "agent-obs@latest", "server"]
    }
  }
}
```

2. **Self-reporting instruction** — Create `.cursorrules` (or `.cursor/rules/agent-obs.md`):

```
After every tool call, you MUST call the agent-obs MCP server's log_tool_call tool.
Include toolName, status (success/error), outputSummary, and durationMs.
Start each session with start_session and end with end_session.
```

3. **Verify** — Run `agent-obs dashboard`, open http://localhost:9400, and look for your session after the agent completes a task.

## Architecture

```
                   ┌──────────────────────────┐
                   │   agent-obs server       │
                   │    (self-reporting)      │
                   │                          │
                   │ Agent calls obs_record_* │
                   │ tools directly via MCP   │
                   │ • tool calls             │
                   │ • token usage            │
                   │ • decisions              │
                   │ • grades                 │
                   └──────────┬───────────────┘
                              │
                              ▼
                   ┌──────────────────────────┐
                   │     agent-obs proxy      │
                   │ (fallback — MCP-only)    │
                   │                          │
MCP Client ───────▶│ intercepts tool calls ──▶│ MCP Server
  (Claude/Cursor)   │ logs to SQLite           │  (filesystem,
                   └──────────┬───────────────┘   github, etc.)
                              │
                              ▼
                   ┌──────────────────────────┐
                   │   ~/.agent-observability │
                   │       SQLite DB          │
                   │                          │
                   │  • sessions              │
                   │  • tool_calls            │
                   │  • token_usage           │
                   │  • decisions             │
                   │  • grades                │
                   └──────────┬───────────────┘
                              │
                              ▼
                   ┌──────────────────────────┐
                   │   agent-obs dashboard    │
                   │     (port 9400)          │
                   │                          │
                   │  GET  /api/sessions      │
                   │  GET  /api/sessions/:id  │
                   │  GET  /api/search?q=     │
                   │  GET  /api/stats         │
                   │  POST /api/export        │
                   └──────────────────────────┘
```

## API Reference

### REST Endpoints (Dashboard)

| Method | Path | Description |
|--------|------|-------------|
| `GET` | `/api/sessions` | List all sessions. Query params: `?limit=20&offset=0&grade=B` |
| `GET` | `/api/sessions/:id` | Get full session detail with all tool calls, tokens, decisions, and grades |
| `GET` | `/api/sessions/:id/tool-calls` | List tool calls for a session. Query params: `?status=error&server=filesystem` |
| `GET` | `/api/sessions/:id/tokens` | Get token usage history for a session |
| `GET` | `/api/sessions/:id/decisions` | Get decision points for a session |
| `GET` | `/api/search` | Full-text search across tool call inputs/outputs. Query param: `?q=read_file` |
| `GET` | `/api/stats` | Aggregate statistics: total sessions, avg grade, total tokens, total cost |
| `POST` | `/api/export` | Export session data as JSON. Body: `{ "sessionIds": ["abc123"], "format": "json" }` |
| `GET` | `/api/health` | Health check. Returns `{ "status": "ok", "dbSize": "2.4MB", "sessionCount": 47 }` |

### MCP Tools (Server Mode)

| Tool | Parameters | Returns |
|------|-----------|---------|
| `obs_record_tool_call` | `sessionId`, `toolName`, `serverName`, `duration` (ms), `status` (success/error), `input`, `output` | `{ "id": "call-uuid", "recorded": true }` |
| `obs_record_token_usage` | `sessionId`, `inputTokens`, `outputTokens`, `model` | `{ "totalTokens": 1850, "estimatedCost": "$0.023" }` |
| `obs_record_decision` | `sessionId`, `context`, `options`, `chosen`, `reasoning` | `{ "id": "decision-uuid", "recorded": true }` |
| `obs_record_grade` | `sessionId`, `grade` (A/B/C/D/F), `reasoning` | `{ "grade": "B", "recorded": true }` |
| `obs_get_session_report` | `sessionId` | Full session JSON with all calls, tokens, decisions, grade |
| `obs_list_sessions` | `limit`, `offset` | Array of `{ id, description, grade, createdAt, tokenTotal, estimatedCost }` |

## What Gets Tracked

### Every tool call
- **Tool name** — e.g. `read_file`, `execute_command`, `search_code`
- **Server** — which MCP server handled it
- **Duration** — wall-clock milliseconds
- **Status** — `success` or `error`
- **Input** — full arguments passed to the tool
- **Output** — full result returned (truncated at 64KB for storage)

### Token consumption
- **Input tokens** — prompt and context
- **Output tokens** — generated response
- **Total tokens** — sum per session
- **Per-call breakdown** — token delta for each LLM round-trip

### Cost estimation
Costs are estimated based on published API pricing:

| Model | Input (per 1M tokens) | Output (per 1M tokens) |
|-------|----------------------|------------------------|
| Claude 3.5 Sonnet | $3.00 | $15.00 |
| Claude 3 Opus | $15.00 | $75.00 |
| GPT-4o | $2.50 | $10.00 |
| GPT-4 Turbo | $10.00 | $30.00 |

Costs are tracked per session and displayed in the dashboard. You can add custom model pricing via `~/.agent-observability/models.json`.

### Session grades
Every session receives a letter grade based on efficiency and correctness:

| Grade | Label | Criteria |
|-------|-------|----------|
| **A** | Clean | Zero errors, minimal token waste, no unnecessary tool calls |
| **B** | Minor issues | Some inefficiencies, but no failures |
| **C** | Inefficient | Excessive token usage, redundant tool calls, recoverable errors |
| **D** | Risky | Significant problems — failed tool calls, high cost, wrong tools chosen |
| **F** | Failed | Errors prevented task completion, or agent abandoned the session |

### Decision points
When the agent has multiple possible actions, it can record why it chose one over another. Example:

```json
{
  "context": "need to read a file — it could be at src/config.ts or lib/config.ts",
  "options": ["read_file src/config.ts", "glob **/config.ts", "grep 'CONFIG' *.ts"],
  "chosen": "glob **/config.ts",
  "reasoning": "File might not exist at the expected path; glob guarantees finding it regardless of location"
}
```

Decision points create an audit trail of agent reasoning, making it possible to understand not just what happened, but why.

### Full audit trail
Every action is timestamped and linked to a session. The audit trail answers:

- Who (which agent/user) accessed what data?
- When did each tool call happen?
- What data was read or modified?
- Was the action authorized?

## Grade System

Grades are assigned manually by the agent via `obs_record_grade` or automatically by the dashboard based on session