regression-watch

# Regression Watch

Detect whether Claude Code sessions are getting worse over time across quality and
efficiency metrics, using Agent Monitor data.

## Input

The user provides: **$ARGUMENTS**

This may be:
- empty or "all" — check every regression metric (default)
- "errors" — error-rate regression only
- "cache" — cache hit-rate regression only
- "compaction" — compaction-frequency regression only
- "cost" — cost-per-session regression only
- A window like "last 30d" or "30 vs 90" — set the recent vs baseline window sizes

## Data Sources

| Endpoint | Returns |
|----------|---------|
| `GET /api/analytics` | `daily_events` (365d), `daily_sessions` (365d), `event_types`, `tokens` (total_input, total_output, total_cache_read, total_cache_write — baselines pre-summed), `avg_events_per_session` |
| `GET /api/events?session_id=X` | Event stream incl. `APIError`, `Compaction`, `PreToolUse`/`PostToolUse` — used to localize regressions to specific sessions |
| `GET /api/pricing/cost` | `{ total_cost, breakdown[...] }` — total cost to derive cost-per-session |
| `GET /api/pricing/cost/{sessionId}` | Per-session cost — used to compare recent vs baseline session cost |
| `GET /api/workflows/{sessionId}` | `compaction` (impact), `errorPropagation` (by depth), `effectiveness` — per-session quality signals |
| `GET /api/sessions?limit=N` | Sessions with `started_at`, `cost`, `metadata` — to bucket sessions into time windows |

## Report Sections

### 1. Windowing
Split history into a **baseline window** (older) and a **recent window** (newer).
Default: recent = last 30 days, baseline = the 30–90 day range before it. Use
`daily_events`/`daily_sessions` for series metrics and `GET /api/sessions?limit=N`
to assign sessions to each window by `started_at`.

### 2. Error Rate Regression
- Recent error rate = `APIError count / total events` in the recent window
  (from `event_types` and `daily_events`, or per-session `GET /api/events`).
- Compare to the baseline rate. Flag if recent is higher.
- Report the absolute and relative change and which sessions contributed most
  `APIError` events.

### 3. Cache Hit Rate Regression
- Cache hit rate = `total_cache_read / (total_cache_read + total_input)`.
- Compute for each window (per-window input/cache_read from session metadata or
  the pricing breakdown). Flag a **falling** hit rate — that means more
  uncached input tokens and higher cost.

### 4. Compaction Frequency Regression
- Compaction frequency = `Compaction events / session` per window (from
  `event_types` / `daily_events`, confirmed via per-session
  `GET /api/workflows/{id}` `compaction`). Flag a **rising** rate — context is
  overflowing more often.

### 5. Cost-per-Session Regression
- Cost-per-session = window total cost / window session count, using
  `GET /api/pricing/cost` overall and `GET /api/pricing/cost/{id}` for the
  sessions in each window. Flag a **climbing** value.

### 6. Verdict
Roll up which metrics regressed, rank by relative worsening, and name the most
likely driver (e.g., cache hit rate fell → cost per session climbed).

## Output

- A Markdown table: metric | baseline | recent | Δ | direction (▲ worse / ▼ better) | verdict.
- Tag each regressed metric 🔴 (clear regression), 🟡 (mild/within noise), or 🟢 (improved).
- Currency in USD to 4 decimals; rates as percentages to 2 decimals.
- List the specific session IDs that contributed most to any regression.
- End with the single highest-priority regression to address and a concrete next step.
- Read-only: only report what the API returns; never fabricate baselines.