agent-observability-spec

# Agent Observability Spec Skill

You can't fix what you didn't record. For LLM systems the unit of observability is the *trace* — everything the model saw and did — because behaviour, not uptime, is what fails. This skill specifies what to capture, what to compute from it, and when to page someone.

## What This Skill Produces

- A **trace schema**: per-request spans and the fields each must carry
- **Metric definitions** across health, quality, cost, and behaviour — each with a threshold and owner
- A **sampling and retention policy** that keeps cost sane and debugging possible
- A **privacy note**: what logged content contains, who can see it, and how long it lives

## Required Inputs

Ask for (if not already provided):
- **The system's shape** — single LLM call, RAG pipeline, or multi-step tool-using agent
- **Traffic volume and cost sensitivity** — full tracing at 10M req/day is a budget decision
- **What "misbehaving" means here** — the two or three failure modes that matter most (wrong facts? wrong actions? cost? refusals?)
- **Existing observability stack** (Datadog, Langfuse, OTel, homegrown) — spec into it, not around it

## Trace Schema

Every request produces one trace; every model call, retrieval, guardrail check, and tool execution is a span. Minimum fields:

| Span | Must capture |
|---|---|
| **Request root** | request id, user/session (pseudonymous), feature + prompt version, model id, total tokens, total cost, latency, terminal status |
| **Model call** | full input context (or content-addressed ref), output, finish reason, tokens in/out, cached-token share, temperature |
| **Retrieval** | query, top-k ids + scores, which chunks entered the context |
| **Tool call** | tool name, arguments, result (or ref), duration, error |
| **Guardrail** | check name, verdict, and *what it did* (blocked / rewrote / flagged) |
| **User signal** | edits, regenerates, thumbs, abandonment — joined to the trace id |

The test of the schema: **an engineer can replay any incident from its trace alone** (see `agent-incident-postmortem`).

## Metrics and Alerts

Define four families; every metric gets a threshold, a window, and an owner.

- **Health** — error rate, p50/p95 latency, timeout rate, provider 429/5xx rate. *Page* on these.
- **Cost** — cost per request (p50, p99), tokens per request, cache hit rate, daily spend vs. budget (pair with `llm-cost-latency-budget`). *Alert* on p99 and daily-budget burn — cost incidents are caused by the tail, not the mean.
- **Quality proxies** — format/schema violation rate, refusal rate, groundedness-check failure rate, judge score on a sampled slice, regenerate/edit rate. *Alert on drift* vs. a rolling baseline: absolute thresholds go stale, deltas don't.
- **Behaviour (agents)** — steps per task, tool-error rate, loop detection (same tool + same args N times), unauthorised-action attempts caught by guardrails. *Page* on the last one.

## Sampling & Retention

- **Metadata for 100%** of requests (ids, versions, tokens, cost, status) — this is cheap and non-negotiable.
- **Full content traces:** 100% for errors, guardrail hits, and negative user signals; [1-10]% random sample for the rest, adjusted to volume.
- **Retention:** full content [30-90] days, metadata [12+] months for trend baselines; incident traces pinned indefinitely.
- **Privacy:** logged context contains user data — state where it lives, who has access, how deletion requests reach it, and that traces are scrubbed or access-gated before wide sharing.

## Output Format

### Observability Spec: [feature/agent]

**System shape:** [calls/pipeline/agent] · **Volume:** [req/day] · **Stack:** [tooling]

**Trace schema:** [the span table, tailored]

**Metrics:**
| Metric | Family | Threshold / baseline | Window | Alert → owner |
|---|---|---|---|---|

**Sampling & retention:** [the policy]

**Privacy:** [content classification, access, deletion path]

**Dashboards:** [the 2-3 views: live health, quality drift, cost]

**First incident drill:** pick yesterday's worst trace and confirm it can be replayed end-to-end from the stored data.

## Quality Checks

- [ ] Any incident is replayable from its trace alone — the schema was tested against that bar
- [ ] Every metric has a number, a window, and a named owner — no orphan dashboards
- [ ] Quality alerts are drift-based against a rolling baseline, not absolute guesses
- [ ] Sampling keeps 100% of error/guardrail/negative-signal traces
- [ ] The privacy note exists and names retention and access — logged prompts are user data

## Anti-Patterns

- [ ] Do not log only inputs and outputs — without retrieval and tool spans, root cause analysis is guesswork
- [ ] Do not alert on mean cost or mean latency — the tail is where both incidents live
- [ ] Do not run judge-based quality scoring on 100% of traffic — sample; spend the budget on better baselines
- [ ] Do not treat observability as launch-week scaffolding — drift metrics only work with months of baseline
- [ ] Do not ship an agent that can take actions without logging the guardrail verdicts alongside the actions