agent-observability-spec
Specify the tracing, metrics, and alerting for an AI agent or LLM feature in production. Use when asked what to log for an LLM app, design agent tracing or spans, define quality and cost monitors, or answer 'how do we know if the agent is misbehaving?'. Produces an observability spec with a trace schema, metric definitions with owners and alert thresholds, sampling and retention policy, and a privacy note for logged content.
git clone --depth 1 https://github.com/mohitagw15856/pm-claude-skills /tmp/agent-observability-spec && cp -r /tmp/agent-observability-spec/exports/openclaw/agent-observability-spec ~/.claude/skills/agent-observability-specSKILL.md
# Agent Observability Spec Skill You can't fix what you didn't record. For LLM systems the unit of observability is the *trace* — everything the model saw and did — because behaviour, not uptime, is what fails. This skill specifies what to capture, what to compute from it, and when to page someone. ## What This Skill Produces - A **trace schema**: per-request spans and the fields each must carry - **Metric definitions** across health, quality, cost, and behaviour — each with a threshold and owner - A **sampling and retention policy** that keeps cost sane and debugging possible - A **privacy note**: what logged content contains, who can see it, and how long it lives ## Required Inputs Ask for (if not already provided): - **The system's shape** — single LLM call, RAG pipeline, or multi-step tool-using agent - **Traffic volume and cost sensitivity** — full tracing at 10M req/day is a budget decision - **What "misbehaving" means here** — the two or three failure modes that matter most (wrong facts? wrong actions? cost? refusals?) - **Existing observability stack** (Datadog, Langfuse, OTel, homegrown) — spec into it, not around it ## Trace Schema Every request produces one trace; every model call, retrieval, guardrail check, and tool execution is a span. Minimum fields: | Span | Must capture | |---|---| | **Request root** | request id, user/session (pseudonymous), feature + prompt version, model id, total tokens, total cost, latency, terminal status | | **Model call** | full input context (or content-addressed ref), output, finish reason, tokens in/out, cached-token share, temperature | | **Retrieval** | query, top-k ids + scores, which chunks entered the context | | **Tool call** | tool name, arguments, result (or ref), duration, error | | **Guardrail** | check name, verdict, and *what it did* (blocked / rewrote / flagged) | | **User signal** | edits, regenerates, thumbs, abandonment — joined to the trace id | The test of the schema: **an engineer can replay any incident from its trace alone** (see `agent-incident-postmortem`). ## Metrics and Alerts Define four families; every metric gets a threshold, a window, and an owner. - **Health** — error rate, p50/p95 latency, timeout rate, provider 429/5xx rate. *Page* on these. - **Cost** — cost per request (p50, p99), tokens per request, cache hit rate, daily spend vs. budget (pair with `llm-cost-latency-budget`). *Alert* on p99 and daily-budget burn — cost incidents are caused by the tail, not the mean. - **Quality proxies** — format/schema violation rate, refusal rate, groundedness-check failure rate, judge score on a sampled slice, regenerate/edit rate. *Alert on drift* vs. a rolling baseline: absolute thresholds go stale, deltas don't. - **Behaviour (agents)** — steps per task, tool-error rate, loop detection (same tool + same args N times), unauthorised-action attempts caught by guardrails. *Page* on the last one. ## Sampling & Retention - **Metadata for 100%** of requests (ids, versions, tokens, cost, status) — this is cheap and non-negotiable. - **Full content traces:** 100% for errors, guardrail hits, and negative user signals; [1-10]% random sample for the rest, adjusted to volume. - **Retention:** full content [30-90] days, metadata [12+] months for trend baselines; incident traces pinned indefinitely. - **Privacy:** logged context contains user data — state where it lives, who has access, how deletion requests reach it, and that traces are scrubbed or access-gated before wide sharing. ## Output Format ### Observability Spec: [feature/agent] **System shape:** [calls/pipeline/agent] · **Volume:** [req/day] · **Stack:** [tooling] **Trace schema:** [the span table, tailored] **Metrics:** | Metric | Family | Threshold / baseline | Window | Alert → owner | |---|---|---|---|---| **Sampling & retention:** [the policy] **Privacy:** [content classification, access, deletion path] **Dashboards:** [the 2-3 views: live health, quality drift, cost] **First incident drill:** pick yesterday's worst trace and confirm it can be replayed end-to-end from the stored data. ## Quality Checks - [ ] Any incident is replayable from its trace alone — the schema was tested against that bar - [ ] Every metric has a number, a window, and a named owner — no orphan dashboards - [ ] Quality alerts are drift-based against a rolling baseline, not absolute guesses - [ ] Sampling keeps 100% of error/guardrail/negative-signal traces - [ ] The privacy note exists and names retention and access — logged prompts are user data ## Anti-Patterns - [ ] Do not log only inputs and outputs — without retrieval and tool spans, root cause analysis is guesswork - [ ] Do not alert on mean cost or mean latency — the tail is where both incidents live - [ ] Do not run judge-based quality scoring on 100% of traffic — sample; spend the budget on better baselines - [ ] Do not treat observability as launch-week scaffolding — drift metrics only work with months of baseline - [ ] Do not ship an agent that can take actions without logging the guardrail verdicts alongside the actions
Conduct a structured ethical review of an AI or ML feature, model, or product. Use when preparing to deploy an AI system, assessing algorithmic risk, auditing a model for bias, or producing a responsible AI impact assessment. Produces a structured ethics review covering fairness, transparency, privacy, safety, accountability, and societal impact with a risk tier score, pre-deployment checklist, and prioritised mitigations.
Structure AI and ML product decisions with the rigour of any product decision. Use when building AI-powered features, evaluating LLM integrations, designing AI products, or assessing AI readiness. Produces a complete AI product canvas covering problem definition, model approach, data requirements, evaluation framework, UX design, responsible AI checklist, and launch monitoring plan.
Transform feature briefs into structured design briefs that give designers the context they need before opening Figma. Use when asked to write a design brief, create a design handoff, brief a designer on a new feature, or translate a PRD into design requirements. Produces a brief with user goal, emotional context, success criteria, constraints, edge cases, and out-of-scope boundaries.
Design statistically rigorous A/B tests and interpret experiment results. Use when asked to design an experiment, run an A/B test, calculate sample size, interpret test results, or assess whether an experiment was successful. Produces a complete experiment design with hypothesis, sample size, run time, success criteria, and risk flags — or a results interpretation with ship/iterate/kill recommendation.
Synthesises user signals from multiple research sources into a unified, weighted insight brief. Use when you have data from interviews, support tickets, NPS verbatims, app reviews, or sales calls and need to reconcile contradictions, surface the underlying need behind requests, or answer 'what are users really telling us'. Produces ranked insights with confidence ratings, source weighting rationale, divergent signal analysis by user segment, and a research gap identification section.
Structure a product data analysis, metric deep-dive, funnel analysis, or cohort study. Use when asked to analyse product metrics, investigate a drop in conversion, explain a data change to stakeholders, or find the root cause of a metric movement. Produces a structured analysis with question, root cause, confidence level, and recommended action.
Interpret product metrics against goals and surface actionable signals. Use when asked to analyse product health, review key metrics, investigate a performance issue, produce a health report, or assess product-market fit signals. Produces a structured health report with RAG status, trend analysis, root cause hypotheses, and prioritised actions.
Structure a retention analysis, churn investigation, or engagement deep-dive for any product team. Use when asked to analyse user retention, investigate churn, measure DAU/MAU, or build a retention improvement plan. Produces a retention snapshot with root cause hypotheses, aha-moment correlation, and prioritised interventions.