agent-session-format-check

# Agent Session Format Check

Answer one precise question for every supported agent:

> Can current Agent Sessions code support the latest available session/storage/usage
> format from the latest available agent build?

The answer must be layered. Do not collapse version checks, stale samples, schema
fingerprints, discovery contracts, and usage probes into a vague severity label.

**Evidence-first:**
- Gather a report + sample paths first.
- Do not change parsers/fixtures/docs without explicit user approval.

**Related skill:** `agent-support-matrix` — maintains the matrix YAML, ledger, and
update-checklist workflow. This skill focuses on *detection and evidence collection*;
`agent-support-matrix` focuses on *recording and gating version bumps*.

**Process doc:** `docs/agent-support/monitoring.md` — defines the compatibility verdicts,
legacy severity model, cadence, and escalation workflow that feed into this skill.

---

## 1  Quick Start (all agents)

1. Run weekly monitoring:
   ```
   ./scripts/agent_watch.py --mode weekly
   ```
   Report path prints to stdout and is written under
   `scripts/probe_scan_output/agent_watch/*/report.json`.

2. In `report.json`, check each agent under `results.<agent>`:
   - `compatibility.verdict`, `compatibility.scope`, `compatibility.blockers`,
     and `compatibility.next_action`
   - `verified_version`, `installed.parsed_version`, `upstream.parsed_version`
   - `compatibility.latest_status` to distinguish `current_fetch_known`,
     `cached_latest`, and unknown latest-source states
   - `weekly.local_schema` (newest local session used for fingerprinting)
   - `weekly.schema_diff` and `evidence.schema_matches_baseline`
   - `evidence.sample_freshness` and `evidence.fresh_evidence_source`
   - `compatibility.latest_real_session_failure` when a prebump attempt failed
   - `severity` and `recommendation` only as legacy escalation fields

3. **Usage / limits reading (Codex + Claude) — always verify every weekly run.**
   These drift independently of session schema (see §2), so a clean schema does
   **not** imply healthy usage reading. Each agent's
   `results.<agent>.weekly.probes` is a **list**; for every relevant entry confirm
   `ok == true` and `exit_code == 0`:
   - Codex — `label == "codex_status_probe"` (parse `codex_status_json`): the
     active CLI status channel (`five_hour`, `weekly` percent-left). The passive
     channel is the session JSONL `token_count` / `rate_limits` events, covered by
     the schema fingerprint above.
   - Claude — `label == "claude_usage_probe"` (parse `claude_usage_json`): the
     **authenticated** `/usage` reading (`session_5h`, `week_all_models`,
     `week_opus`). Also `label == "claude_status"` (parse `claude_status_json`):
     status.claude.com indicator/incidents.
   A failed or unparsed usage probe is a usage-format or auth regression even when
   versions match and the session schema is clean — never skip it, and report each
   probe's `ok` explicitly rather than collapsing it into the compatibility verdict.

4. **Classify every new field for value, not just for safety (§1e).** A field that parses
   cleanly is *safe*, which is not the same as *handled*. Do not close a drift finding
   until each new key or type has been called noise, watch, or feature-candidate — and
   candidates filed in `docs/backlog.md`, not left as a remark in the ledger.

Interpretation:
- `supports_latest`: latest known build is covered by
  `evidence.fresh_evidence_source == "latest_prebump_report"` and
  `compatibility.latest_real_session_evidence == true` with
  `compatibility.latest_status == "current_fetch_known"`.
- `supports_installed_only`: installed build is covered by non-stale real local
  evidence, but latest is newer, cached from a prior report, unknown, or lacks
  fresh real-session proof.
- `latest_unknown`: no configured/reachable latest source or no real-session
  driver exists; do not claim latest support. **Check `STEWARDS.md` before calling this
  a finding** — for a steward-verified agent not installed here, this verdict is the
  expected output and the agent belongs to its steward, not to you (§1g).
- `blocked_stale_sample`: evidence predates the installed CLI; run prebump before claiming support.
- `blocked_no_fresh_evidence`: a version changed but no fresh matching sample proves support.
- `format_drift_detected`: unknown schema/storage/usage fields appeared; update fixtures/parsers.
- `blocked_thin_sample`: the sample was both narrow and tiny, so it evidenced nothing either
  way (§5a). Generate a session that actually uses tools — not a one-line prompt.
- `monitoring_broken`: latest source, usage probe, or discovery contract failed.
- `real_session_auth_failed` in blockers: the real-session driver ran but the
  sandboxed agent was not authenticated; re-auth or provide the configured env
  token, then rerun prebump.

---

## 1a  Real-Session Prebump Validation (required before latest claims)

Weekly scanning samples the newest on-disk session, which can predate a CLI
upgrade and give a false "safe to bump" call (the codex 0.120.0 trap and the
copilot `session.shutdown` trap). When weekly reports
`recommendation == run_prebump_validator` — or before you stage any
`max_verified_version` bump or latest-support claim — run the prebump path for
every active agent being claimed. The driver exercises the currently installed
CLI once inside a sandbox and diffs its output against the fixture baseline:

```
./scripts/agent_watch.py --mode prebump --agent codex --agent claude
```

Exit-code contract:
- `0` — every requested agent produced a fresh session and the schema
  matches baseline. Safe to bump.
- `2` — at least one fresh session's schema does **not** match baseline.
  Do **not** bump; investigate the schema diff in
  `scripts/probe_scan_output/agent_watch/<slug>-prebump/report.json`.
- `3` — at least one driver failed (timeout, auth, CLI not found, or
  discovery contract violation — wrong session root, wr