agent-tracing

# Agent Tracing CLI Guide

`@lobechat/agent-tracing` is a zero-config local dev tool that records agent execution snapshots to disk and provides a CLI to inspect them.

## How It Works

In `NODE_ENV=development`, `AgentRuntimeService.executeStep()` automatically records each step to `.agent-tracing/` as partial snapshots. When the operation completes, the partial is finalized into a complete `ExecutionSnapshot` JSON file.

**Data flow**: executeStep loop -> build `StepPresentationData` -> write partial snapshot to disk -> on completion, finalize to `.agent-tracing/{timestamp}_{traceId}.json`

**Context engine capture**: In `RuntimeExecutors.ts`, the `call_llm` executor calls `ctx.tracingContextEngine(input, output)` after `serverMessagesEngine()` processes messages. `AgentRuntimeService.executeStep` buffers the call per step and forwards it to `OperationTraceRecorder.appendStep` as the typed `contextEngine` field. CE flows through this side channel rather than the `events` array so its heavy payload (agentDocuments, systemRole, …) never enters the Redis state pipeline (LOBE-9110).

## Package Location

```
packages/agent-tracing/
  src/
    types.ts          # ExecutionSnapshot, StepSnapshot, SnapshotSummary
    store/
      types.ts        # ISnapshotStore interface
      file-store.ts   # FileSnapshotStore (.agent-tracing/*.json)
    recorder/
      index.ts        # appendStepToPartial(), finalizeSnapshot()
    viewer/
      index.ts        # Terminal rendering: renderSnapshot, renderStepDetail, renderMessageDetail, renderSummaryTable, renderPayload, renderPayloadTools, renderMemory
    cli/
      index.ts        # CLI entry point (#!/usr/bin/env bun)
      inspect.ts      # Inspect command (default)
      partial.ts      # Partial snapshot commands (list, inspect, clean)
    index.ts          # Barrel exports
```

## Data Storage

- Completed snapshots: `.agent-tracing/{ISO-timestamp}_{traceId-short}.json`
- Latest symlink: `.agent-tracing/latest.json`
- In-progress partials: `.agent-tracing/_partial/{operationId}.json`
- Downloaded remote snapshots: `.agent-tracing/_remote/{operationId}.json`
- `FileSnapshotStore` resolves from `process.cwd()` — **run CLI from the repo root**

## Remote Traces (Production / Staging)

Server deployments also upload completed snapshots to object storage (zstd-compressed; the key is stored in `agent_operations.trace_s3_key`).

**Preferred: `lh trace op`.** The server resolves the key and signs the object for the caller's own scope, so a LobeHub login is the only requirement — no `TRACING_BASE_URL`, no bucket domain, and no SQL to turn a topic id into an operation id:

```bash
lh trace op list --topic tpc_xxx # operations of a topic, newest first, with a TRACE column
lh trace op inspect op_xxx_agt_xxx_tpc_xxx_xxxx
lh trace op inspect op_xxx_agt_xxx_tpc_xxx_xxxx -T # tool injection (enabledToolIds, manifests)
```

`TRACE = —` in `list` means no snapshot was recorded for that run (it predates trace upload, or upload was off). A recorded snapshot can still 404 in storage after its retention window.

Backend: the `agentTrace` lambda router (`getSnapshotUrl` / `listOperations`). Note it is `blocked` for restricted API keys, so these commands need a real session, not a scoped key.

**Fallback: the standalone `agent-tracing` CLI.** It has no LobeHub session, so it builds the object URL itself and needs the bucket's public domain configured:

- env var: `TRACING_BASE_URL=https://<bucket-public-domain>/agent-traces`
- or `.agent-tracing/.env` in the repo root with the same `TRACING_BASE_URL=...` line

The deployment-specific value is private to each deployment and intentionally not recorded in this repo. Find the operation id by hand first:

```sql
SELECT id, trace_s3_key FROM agent_operations WHERE topic_id = 'tpc_xxx';
```

Either way the snapshot is cached to `.agent-tracing/_remote/<opId>.json`, and every `inspect` flag works the same as for local traces.

Implementation: `packages/agent-tracing/src/store/loadSnapshot.ts` (resolution order: local store → `_remote/` cache → injected `resolveDownloadUrl` → `TRACING_BASE_URL`) and `store/remote-store.ts` (URL built as `{base}/{agentId}/{topicId}/{opId}.json.zst`). Reading a compressed snapshot needs Node >= 22.15.

## Goal Trajectories

A goal is one complete _goal_ execution the way an operation is one complete agent execution, so it gets the same trace format one level up: `GoalTrajectory : AdvanceSnapshot` mirrors `ExecutionSnapshot : StepSnapshot`. There is no table of advances, exactly as there is no `agent_steps` table — `goal_traces` holds one rollup row per goal plus the object key, and the detail lives in the object.

The leaves join back down: an advance records the `operationId`s it put in flight (on `tick.effects[].operationId`), so `lh trace op inspect <opId>` continues from where the goal trace stops.

```bash
agent-tracing goal               # list local goal trajectories
agent-tracing goal goal_xxx      # the run: triggers, outcomes, graph, gates, ops
agent-tracing goal goal_xxx -a 3 # one advance in full, with the frontier it ranked
agent-tracing goal goal_xxx -j   # raw JSON
```

What each tick records is the **decision input**, not just the result: the graph it read (as a delta against the previous tick), the budget it evaluated, the responsible task's state, and every eligible work node **including the ones it passed over**. Without the losers a trace cannot answer "why not that node".

**Storage** follows the operation switch, so a deployment that keeps one keeps the other:

- Completed: `goal-traces/{goalId}.json.zst`
- In progress: `goal-traces/_partial/{goalId}.json.zst` (an unfinished long-horizon goal is the normal thing to inspect)
- Dev: `.goal-tracing/{goalId}.json`

**Replay.** `replayGoalAgainstCurrentCoordinator(trajectory)` re-runs the real `decideNextMove` over the recorded inputs and reports `{advanceSeq, tickIndex, field, recorded, replayed}` wherever the current coordinator