live-debug

# Live debugging

`just run_local` and `just stack up` start the LGTM collector by default and
the agent browser with `--with-chrome` (both global: one per machine, shared
across instances, left running):

| Endpoint | What |
| --- | --- |
| http://localhost:3001 | Grafana (anonymous admin) — traces + logs UI |
| http://localhost:3200 | Tempo API — traces |
| http://localhost:3100 | Loki API — service logs |
| http://localhost:9090 | Prometheus API — metrics |
| localhost:4317 / 4318 | OTLP intake (gRPC / HTTP), alias `otel-collector` |
| http://localhost:9222 | Headless Chrome (CDP) |

`--traces off` disables the collector; `--traces jaeger|datadog` swaps it.
All collectors bind 4317/4318 — exactly one runs at a time.

Every Rust service exports spans AND its `tracing` events (as correlated log
records) over OTLP. The frontend exports browser spans through the proxy and
propagates `traceparent`, so one trace covers browser → proxy → services.
Wiring happens at stack start: if you start a collector by hand, restart the
stack to pick it up.

## Query telemetry (prefer APIs over the Grafana UI)

The `grafana` MCP server (`.mcp.json` / `opencode.json` / `.cursor/mcp.json`,
Docker `mcp/grafana` on the host network) is pointed at this Grafana. Prefer its tools:

- Logs: `query_loki_logs`, `list_loki_label_values`, `find_error_pattern_logs`
- Traces: `tempo_traceql-search`, `tempo_get-trace`,
  `tempo_get-attribute-values`, `tempo_docs-traceql` (proxied from Tempo's
  own MCP server; every `tempo_*` call needs `datasourceUid: "tempo"`)
- Metrics: `query_prometheus`; plus `search_dashboards`, `generate_deeplink`

Datasource UIDs are stable: `loki`, `prometheus`, `tempo`, `pyroscope`.

Typical calls — service names are the binary names (`email_service`,
`document-storage-service`); match on route, duration, or any span attribute:

- `tempo_traceql-search` `{datasourceUid: "tempo", query: '{resource.service.name="email_service" && status=error}'}`
  (also `'{span.http.route="/documents" && duration>500ms}'`), then
  `tempo_get-trace` with the returned trace ID. Searches default to the past
  hour; widen with RFC3339 `start`/`end`.
- `query_loki_logs` `{datasourceUid: "loki", logql: '{service_name="email_service"} |= "error"'}`
  — log lines carry `trace_id`/`span_id` for correlation. Discover services
  with `list_loki_label_values` on `service_name`.

Timing quirks: Tempo's search index flushes every ~30s — a trace you just
produced is fetchable by ID immediately but may not show in search yet.

Everything above is also plain HTTP: TraceQL search at
`http://localhost:3200/api/search?q=<traceql>`, trace fetch at
`/api/traces/<id>`, LogQL at
`http://localhost:3100/loki/api/v1/query_range?query=<logql>` (`start`/`end`
are unix epoch **nanoseconds**). Use the HTTP form when there is no MCP (pi),
and for bulk retrieval you want to reduce before reading — a full trace can
be 50+ spans, so `curl /api/traces/<id> | jq` (filter to slow spans, compute
offsets) beats dumping `tempo_get-trace` output into context. `docker compose
-p macro logs -f <service>` still works for raw stdout, but Loki is queryable
and survives restarts.

Verbosity knobs (set in the shell before `just run_local`, or per service in
Doppler): `RUST_LOG` filters console + Loki output; `OTEL_TRACE_FILTER`
independently filters exported spans (default `info`). Lowering `RUST_LOG`
never silences traces.

## Drive the shared headless Chrome

`--with-chrome` runs a headless Chrome in Docker with CDP on 9222 (if 9222
doesn't answer, start it: the compose command is in the `headless-chrome`
comment in `docker/docker-compose.yml`). The `chrome-devtools` MCP server
(`.mcp.json` / `opencode.json` / `.cursor/mcp.json`) is already pointed at
it — prefer its tools (navigate, snapshot, click, evaluate, console, network) for browser
work. State (cookies, login) persists across agent sessions until the
container restarts.

- The container uses host networking, so plain localhost URLs work: the app
  is **http://localhost:3000/app**, the proxy `http://localhost:8090` (named
  instances remap these ports; read the stack summary).
- A human can watch the browser live at **http://localhost:6080/vnc.html**
  (noVNC over the Xvfb display) — work in the visible window, not isolated
  contexts, when someone may be watching.
- Login is passwordless: any email works, and the login API returns the code
  in its response (also visible in Mailpit at http://localhost:8025).
- From Playwright instead: `chromium.connectOverCDP('http://localhost:9222')`.
  For token-injection and route-interception recipes see
  `apps/web/docs/playwright-debugging.md`.

Correlate a browser repro with backend traces: note the time, then search
Tempo for that window — the browser's `traceparent` means the frontend action
and the Rust handler share one trace ID.

### chrome-devtools technique

- Snapshot-first: `take_snapshot` after every navigation or pane change; act
  only on uids from the latest snapshot (uid prefixes bump on re-render).
  Macro snapshots are huge — split panes duplicate the doc text — so save
  big ones to a file (`filePath` param) and grep them.
- `navigate_page` can time out while the SPA actually loaded (cold Vite
  compile); follow with `wait_for` on expected text instead of re-navigating.
- `wait_for` matches any text presence, including placeholders. For "AI
  finished"-style conditions, poll in `evaluate_script` for the Stop
  button's absence — the only reliable completion signal.
- `fill` works on plain inputs but NOT contenteditable: click to focus, then
  `type_text` (Enter splits paragraphs/sends). Combobox token fields need
  type → wait for the "N options available" live region → Enter to tokenize.
- If a radio/tab control won't click ("did not become interactive"), click
  its adjacent label text node instead.
- On any error dialog or blank state: `list_console_messages` +
  `list_network_requests` (filter xhr/fetch), then `get_network_request` for