tracely

# Tracely

Trace-native CI/CD for AI agents. One loop:

```
production trace → failure detection → regression test → CI gate
```

The trace is the source of truth. Evaluators, failure clusters, regression cases, gates and trends
are all **derived from it** — there are no hand-authored datasets. Docs: <https://doc.tracely-ai.com>.

## Pick the path first

Do not start writing spans. Ask what the code already looks like, then pick:

| The user's code | Path | Effort |
|---|---|---|
| Calls OpenAI / Anthropic / Gemini / Mistral / Bedrock / Groq SDKs | **Automatic** — `init(instrument="auto")` | 1 line |
| Uses LangChain / LangGraph / LlamaIndex / CrewAI / LiteLLM | **Automatic** + that extra | 1 line |
| Uses OpenAI Agents SDK / Claude Agent SDK / Google ADK | **Automatic**, named explicitly | 1 line |
| Has business logic worth seeing (routers, tools, retrievers) | Automatic **+ `@observe`** | 1 decorator each |
| Needs spans the auto path can't produce (custom retrievers, guardrails, handoffs, multimodal I/O, hand-rolled providers) | **Manual context managers** | `references/manual.md` |
| Is TypeScript / Go / Ruby / anything not Python | **Emit OTLP directly** — no SDK | `references/automatic.md` § other languages |
| Already emits OpenTelemetry / OpenInference / OpenLLMetry | Point the existing exporter at Tracely, add 2 attributes | `references/automatic.md` § other languages |

**Default to automatic.** Manual spans are the escape hatch, not the starting point, and the two
compose — manual spans nest inside auto-instrumented traces in the same tree.

## Connect

```python
import tracely_sdk as tracely   # pip install "tracely-ai[openai]"  ([anthropic] [langchain] [all])

tracely.init(
    endpoint="http://localhost:8000",   # hosted: https://api.tracely-ai.com
    api_key="tracely_dev_key",          # an ingest key — Settings → API keys. The key IS the workspace.
    service_name="support-agent",
    env="prod",                         # prod | staging | ci | dev — the gating axis
    instrument="auto",
)
```

Python ≥ 3.10. Import name is `tracely_sdk`; the CLI is `tracely`. `init()` is idempotent — call it
once at startup. Prefer `os.environ` for `endpoint`/`api_key` in real code; never inline a key.

## The 90% path

```python
with tracely.trace(agent="support-agent", conversation="conv-1", user="u_42"):
    client.chat.completions.create(model="gpt-4o", messages=[...])   # traced, no span code
```

`trace()` opens no span — it stamps run context (`agent`, `conversation`, `turn`, `user`, `env`,
arbitrary metadata) onto **every** span inside it, including the ones the instrumentor created.
It also works as a decorator on sync or async functions.

## Six rules that decide whether Tracely actually works

Getting these wrong doesn't error — it silently produces a useless workspace. Check them in every
review of someone's instrumentation.

1. **The agent name is declared, never inferred.** It's the dimension gates, clusters, scenarios
   and trends group by, and Tracely reads exactly one attribute for it: `tracely.agent.id` — set by
   `init(service_name=…)`, overridden by `init(agent=…)` for the app or `trace(agent=…)`/`agent(...)`
   per run. A framework's own `gen_ai.agent.name` is deliberately ignored: a harness stamps one on
   every sub-agent it spins up, which would register dozens of agents nobody chose. Name neither and
   every trace in the workspace lands under a single `default` agent. **One codebase serving many
   customers → `trace(tenant=customer_id)`**: each tenant is its own Agent (endpoint, scenarios,
   gate, clusters, cases; the traces list filters by it) and `agent=` stays the per-span label.
2. **`conversation=` is what makes a conversation.** Each turn is its own trace; passing the same
   conversation id to every turn is the only thing that threads them. Without it, a 12-turn support
   thread is 12 unrelated rows, and every conversation-level evaluator has nothing to grade.
3. **`env` is the gating axis.** `prod` failures become regression cases; `ci` traces are what the
   gate grades. Tagging CI runs as `prod` poisons the case pool with test data.
4. **Errors are the failure signal.** A failed tool must be marked — `tracely.error(span, msg)`, an
   exception inside `@observe`, or an OTel `ERROR` status. Detection, clustering and the gate all key
   off it. A tool that returns `{"error": "..."}` as a *successful* span is invisible.
5. **`flush()` before the process exits.** Scripts, Lambdas, CLI runs and tests lose their last
   spans otherwise. Long-lived servers don't need it.
6. **If Tracely calls your agent, honour the `traceparent` header.** Scenarios and `simulate` mint
   the trace id and send it. Ignore it and the gate sees only text in / text out — blind to your
   tool calls, so tool expectations report `SKIP` instead of grading. See `references/ci-gate.md`.

Two more that bite on specific stacks:

- Streaming OpenAI calls need `stream_options={"include_usage": True}` or token counts (and
  therefore cost) are lost.
- LangChain + a provider instrumentor double-traces. Under `"auto"` LangChain wins and the provider
  instrumentors are skipped; pass an explicit list only if you also make direct provider calls.

## Your own spans — `@observe`

The cheapest way to see business logic. Args → input, return → output, exceptions → `level=ERROR`,
auto-nested via OTel context with no parent wiring.

```python
@tracely.observe(as_type="tool")
def get_weather(city: str) -> dict:
    return {"city": city, "tempF": 64}
```

`as_type` ∈ `agent` · `delegate` · `generation` · `tool` · `skill` · `chain` · `retriever` ·
`thinking` · `embedding` · `guardrail` · `span`. Decorating tools also makes them **hermetically
replayable** in CI for free.

## Manual spans

When the auto path can't express it. Every helper is a context manager; nesting builds the tree.

```python
with tracely.agent("support-agent", version="v4", conversation="conv-1", turn=0) as a:
    tracely.set