genai-conformance

# GenAI Conformance

The repo ships a self-contained conformance harness at [python/openinference-instrumentation/scripts/conformance/](../../python/openinference-instrumentation/scripts/conformance/) that exercises OpenInference instrumentors against deterministic mock provider APIs, exports OTLP traces to `weaver registry live-check`, and prints a console summary of registry attributes seen / missing / advice-level counts. It validates the **dual-write** logic in [_genai_conversion.py](../../python/openinference-instrumentation/src/openinference/instrumentation/_genai_conversion.py) that translates OpenInference's native attributes (`llm.*`, `input.*`, `output.*`, `openinference.*`) into the OTel GenAI semantic conventions (`gen_ai.*`).

## When to Use

- User asks to run the conformance harness, "test conformance", or "run weaver".
- User asks to maximize / improve `gen_ai.*` registry coverage.
- User wants to extend the dual-write conversion in `_genai_conversion.py`.
- User wants to add a new provider, a new test scenario, or a new mock endpoint.
- User mentions specific `gen_ai.*` attributes (response.id, system_instructions, tool.call.*, retrieval.*, etc.) and whether they're being emitted.

## Layout

```
scripts/conformance/
├── run.py                 # orchestrator (PEP 723, stdlib only)
├── mock_server.py         # Flask mock with all providers' endpoints
├── anthropic_conformance.py      # PEP 723 + editable [tool.uv.sources]
├── openai_conformance.py         # PEP 723 + editable [tool.uv.sources]
├── google_genai_conformance.py   # PEP 723 + editable [tool.uv.sources]
├── README.md
└── results/                      # gitignored Weaver output
```

Each provider script declares its deps as PEP 723 inline metadata and pins the local OpenInference packages via `[tool.uv.sources.<pkg>]` blocks (multi-section dotted-key form — single-line inline tables exceed ruff's 100-char limit). `run.py` invokes everything via `uv run`. Filenames avoid the bare provider name (`openai.py`, `anthropic.py`) because that would shadow the SDK package on `sys.path[0]`.

`run.py` lives in [PROVIDER_SCRIPTS](../../python/openinference-instrumentation/scripts/conformance/run.py) — a tuple iterated for both prewarm and execution. To add a provider, append to `PROVIDER_SCRIPTS` and add the corresponding `<provider>_conformance.py` and any new mock endpoints.

## Running

```bash
uv run python/openinference-instrumentation/scripts/conformance/run.py
```

First run downloads pinned `weaver v0.22.1` and `semantic-conventions v1.40.0` to `~/.cache/oi-conformance/`; subsequent runs are fast. uv caches each provider script's env by PEP 723 metadata hash.

## Interpreting the summary

- **Registry attributes seen** — `gen_ai.*` (and a few `service.*` / `telemetry.sdk.*`) attrs the run emitted, with sample counts.
- **Non-registry attributes seen** — OpenInference's native vocabulary. These show up as Weaver `missing_attribute` violations by design — they aren't (and shouldn't be) in the OTel registry.
- **Missing registry attributes (`gen_ai.*`)** — registry attrs the run did *not* emit. Categorize each one:
  1. **Real dual-write gap** — provider API has the data, instrumentor captures it as an OI attr, but `_genai_conversion.py` doesn't map it. **Fixable in conversion.**
  2. **Test scenario gap** — conversion handles it, but the test doesn't exercise the relevant scenario (e.g. `gen_ai.tool.call.*` need a TOOL span; `gen_ai.embeddings.*` need an EMBEDDING span). **Fixable in `<provider>_conformance.py`.**
  3. **Mock data gap** — instrumentor would capture it if the response included it (e.g. `gen_ai.usage.cache_read.input_tokens` requires `cache_read_input_tokens` in the mock's usage block). **Fixable in mock_server.py.**
  4. **Provider doesn't support it** — e.g. Anthropic has no `frequency_penalty`. Document and skip.
  5. **Application-level / not auto-emittable** — `gen_ai.agent.*`, `gen_ai.evaluation.*`, `gen_ai.prompt.name`, `gen_ai.data_source.id`. Require explicit user attribution; out of scope for SDK instrumentation.
  6. **Metric-only** — `gen_ai.token.type` lives on `gen_ai.client.token.usage` metric, not spans.
- **Advice levels** — `violation` counts are predominantly `missing_attribute` for the OI native vocab (expected); `improvement` counts are `not_stable` warnings for development-stage `gen_ai.*` attrs (also expected). The dual-write itself is well-formed — Weaver does not flag type/shape/value errors on the emitted `gen_ai.*` attrs.

## Iterating to maximize coverage

For category 1 (dual-write gap):

1. Inspect `results/live_check.json` to see exactly what OI attributes the instrumentor emitted (look for the relevant span's `attributes` array).
2. Decide where to extend `_genai_conversion.py` (`get_genai_request_attributes`, `get_genai_response_attributes`, etc.).
3. **Always add a unit test in [test_genai.py](../../python/openinference-instrumentation/tests/test_genai.py)** for the new path. The existing tests cover the major span kinds; mirror that style.
4. Re-run the conformance harness. Verify the missing list shrinks and no existing `gen_ai.*` attribute regressed.

For category 2 (test scenario gap):

- Use `OITracer(trace.get_tracer(__name__), TraceConfig(enable_genai_semconv=True))` to manually emit non-LLM spans (TOOL, RETRIEVER, EMBEDDING, AGENT) inside a provider script. The Anthropic script already does this for TOOL / RETRIEVER / EMBEDDING — copy the pattern.

For category 3 (mock data gap):

- Mock responses are simple dicts at the top of `mock_server.py`. The Anthropic mock already returns `cache_creation_input_tokens` / `cache_read_input_tokens`; the OpenAI mock returns `prompt_tokens_details.cached_tokens`. Add fields the SDK will surface and the OI instrumentor will turn into `LLM_TOKEN_COUNT_PROMPT_DETAILS_CACHE_*`.

## Bumping the semconv version

The harness pins `SEMCONV_VERSION` (currently `v1.41.1`) and `WEAVER_VERSION` (currently `v0.23.0`) in `run.py`. When O