digline

# digline

<picture>
  <source media="(prefers-color-scheme: dark)" srcset="docs/assets/flow/digline-flow.svg">
  <img src="docs/assets/flow/digline-flow-light.svg" alt="digline in an agentic flow" width="960">
</picture>

---

<br>

**Regression testing for LLM applications — with the baseline in your repository, not on someone's server.**

[![PyPI](https://img.shields.io/pypi/v/digline)](https://pypi.org/project/digline/)
[![Downloads](https://static.pepy.tech/badge/digline/month)](https://pypistats.org/packages/digline)
[![Python 3.12+](https://img.shields.io/pypi/pyversions/digline)](pyproject.toml)
[![License: Apache-2.0](https://img.shields.io/badge/license-Apache--2.0-green)](LICENSE)
[![CI](https://github.com/digline/digline/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/digline/digline/actions/workflows/ci.yml)
[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/14570/badge)](https://www.bestpractices.dev/projects/14570)
[![OpenSSF Scorecard](https://api.scorecard.dev/projects/github.com/digline/digline/badge)](https://scorecard.dev/viewer/?uri=github.com/digline/digline)

Your prompt worked on Tuesday. On Thursday it works a little less — not enough
to break, enough for a user to notice in two weeks. No ordinary test catches it,
because there is no correct output to compare against, only a better or a worse
one.

digline gives you an **approved reference** — the baseline — and on every change
tells you whether you are below it: which case, which check, by how much. The
baseline is a JSON file in your repository, so it goes through code review and
it rolls back with `git`. No server, no account, no network call you have not
configured yourself.

```console
$ digline compare --suite suite.py --run latest
2 checks got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.

how-do-i-return · llm_rubric · Score fell from 1.000000 to 0.700000.
how-do-i-return · contains · Went from passing to failing (1.000000 → 0.000000).
```

## Why digline

Most evaluation tools tell you whether an output is below a threshold. digline
also tells you whether it is *worse than it was* — the drift from 0.91 to 0.78
that trips no threshold and is the first thing a user feels.

The suite is **Python, not YAML**: a judge is an object, a target is a function,
and what may leave a perimeter is declared in code — none of which a
configuration file expresses without reinventing a language. Where a suite *is*
plain data it can be TOML instead, and the two forms build the same objects —
what TOML cannot express, it refuses by name rather than half-supporting. Built
for teams shipping LLM features for someone else, who have to show a customer
what was tested, when, under which commit, and who approved it.

Wondering how digline differs from promptfoo, DeepEval, or observability
platforms? See [How digline compares](https://digline.dev/comparison/).

## Quickstart

With uv (recommended):

```bash
uv init && uv add digline
```

or with pip in an existing environment: `pip install digline`.

**Requires Python 3.12+**, which uv fetches for you if you do not have it. On
the pip path an older interpreter says `ERROR: No matching distribution found`
with `from versions: none`, which does not say why — that is what it means.

**If you use Claude Code**, digline also installs as a plugin. What it ships
first is the judgement layer. The [`operating-digline`](plugins/digline/skills/operating-digline/SKILL.md)
skill carries the same rules as [`AGENTS.md`](AGENTS.md), and loads when the
agent is about to run a suite or read a comparison. Beside it is the MCP server,
which measures, reads and explains, and cannot promote a baseline. Two lines
plus one dependency: the server imports your suite, so it runs from your
project's `.venv`.

```bash
uv add --dev digline-mcp
claude plugin marketplace add digline/digline
claude plugin install digline@digline --scope project
```

The agent's own shell commands are not the server. A bare `digline` typed
there resolves through the `PATH` of the shell Claude Code started, which comes
from your login profile and from whatever environment was active when you
launched it, and it can be another project's install at another version.
`uv run digline` resolves through your project's environment.

What it installs, and what its hook does and does not guarantee:
[`plugins/digline/README.md`](plugins/digline/README.md).

In another agent, the skill alone installs through [skills.sh](https://skills.sh):
`npx skills add digline/digline --skill operating-digline`. That is the skill
without the MCP server or the hook, taken from `main` rather than a release.

`suite.py` — complete and runnable, no API key:

```python
"""suite.py — complete and runnable: no API key, nothing else to install."""

from digline.core import Contains, CostBudget, JudgeReply, LlmRubric
from digline.run import Case, Response, Suite

ANSWERS = {
    "where-is-my-order": "Order 4821 ships Thursday. — Northwind Support",
    "how-do-i-return": "Any item, within 30 days, unused. — Northwind Support",
}


def judge(prompt: str) -> JudgeReply:
    """Your judge. digline composes `prompt` from the rubric, the question and
    the answer; it wants a score in [0, 1] and a reason back."""
    signed = "Northwind Support" in prompt
    concise = len(prompt.split()) <= 60
    return JudgeReply(
        score=0.4 + 0.3 * signed + 0.3 * concise,
        reason=f"signed={signed}, concise={concise}",
    )


def target(case: Case) -> Response:
    """Your application, called once per case. Canned here so this runs as is."""
    text = ANSWERS[case.id]
    return Response(output=text, cost_usd=0.004 + 0.001 * len(text) / 100)


suite = Suite(
    tenant="northwind",
    environment="staging",
    name="support",
    assertions=[
        Contains(needle="Northwind Support"),
        LlmRubric(
            rubric="Does the reply answer the question in at most three sentences?",
            judge=judge,
            threshold=0.7,
            tolerance=0.05,
        ),
        CostBudget(max_usd=0.02, tolerance=0.05),
    ],
    cases=[Case(id="where-is-my-order"), Case(id="how-do-i-return")],
)
```

`tenant="northwind"` is ours, like every tenant in this repository's examples,
and it has to become yours before the first run. A tenant is one customer: one
perimeter, and the directory your store is keyed on, `.digline/<tenant>/`.
Name it after the customer the suite is for. A suite copied to the next
customer with this line unchanged files that customer's runs in the first
one's directory, compares them against the first one's reference and promotes
over it, each with exit `0`.

When your judge is a real model, add a provider plugin:
`uv add digline-anthropic` (or `pip install digline-anthropic`), likewise
`digline-openai` and `digline-bedrock`.

```console
$ digline run --suite suite.py
2026-08-26T15-44-09-282929-00-00-e7421ec503ccefe8

$ digline promote --suite suite.py --run latest --replacing none
tenant 'northwind' · support baseline set to 2026-08-26T15-44-09-282929-00-00-e7421ec503ccefe8
```

Now make it worse — delete `— Northwind Support` from the second answer —
and ask again:

```console
$ digline run --suite suite.py
2026-08-26T15-44-09-492722-00-00-e7421ec503ccefe8

$ digline compare --suite suite.py --run latest
2 checks got worse compared with the reference. Every case could be judged. No case is suspended. The suite is unchanged from the reference.

how-do-i-return · llm_rubric · Score fell from 1.000000 to 0.700000.
how-do-i-return · contains · Went from passing to failing (1.000000 → 0.000000).

$ echo $?
1
```

The exit code is the answer: `0` fine, `1` got worse, `2` could not be judged.
Everything lands in `.digline/<tenant>/` — `baselines/` committed, `runs/`
git-ignored through a `.gitignore` digline writes for you.

## What it checks

**Per case** — pure functions `(inputs) -> Verdict`, no I/O, callable on their own:

| Assertion | Use it when |
|---|---|
| `Equals`, `Contains`, `NotContains`, `Affix`, `Regex` | the output must, or must not, contain something specific |
| `IsJson`, `JsonSchema` | the output is structured |
| `Length` | answers are growing, or must fit a channel |
| `Levenshtein` | "close enough" to `Case.expected`, graded rather than binary |
| `LlmRubric` | the criterion is a judgement — is it polite, does it stay on policy |
| `Faithfulness` | RAG: is the answer supported by the retrieved context |
| `FromAutoevals` | you already have an `autoevals` scorer and want it under a baseline |
| `PiiAbsent` | the output reaches a person — IBAN, codice fiscale, partita IVA, email, phone, checksum-verified where one exists |
| `ToolsCalled` | the target is an agent — a function, or an OpenAI, Anthropic or Bedrock target, whose plugins record every tool call — which tools it called, in order: an answer produced without the lookup that should have produced it |
| `ToolCalledWith` | the other half of a trajectory: the arguments a tool was called with — the right tool asked the wrong question |
| `CostBudget`, `LatencyBudget` | always. Graded, so a cost creeping up *within* budget is still visible |
| `Repeated` | the judge oscillates: grade the same output `n` times and fold the votes |

A suite pointed at an OpenAI-compatible endpoint, a gateway or a self-hosted
model declares what it actually charges — `[target.pricing]` in TOML,
`override()` in Python — so a `CostBudget` reads your prices rather than the
plugin's list.

**Per run** — one verdict on the whole suite, the kind that goes in a contract:

| Aggregate | Use it when |
|---|---|
| `Precision` | false positives are what your users see |
| `Recall` | what is missed is what your users miss |
| `Accuracy`, `F1` | you need a single number for both |

Every assertion carries a **threshold that can fail** — there is no default that
passes vacuously, and `Contains("")` is a `Value