oc-prompt-ops

# Prompt Ops

Treat prompts as **code**: source-controlled, diffable, semver'd, and gated on
an eval suite the same way application code is gated on tests. A prompt is the
single most behavior-defining string in an LLM app — and the one most teams
edit live, in a console, with no version history and no way to tell whether
"it got better" is real or vibes. This skill makes a prompt change a reviewable
diff with a measured score delta attached.

This is **not** a tri-agent harness. It's an operations layer the model-facing
skills build on: `oc-claude-api` owns the request surface (model routing,
caching, tool wiring), `oc-agent-forge` owns agent topology, `oc-rag-forge`
owns retrieval. Prompt Ops owns the part underneath all three — the prompt
text, the eval datasets that score it, and the regression gate that stops a
"small wording tweak" from quietly tanking quality on a migration.

opchain dogfoods this skill on itself. The worked example referenced throughout
is **`prompts/opchain-eval/`** — opchain's own eval set (`inputs.jsonl`,
`expected.jsonl`, `eval.yaml`), published in Sprint 3 as the canonical
`/oc-prompt eval` artifact. Wherever this doc says "the eval set", that directory
is the live instance.

> **Model facts come from `oc-claude-api` / the `claude-api` skill, not memory.**
> Judge-model choice, model IDs, and Batch-API economics in this skill are sourced
> there. Current models: **Fable 5** (`claude-fable-5`), **Opus 4.8**
> (`claude-opus-4-8`), **Sonnet 4.6** (`claude-sonnet-4-6`), **Haiku 4.5**
> (`claude-haiku-4-5`). When a prompt is pinned to a model, pin it to one of
> these exact IDs.

---

## /oc-prompt — Command Reference

```
PROMPT OPS COMMANDS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  PROMPT-AS-CODE
  /oc-prompt              Version / organize a prompt set (prompt-as-code layout)
  /oc-prompt diff         Diff two prompt versions + their eval-score deltas

  EVALUATION
  /oc-prompt eval         Run a prompt version against an eval dataset → scorecard
  /oc-prompt goldset      Build or extend the eval goldset (inputs/expected/rubric)
  /oc-prompt judge        Configure / calibrate the LLM-as-judge grader

  REGRESSION / DRIFT
  /oc-prompt regress      Re-run the eval suite and gate on score regression
  /oc-prompt baseline     Freeze the current scores as the regression baseline
  /oc-prompt drift        Re-run the frozen baseline to detect prompt/model drift

  UTILITIES
  /checkpoint             Show checkpoint status

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Type any command to begin. /oc-prompt to see this again.
```

---

## How This Skill Fits the Build Pipeline

```
oc-claude-api (model routing) ──pins the model an eval runs against──┐
                                                                     │
oc-prompt-ops owns:                                                  ▼
   prompts/<name>/<version>/prompt.md        ┌────────────────────────────┐
   prompts/<name>/eval/inputs.jsonl     ────►│  /oc-prompt eval           │
   prompts/<name>/eval/expected.jsonl        │  run prompt × goldset       │
   prompts/<name>/eval/eval.yaml             │  grade → scorecard          │
                                             └─────────────┬──────────────┘
                                                           │ score delta
                                       ┌───────────────────┴────────────────┐
                                       ▼                                     ▼
                          /oc-prompt regress (CI gate)         /oc-prompt drift (scheduled)
                          block merge on score drop            flag model/prompt drift
                                       │
              ┌────────────────────────┼────────────────────────┐
              ▼                        ▼                         ▼
        oc-agent-forge          oc-rag-forge             oc-claude-api migrate
        (agent eval harness)    (gen-prompt eval)        (eval-gate the diff PR)
```

The skill's output is an **evaluated, versioned prompt** plus a regression
suite. Sibling skills consume that harness: `oc-agent-forge` and `oc-rag-forge`
run their own goldsets *through* this skill's eval runner rather than
reinventing one, and `oc-claude-api migrate` gates its model-migration diff on
the prompt's eval scores not regressing on the new model.

---

## Principle 1: Prompt-as-Code

A prompt that lives only in a SaaS console or interpolated inside a function is
an undiffable, unversioned production dependency. Prompt Ops moves it into the
repo and treats it like any other source artifact.

**The convention** (full layout in `references/prompt-versioning.md`):

```
prompts/
└── <prompt-name>/
    ├── CHANGELOG.md            # one entry per version, with the eval delta
    ├── v1.0.0/
    │   └── prompt.md           # the prompt text — the only source of truth
    ├── v1.1.0/
    │   └── prompt.md
    └── eval/
        ├── inputs.jsonl        # eval inputs (one JSON object per line)
        ├── expected.jsonl      # expected outputs / grading targets, keyed by id
        └── eval.yaml           # rubric: which grader, thresholds, judge config
```

Four rules make it "code":

1. **Source-controlled.** The prompt text lives in `prompt.md`, reviewed in a PR
   like any other change. No live console edits — those are invisible to git and
   to the eval gate.
2. **Diffable.** A prompt change is a line diff. `/oc-prompt diff v1.0.0 v1.1.0`
   shows the text delta *and* the score delta side by side, so a reviewer sees
   both what changed and what it did.
3. **Lockstep-versioned.** The prompt and the eval set version together. A prompt
   bump that needs new eval cases bumps both in the same commit, so a version is
   always evaluable against its own contemporaneous goldset.
4. **Semver'd by behavior.** PATCH = typo/format fix, no score movement expected.
   MINOR = capability added, scores should hold or rise. MAJOR = behavior changed
   such that the old expected ou