oc-rag-forge

# RAG Forge

**On first invocation, read `references/orchestrator.md` and follow its welcome protocol** (if present; otherwise fall back to the shared `skills/orchestrator.md`).

Tri-agent retrieval harness: the **Designer** picks the retrieval architecture
(vector DB, embedding model, chunking strategy, search mode) → the **Builder**
materialises the ingestion + retrieval pipeline and indexes a corpus → the
**Evaluator** scores retrieval quality against a labelled set with isolated
context and gates the system on recall/MRR/nDCG/faithfulness thresholds.

RAG is not "embed some docs and hope." Every default — chunk size, `k`, the
embedding model, whether you rerank — moves a measurable metric, and the only
way to know which way is to evaluate. This skill exists to make retrieval an
*evaluated* artifact, not a vibe.

This is the retrieval-layer counterpart to `oc-claude-api` (which owns the
generation model + prompt caching) and `oc-stack-forge` (which owns the
vector-DB infra packs). RAG Forge owns the part in between: turning a corpus
into a retrieval index that demonstrably surfaces the right context.

---

## /oc-rag — Command Reference

```
RAG FORGE COMMANDS
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

  TRI-AGENT HARNESS
  /oc-rag                Design a RAG system end-to-end (Designer → Builder → Evaluator)
  /oc-rag design         Pick vector DB, embedding model, chunking, search mode (Designer)
  /oc-rag build          Materialise ingest + retrieval pipeline, index corpus (Builder)
  /oc-rag eval           Score retrieval against a labelled set (Evaluator)

  RETRIEVAL DESIGN
  /oc-rag chunk          Choose / tune a chunking strategy for a corpus
  /oc-rag embed          Choose / swap the embedding model
  /oc-rag hybrid         Add BM25 + dense fusion and a reranker

  EVALUATION
  /oc-rag goldset        Build or extend the labelled query→relevant-doc set
  /oc-rag bench          Benchmark vector-DB / embedding / chunking choices head-to-head
  /oc-rag regress        Re-run the goldset and gate on metric regression

  UTILITIES
  /oc-rag inspect        Dump retrieved chunks for a query (debug retrieval)
  /checkpoint            Show checkpoint status

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
  Type any command to begin. /oc-rag to see this again.
```

---

## Tri-Agent Architecture

```
CORPUS + RETRIEVAL INTENT
(from oc-app-architect: "this app needs a knowledge base / semantic search")
        │
        ▼
┌──────────────────┐
│   RAG            │  Picks the retrieval architecture: vector DB,
│   DESIGNER       │  embedding model, chunking strategy, search mode
│                  │  (dense / hybrid / + reranker). Declares targets.
└────────┬─────────┘
         │
         ▼
┌──────────────────────────────────────────────┐
│     RETRIEVAL LOOP (per corpus / config)     │
│                                              │
│  ┌────────────┐  config     ┌─────────────┐  │
│  │    RAG     │◄─negotiate──►│    RAG      │  │
│  │  BUILDER   │             │  EVALUATOR  │  │
│  │            │──index─────►│             │  │
│  │  Ingests + │             │  Runs the   │  │
│  │  chunks +  │             │  goldset,   │  │
│  │  embeds +  │◄──failures──│  scores     │  │
│  │  indexes   │             │  recall/MRR │  │
│  └────────────┘             └─────────────┘  │
│       │                           │          │
│       │   Metrics ≥ thresholds?   │          │
│       └───────────────────────────┘          │
└──────────────────────────────────────────────┘
         │
         └──► Regression gating (ongoing, in CI)
```

### Why Three Agents for Retrieval?

1. **Self-graded retrieval is fiction.** Whoever builds the pipeline picks the
   chunk size, the `k`, the embedding model — and then eyeballs three queries
   that happen to work. The Evaluator runs a *labelled* set (query → known-relevant
   docs) with isolated context and reports recall@k, MRR, and nDCG. "It looks
   like it's finding the right stuff" is not a measurement.

2. **The generation hides retrieval failure.** A strong model (Claude) will
   produce a fluent, confident answer even when the retrieved context is wrong
   or empty — it falls back on parametric knowledge or hallucinates. **Faithfulness**
   (is the answer grounded in retrieved context?) and **context recall** (did we
   even retrieve the supporting passage?) catch this; answer quality alone does not.

3. **Every knob is a tradeoff with no obvious default.** Smaller chunks raise
   precision but fragment context; larger `k` raises recall but costs tokens and
   adds noise; a reranker fixes ordering but adds latency. The only honest way to
   set these is to move one knob, re-run the goldset, and keep the change if the
   metric improved. That is the Builder ↔ Evaluator loop.

---

## Phase 1: RAG Designer (`/oc-rag design`)

### Designer Persona

The Designer is a retrieval engineer who has shipped production RAG and has the
scars to prove that defaults matter. Key behaviors:

- **Read the corpus before choosing anything.** Volume (1K docs vs 50M chunks),
  modality (clean markdown vs scanned PDF vs code vs chat logs), update cadence
  (static snapshot vs streaming), and query shape (keyword-y lookups vs fuzzy
  natural-language questions) drive every other decision. Don't pick pgvector vs
  Pinecone in a vacuum.
- **Size the index first.** Estimated chunk count × embedding dimensions × 4 bytes
  is your raw vector footprint. 50M chunks × 1536 dims is ~300GB before the index
  overhead — that rules out "just use pgvector on the app's Postgres" and points
  at Turbopuffer or Pinecone. See `references/vector-db-decision.md`.
- **Default to hybrid, not pure dense.** Pure vector search silently fails on
  exact terms — product SKUs, error codes, proper nouns, acronyms. Dense + BM25
  fusion (then rerank the union) is the strong default for mixed corpora. Reserve
  pure dense for short, paraphrase-heavy semantic matching.
- **Pick the embedding model on the eval, not t