model-validation

# Model-Validation Skill

## Purpose

This skill pressure-tests the **validation study for an engineer-built medical-imaging model** — the
common case where a clinical team receives a trained segmentation / classification / detection model
from an engineering collaborator and must validate it and write it up. It is the imaging-model
specialization of `/design-study`: where `design-study` covers general validity and
`design-ai-benchmarking` covers AI-versus-human-expert reader studies, this skill owns the
**partition, leakage, reproducibility, and metric-selection mechanics** that decide whether a reported
Dice / AUROC / sensitivity is trustworthy.

It is **advisory and deterministic-audit only**. It writes decision notes and runs a stdlib gate on the
split table; it never builds, trains, or alters the model, and it never replaces MONAI / nnU-Net /
TorchIO — those produce the model, this validates and publishes it.

## When to use
- A trained imaging model (in-house, vendor, or open-weights) needs a clinical-validation study designed
  or audited before submission.
- You have, or can produce, the **split-assignment table** (which patient went to train / val / test).

## When NOT to use
- Building or training the model → out of scope (integrate MONAI / nnU-Net).
- AI-versus-human-expert reader study → `/design-ai-benchmarking`.
- LLM / MLLM evaluation → `/mllm-eval` (when available).
- General study/validity review → `/design-study`.
- Statistical execution (DeLong, ICC, bootstrap CIs, calibration tables) → `/analyze-stats`.
- Item-by-item reporting-guideline audit of a finished manuscript → `/check-reporting`.
- Reviewing a finished manuscript → `/self-review` or `/peer-review` (which load the MD0–MD11
  reviewer-side probe).

## Workflow

The design/audit rationale behind Phases 2–7 — the full data-leakage taxonomy, the
internal-vs-genuine-external validation ladder, comparator design, single-run vs multi-seed
variance, test-set sizing, and the CLAIM 2024 / TRIPOD+AI / STARD-AI reporting map — is in
`${CLAUDE_SKILL_DIR}/references/validation_design.md` (load on demand). The patient-disjointness
verdict itself is proven by `scripts/check_split_leakage.py` (Phase 2), not from that prose.

### Phase 1 — Reconstruct the task, the intended-use horizon, and the analysis unit
State the model's task (segmentation / classification / detection), its **intended-use horizon**
(screening, triage, pre-procedure, post-hoc), the **single headline metric** the conclusion leans on,
and the **analysis unit** the metric must respect (per-patient vs per-lesion vs per-image). Everything
downstream is read against this.

### Phase 2 — Leakage audit (the deterministic gate, run first)
The most metric-inflating defect is a split that is **not disjoint at the patient level**. Produce the
emitted split-assignment table (`patient_id,split`) and run the gate:

```bash
python3 ${CLAUDE_SKILL_DIR}/scripts/check_split_leakage.py \
  --splits <split_assignment.csv> --out qc/split_leakage.json --strict
```

`PATIENT_OVERLAP` (a patient in ≥ 2 partitions) and `MISSING_SEED` (an unreproducible split) are
proven by set arithmetic — not heuristics. Then walk the rest of the **leakage taxonomy** (Kapoor &
Narayanan, *Patterns* 2023) that the table cannot show: **preprocessing-before-split** (normalisation,
resampling, foundation-model embeddings, or ComBat harmonisation fit on the whole cohort before
partitioning), **site / scanner / burned-in-label shortcuts**, and **temporal leakage** (a random split
where future and past coexist). The decisive question: *could any value used in training have been
computed only with knowledge of a test case?*

### Phase 3 — Validation tier (internal split vs genuine external)
Classify the evidence honestly: apparent → internal random split → cross-validation → temporal →
geographic / external (different site, scanner, vendor) → multi-site external. Cross-validation and
bootstrap are development-time optimism corrections, **not** external validation. Flag a generalisability
or deployment claim that outruns an internal-only design, and "developed with external validation" where
the single external set was used for tuning. Also confirm the test set was touched **once** — no
architecture search, hyperparameter sweep, early-stopping, or operating-point / threshold choice read
the test set.

### Phase 4 — Comparator design
Decide what the model is compared against: clinical-only baseline, incremental value over an existing
score, or reader comparison. For a reader comparison, hand the rubric / inter-rater design to
`/design-ai-benchmarking`.

### Phase 5 — Metric selection (task-correct, prevalence-aware)
Match the metric to the task and the prevalence (Metrics Reloaded — Maier-Hein & Reinke et al.,
*Nat Methods* 2024): **segmentation** pairs an overlap metric (Dice / IoU) with a **boundary metric**
(HD95 / Normalised Surface Distance), per-structure not only global; **classification under imbalance**
reports **AUROC and AUPRC with CIs** plus sensitivity / specificity and prevalence-dependent PPV / NPV,
never bare accuracy on a balanced set; **detection** reports FROC / mAP with the IoU match criterion
stated. Require the headline metric as **mean ± SD across ≥ 3 seeds / runs**, or a fixed reported seed
with the determinism caveat. The per-case metric computation + the deterministic metric-reporting gate
live in `/model-evaluation` (which emits the per-case table for `/analyze-stats`). For **interactive /
promptable segmentation** (SAM2 / MedSAM2 / nnInteractive) the metric set adds the interaction axis —
number of clicks / interactions-to-threshold (NoC), initial-vs-converged Dice, and per-case
interaction / inference time (`/model-evaluation --task interactive`). When the evaluation runs two arms
(simulated prompting + human-operator validation), record **protocol fidelity** across arms — identical
prompt types, stopping rule, target threshold, and seeds — as an explicit validity item: arm-to-