profile-imaging

# Profile-Imaging Skill

## Purpose

A dataset decides more of a study than the architecture does, and it decides it **first**. Before
anything is preprocessed, split, or trained, a handful of facts are already true about the data, and
each one closes off or opens up a research plan:

- If the target occupies 0.4 % of the volume, accuracy is not a metric — predicting background
  everywhere scores 99.6 %.
- If through-plane spacing runs 1.5–8 mm inside a single institution, resampling is not a default to
  accept quietly; it is the most consequential preprocessing choice in the study, and it is also the
  axis along which an external dataset will differ.
- If the directory named `imagesTs` has no labels, it is not a test set, and the held-out set has to
  come from somewhere else — better known before training than after.
- If the organ volume spans 56–502 mL when normal is roughly 100–250, the cohort contains disease
  that a subgroup analysis should be **pre-specified** for, rather than discovered post hoc.

None of that requires a model, a GPU, or an engineer. It requires reading every file once and
writing down what is there. This skill does that, and then audits the plan against it.

It is the **front door** of the model-engineering lane:
`profile-imaging (describe)` → `/design-study` + `/architecture-zoo` (decide) →
`/preprocess-imaging` (plan the pipeline) → `/model-scaffold` (build) → `/model-validation` →
`/model-evaluation`.

## When to use
- You have a dataset and a task, and need to know what the data will and will not support before
  committing to a plan.
- You inherited a dataset and need its integrity established (labels intact, splits labelled,
  label values as declared) before anyone trains on it.
- You are about to write a Methods section that describes the cohort and its acquisition.

## When NOT to use
- Tabular / clinical variables → `/generate-codebook` (data dictionary) and `/clean-data`.
- Designing the preprocessing pipeline and auditing it for data-stage leakage →
  `/preprocess-imaging` (it consumes what this skill describes).
- Auditing the train/val/test split table → `/model-validation`.
- Choosing an architecture → `/architecture-zoo`. Building the repo → `/model-scaffold`.
- Held-out metrics, calibration, subgroup results → `/model-evaluation` then `/analyze-stats`.

## Workflow

### Step 1 — profile every case

```bash
python3 scripts/profile_imaging_dataset.py \
    --split train:imagesTr:labelsTr \
    --split test:imagesTs \
    --dataset "MSD Task09 Spleen" \
    --declared-labels 0=background,1=spleen \
    --target-label 1 \
    --plan resample=true,reorient=false,loss=dice_ce,metrics=dice+hd95 \
    --out eda/profile.json
```

One record per case: grid, spacing, orientation, intensity percentiles, the label values actually
present, foreground fraction, and target volume in mL. A `--split` given no label directory is
recorded as **unlabelled** — which is itself a finding.

**`--target-label` on a multi-structure atlas.** Foreground defaults to every non-zero index, which
is the whole annotated anatomy. Run a single-organ study against a 15-organ atlas and the reported
fraction describes the upper abdomen, not the target — measured on AMOS22 that is 3.2 % rather than
the spleen's 0.2 %, so the pooled number sits *above* the 1 % imbalance threshold while the real
target sits far below it, and the imbalance verdicts go quiet exactly where the risk is. Naming the
target also makes `LABEL_EMPTY` mean *this case has no spleen*, which a multi-organ label file
otherwise hides behind the other organs. Pass `--target-label all` for a genuinely multi-class
study; leave it out on a multi-structure atlas and the gate raises `TARGET_LABEL_UNDECLARED`.

Requires `nibabel` + `numpy` (it has to open images). The gate below does not.

### Step 2 — gate the profile against the declared plan

```bash
python3 scripts/check_dataset_profile.py --profile eda/profile.json \
    --out qc/dataset_profile.json --strict
```

Stdlib-only, so the audit re-runs anywhere the JSON travels. Verdicts:

| Verdict | Severity | Fires when |
|---|---|---|
| `LABEL_SHAPE_MISMATCH` | Major | label grid ≠ image grid |
| `LABEL_EMPTY` | Major | a labelled case has zero foreground |
| `LABEL_VALUE_UNEXPECTED` | Major | label values outside the declared set |
| `TEST_SET_UNLABELLED` | Major | a split whose name contains test/held-out/external/eval carries no labels |
| `ACCURACY_UNDER_IMBALANCE` | Major | accuracy is planned while the target is a sliver of the volume |
| `LABEL_MISSING` | Minor | a case in a labelled split has no label file |
| `SPACING_HETEROGENEOUS` | Minor | spacing spans ≥ ratio on an axis and no resampling is declared |
| `ORIENTATION_MIXED` | Minor | >1 orientation code and no reorientation declared |
| `INTENSITY_SCALE_INCONSISTENT` | Minor | some cases sit on the HU scale and others do not |
| `EXTREME_IMBALANCE` | Minor | median foreground below the threshold with no Dice-family loss |
| `TARGET_LABEL_UNDECLARED` | Minor | >1 structure declared but no target named, so foreground pools them all |

**The gate flags an undeclared decision, not variability itself.** A dataset with 5× spacing spread
and two orientation codes passes cleanly once resampling and reorientation are declared —
heterogeneity that has been dealt with is not a defect. That distinction is what the challenge card's
clean fixture exists to prove.

`--spacing-ratio` (default 2.0) and `--imbalance-frac` (default 0.01) are **screening defaults, not
published cut-points**: 2× through-plane spacing changes what a fixed-size patch sees, and 1 %
foreground is roughly where plain accuracy stops carrying information. Both are adjustable and both
are printed in the output, so a reader knows what was applied.

### Step 3 — turn the profile into research decisions

The profile is evidence; the decisions are yours, and the ones worth writing down are:

1. **Resampling target** — from the spacing distribution, no