analyze-stats

# Statistical Analysis Skill

You are assisting a medical researcher with statistical analyses for medical research papers.
Generate reproducible code (Python preferred, R when necessary) that produces publication-ready
tables and figures following journal standards for medical imaging research.

## Data Privacy Check

Before reading any data file, check whether it might contain Protected Health Information (PHI):

1. If `*_deidentified.*` files exist in the working directory, use those preferentially.
2. If only raw CSV/Excel files exist (no `*_deidentified.*` counterpart), warn the user (ask in the user's preferred language):
   > "Does this data contain patient identifiers (names, national ID / RRN, contact details, etc.)?
   > If so, please de-identify it first with the `/deidentify` skill."
3. If the user confirms the data is already de-identified or contains no PHI, proceed.
4. **NEVER** display raw PHI values (names, phone numbers, RRN) in your output. If you
   encounter them while reading data, warn the user and suggest running `/deidentify`.

## Reference Files

- **Templates**: `${CLAUDE_SKILL_DIR}/references/templates/` -- reusable analysis scripts
- **Analysis guides**: `${CLAUDE_SKILL_DIR}/references/analysis_guides/` -- on-demand methodology references
- **Table standards**: `${CLAUDE_SKILL_DIR}/references/table-standards/` -- journal-specific table formatting
  - `table-standards.md` -- universal rules, AMA rules, footnote system, mistakes checklist
  - `journal-profiles/` -- YAML profiles per journal (radiology, jama, nejm, lancet, eur_rad, ajr)
  - `table-types/` -- templates per table type (Table 1, diagnostic accuracy, regression, survival/Cox, agreement/reliability, meta-analysis, model comparison, incremental value, reader study (MRMC))
  - `tool-comparison.md` -- R/Python tool comparison and recommended pipelines
- **Figure style**: `${CLAUDE_SKILL_DIR}/references/style/figure_style.mplstyle`
- **Project data**: See CLAUDE.md for data locations under `2_Data/`

Read relevant templates before generating analysis code. For complex analysis types
(regression, propensity score, repeated measures), also load the corresponding guide
from `analysis_guides/` to ensure correct methodology and reporting.

## Workflow

### Phase 1: Data Assessment

1. **Read the data file** (CSV, Excel, TSV, or other tabular format).
2. **Report to the user**:
   - Shape (rows x columns)
   - Column names and inferred types (continuous, categorical, ordinal, binary, datetime)
   - Missing values per column (count and percentage)
   - First 5 rows preview
   - Unique value counts for categorical columns
3. **Identify the analysis unit**: patient, exam, lesion, image, rater, study, etc.

### Phase 2: Analysis Plan

**Precondition (observational studies).** Before proposing an analysis plan for an observational design (cohort, case-control, cross-sectional, registry, or survey), confirm that a literature-grounded variable operationalization exists — a `variable_operationalization.md` from `/define-variables`, or an equivalent codebook-backed definition table. If none exists, **warn** the user and recommend running `/define-variables` first, so exposure / outcome / covariate definitions and cutoffs are citation-backed rather than invented ad hoc from the data dictionary (ad-hoc phenotype/cutoff definitions are a common reviewer-rejection trigger for observational work — see the dictionary-first discipline). This is a WARN, not a hard block: proceed on explicit user confirmation, recording that the operationalization artifact was not available. For stricter projects, treat the missing artifact as a hard stop until `/define-variables` has run. (This mirrors the same precondition already enforced in `/write-protocol` before drafting Methods.)

Based on the data structure and research question, propose an analysis plan:

1. **Auto-detect analysis type** from the table below, or accept user specification.
2. **List specific tests** to be performed.
3. **Identify primary and secondary endpoints**.
4. **State assumptions** that will be checked (normality, homogeneity, independence).
5. **Note any data cleaning** needed (recoding, outlier handling, missing data strategy).
6. **Anchor the estimand to the research question.** If interaction/synergy/effect-modification is the question, the primary estimand is the **interaction parameter itself** (a likelihood-ratio test of the interaction term, or the interaction OR/HR on a single consistent scale) — not a main-effect OR whose CI is then read as "no synergy." If the claim is equivalence or non-inferiority, declare the margin up front (a TOST procedure, or the CI compared against a pre-stated MCID); a non-significant difference is not equivalence without a margin.

7. **Screen every categorical/binary predictor for separation — before fitting anything.**
   A predictor that perfectly predicts the outcome breaks maximum likelihood: no finite MLE
   exists. The failure is silent — `glm` does not error, it returns an odds ratio near 0 (or
   enormous), *p* ≈ 0.99, and an AUC that then gets written into a table. This is routine in
   diagnostic imaging, because the good signs are the pathognomonic ones (T2-FLAIR mismatch,
   the string sign, a halo sign): 100% specificity means an empty cell by construction.

   ```bash
   python3 "${CLAUDE_SKILL_DIR}/scripts/check_separation.py" \
     --data cohort.csv --outcome idh_mutant --auto --strict
   ```

   `COMPLETE_SEPARATION` (an empty cell) and `QUASI_SEPARATION` (a cell below the sparsity
   floor) both halt the plan. The remedy is a **design** decision, not a numerical one:
   Firth's penalised likelihood keeps one model, while a **two-stage rule** — classify the
   sign-positive cases directly, model only the sign-negative remainder — is usually the
   clinically meaningful choice for a pathognomonic sign, because a sign-positive patient is
   already diagnosed and the real question is what to do with everyone