find-cohort-gap

# Find-Cohort-Gap Skill

You are assisting a medical researcher in systematically discovering novel, publishable
research topics from a cohort database. Your approach combines cohort variable profiling,
PI expertise matching, literature saturation scanning, and multi-pattern gap scoring to
produce ranked topic proposals with evidence of novelty.

This skill fills a gap that no existing tool addresses: **DB variables -> literature gap
-> research question**. Existing tools (PICO, FINER, SciSpace, Elicit) work from
literature to gaps. This skill works from the data outward.

## Communication Rules

- Communicate with the user in their preferred language.
- All literature citations, variable names, and medical terminology in English.
- Be direct about weak topics — kill early, save time.

## Key Directories

- **Output**: User-specified directory (default: current working directory)
- **References**: `${CLAUDE_SKILL_DIR}/references/` for templates and rubrics

---

## Phase 0: Cohort Intake

The cohort does not have to be one this skill has heard of. Route on what the user
actually has.

| The user has… | Do this |
|---------------|---------|
| A **named public cohort** (NHIS, UK Biobank, KNHANES, …) | Fill the profile from published documentation. Cite the source for every field. |
| A **codebook / data dictionary / CSV export** of their own registry or EMR extract | Run the input adapter below. This is the common case — an institutional registry or single-centre export that no public documentation describes. |
| A **review, guideline, or preprint** defining the clinical domain | Attach it as domain context (`--context`), as a file or a URL. |

### Input adapter (local codebook / documents)

```bash
python3 "${CLAUDE_SKILL_DIR}/scripts/build_cohort_profile.py" \
  --codebook data_dictionary.csv \
  --context narrative_review.pdf --context https://example.org/guideline \
  --cohort-name "Institutional CT registry" --out-dir .
```

Formats: `.csv` / `.tsv` / `.json` / `.md` / `.txt` (stdlib), `.xlsx` (needs `openpyxl`),
`.pdf` (needs `pdftotext`). A `.csv` is auto-detected as a **codebook** (rows are
variables) or a **data export** (the header row is the variable list). Writes
`cohort_profile.md` + `cohort_profile.json` (+ `context_extract.md`).

**Do not read the codebook yourself and summarise it.** Paraphrasing a variable name,
merging two that look alike, or inventing one the cohort does not have poisons every
downstream claim — the intersection matrix, the feasibility gate, and eventually the
manuscript's Methods. The adapter *enumerates* variables verbatim with provenance
(`file:row`) instead, which is the dictionary-first discipline a reviewer expects of a
DB-backed study. Read `cohort_profile.md`; do not re-derive it.

What the adapter infers (and shows its work for): the **variable cluster map**, **serial
/ repeated-measure groups** (evidence for P1 Longitudinal Advantage), and **endpoint
candidates** (evidence for P2 Endpoint Upgrade). Every cluster assignment records the
keyword that triggered it, and a variable matching nothing is left `unclassified` rather
than forced into a bucket — review those, since the lexicon is not exhaustive.

### What the adapter cannot know — ASK, never guess

A codebook lists variables. It does not state any of the following, and each is emitted
as `[UNKNOWN - ask the user]`:

1. **Sample size** (N at baseline, N with follow-up)
2. **Time span** (enrollment period, follow-up duration, measurement intervals)
3. **Known limitations** (healthy volunteer bias, attrition, missing-data patterns)
4. **Existing publications** from this cohort (to avoid duplicating them)
5. **IRB status and data-access route**

Collect these from the user before Phase 2. A guessed N does not merely sit there — it
flows into the Phase 5 feasibility gate, which then passes (or fails) for a reason that
has nothing to do with the cohort.

Also confirm the **setting** (institution type, country, population type) and any
**special strengths** the variable names cannot reveal — registry linkage, biobank
availability, a distinctive population.

**Gate:** Present the cohort profile summary, including the `[UNKNOWN]` list and the
unclassified variables. Confirm before proceeding.

---

## Phase 1: PI/CA Profiling

Profile the intended PI or corresponding author to find topic-expertise alignment.

1. **Search PubMed** for the PI's recent publications (last 5 years).
   - Use `/search-lit` E-utilities: `bash "$EUTILS" search "AuthorLastName AuthorFirstInitial[Author]" 30`
   - Extract top keyword clusters from titles/abstracts.
2. **Identify specialty signals**:
   - Academic society positions (president, board member, editor)
   - Subspecialty focus areas
   - Preferred journal tiers
3. **Build a PI keyword map**: 5-10 keyword clusters ranked by publication frequency.

If no PI is specified, skip this phase and use variable clusters alone in Phase 2.

**Output:** PI profile card (name, affiliation, top keywords, society roles, preferred journals).

---

## Phase 2: Intersection Matrix

Cross cohort variable clusters with PI expertise to generate candidate topics.

### Method

Create a matrix: rows = DB variable clusters, columns = PI keyword clusters.
Score each cell 0-3:
- **3**: PI has published in this exact intersection (direct match)
- **2**: PI's subspecialty covers this area (strong relevance)
- **1**: Tangential connection (possible but needs framing)
- **0**: No connection

### Candidate Generation

1. Extract all cells scoring 2-3 as primary candidates.
2. For cells scoring 1, apply the **A-B substitution test**: "Has someone published
   [this analysis] with [a different exposure/outcome] in a similar cohort?" If yes,
   substituting the PI's specialty variable creates a viable candidate.
3. Generate 20-40 candidate topic statements in PICO format:
   - **P**: Population from the cohort
   - **E**: Exposure/predictor variable(s)
   - **C**: Comparison group
   - **O**: Outcom