dataset-curation

# Dataset Curation Methodology

You are helping a researcher curate, analyze, or expand a dataset with attention to bias, fairness, and quality.

## Step 1: Distribution Analysis

Before any curation action, understand the current state:

### Per-Class Distribution
- Count instances per class/label/tag
- Compute imbalance ratio (max_count / min_count)
- Identify severely underrepresented classes (< 5% of max class)
- Visualize: bar chart of class frequencies sorted by count

### Co-occurrence Analysis
- Build co-occurrence matrix: which labels appear together
- Identify spurious correlations (e.g., "violence" always co-occurs with "male")
- Check for label leakage between splits

### Metadata Distribution
- Source diversity: how many sources/movies/documents contribute
- Temporal distribution: are all time periods represented?
- Content diversity: genre, style, domain coverage

## Step 2: Bias Assessment

For each identified imbalance or correlation:

1. **Is it real-world reflective?** Some imbalances reflect genuine phenomena
2. **Is it harmful?** Would a model trained on this data make unfair predictions?
3. **Is it fixable?** Can we collect more data, resample, or reweight?

### Fairness Dimensions
Check for bias along relevant protected attributes:
- Gender representation (if applicable)
- Racial/ethnic representation (if applicable)
- Age distribution (if applicable)
- Geographic/cultural diversity (if applicable)

### Bias Metrics
- Demographic parity: equal positive rates across groups
- Equalized odds: equal TPR and FPR across groups
- Representation ratio: group proportion in data vs population

## Step 3: Stratified Sampling

When creating splits (train/val/test):

1. **Primary stratification**: by label/class distribution
2. **Secondary stratification**: by source (prevent source leakage across splits)
3. **Validation**:
   - Chi-squared test for label distribution similarity across splits
   - No source overlap between splits
   - Rare classes have minimum representation in each split

Split ratios depend on dataset size:
- Large (>50k): 80/10/10 or 90/5/5
- Medium (5k-50k): 70/15/15 or 80/10/10
- Small (<5k): k-fold cross-validation preferred

## Step 4: Quality Assessment

For labeled datasets, assess annotation quality:

- **Inter-annotator agreement**: Cohen's kappa, Fleiss' kappa, or Krippendorff's alpha
- **Label noise estimation**: sample and manually verify N labels
- **Edge cases**: identify ambiguous examples that annotators might disagree on
- **Consistency checks**: automated rules for label validity

## Step 5: Expansion Recommendations

If the dataset needs more data:

1. **Priority classes**: which classes benefit most from more data
2. **Source suggestions**: where to find more data for underrepresented classes
3. **Collection strategy**: active learning, targeted scraping, synthetic augmentation
4. **Cost estimation**: time and resources for each approach

## Step 6: Ethical Review Checklist

Before using or publishing any dataset:

- [ ] Content sensitivity: does the data contain sensitive material?
- [ ] Consent: was data collected with appropriate consent?
- [ ] Privacy: are individuals identifiable? Is anonymization needed?
- [ ] Licensing: are data sources used within their license terms?
- [ ] Potential harms: could the dataset be misused?
- [ ] Documentation: is the dataset documented with a datasheet/data card?

## Output Format

Produce:
1. **Distribution report**: per-class counts, imbalance ratios, co-occurrence matrix
2. **Bias findings**: identified biases with severity and actionability
3. **Split recommendation**: stratification strategy with validation results
4. **Expansion plan**: prioritized suggestions for addressing gaps
5. **Ethics checklist**: completed checklist with notes per item