experimental-design-ds

# Experimental Design for Data Science

Experimental design is the discipline of collecting data so that the analysis can answer the intended question. Ronald Fisher, working at the Rothamsted agricultural station in the 1920s, formalized the three pillars -- randomization, replication, and blocking -- that remain the foundation of every modern experiment, from clinical trials to A/B tests on websites. This skill covers experimental design from the data scientist's perspective: planning experiments, ensuring valid causal inference, and avoiding the pitfalls that invalidate conclusions.

**Agent affinity:** fisher (experimental design, ANOVA), tukey (exploratory analysis of experimental results), benjamin (ethical review of experiments)

**Concept IDs:** data-hypothesis-testing, data-confidence-intervals, data-probability-basics, data-sampling-methods

## Fisher's Three Pillars

### 1. Randomization

Random assignment of experimental units to treatment groups ensures that any observed difference is either due to the treatment or due to chance -- not due to confounders. Without randomization, the groups may differ systematically in ways that bias the result.

**Mechanism:** Each unit is assigned to treatment or control by a random process (coin flip, random number generator). This does not guarantee balance -- it guarantees that imbalances are random and quantifiable by probability theory.

**Why it works:** Randomization breaks the association between treatment assignment and all potential confounders, both observed and unobserved. No observational adjustment method can do this for unobserved confounders.

### 2. Replication

Multiple experimental units per group. Replication serves two purposes:

- **Statistical power:** More units means smaller standard errors and greater ability to detect real effects.
- **Generalizability:** Results from one unit might be idiosyncratic. Results replicated across many units are more convincing.

**Replication is not repetition.** Measuring the same unit 10 times is repetition (estimates measurement error). Measuring 10 different units is replication (estimates treatment effect variability).

### 3. Blocking

Grouping experimental units by a known source of variability and randomizing within blocks. If you know that males and females respond differently to a treatment, block by sex and randomize within each block. This reduces unexplained variance and increases power.

**Blocking vs. stratification:** Same concept, different fields. Blocking is the experimental design term (Fisher); stratification is the sampling term.

## The Experimental Lifecycle

| Stage | Goal | Key decisions |
|---|---|---|
| 1. Hypothesis | Define what you want to learn | Primary hypothesis, secondary hypotheses, directional vs. two-sided |
| 2. Metric selection | Define how you will measure the effect | Primary metric, guardrail metrics, sensitivity metrics |
| 3. Power analysis | Determine required sample size | Minimum detectable effect, significance level, power |
| 4. Design | Choose experimental structure | Completely randomized, blocked, factorial, sequential |
| 5. Randomization | Assign units to groups | Simple, stratified, cluster randomization |
| 6. Execution | Run the experiment | Monitor for implementation errors, compliance |
| 7. Analysis | Test the hypothesis | Pre-specified analysis plan, intention-to-treat |
| 8. Interpretation | Draw conclusions | Effect size, confidence interval, practical significance |

## A/B Testing

### The Basic Framework

An A/B test is the simplest randomized experiment: two groups (A = control, B = treatment), one intervention, one primary metric.

**Steps:**

1. Define the metric (conversion rate, revenue per user, time on page).
2. Define the minimum detectable effect (MDE) -- the smallest change worth detecting.
3. Calculate sample size using power analysis.
4. Randomize users into A and B.
5. Run until the pre-specified sample size is reached.
6. Analyze using the pre-specified test (t-test, chi-squared, Mann-Whitney).
7. Report the effect size with a confidence interval.

### Sample Size Calculation

For a two-sample test of proportions (e.g., conversion rates):

n = (Z_{alpha/2} + Z_beta)^2 * (p_1 * (1 - p_1) + p_2 * (1 - p_2)) / (p_1 - p_2)^2

Where:
- Z_{alpha/2} = critical value for significance level (1.96 for alpha = 0.05, two-sided)
- Z_beta = critical value for power (0.84 for power = 0.80)
- p_1 = baseline conversion rate
- p_2 = expected conversion rate under treatment

**Key insight:** Sample size scales with the inverse square of the effect size. Detecting a 1% improvement requires 4x the sample size of detecting a 2% improvement. This is why small effects in large populations require enormous experiments.

### Common A/B Testing Pitfalls

| Pitfall | Problem | Fix |
|---|---|---|
| **Peeking** | Checking results before full sample size; inflates false positive rate | Use sequential testing methods or commit to fixed horizon |
| **Multiple comparisons** | Testing many metrics inflates family-wise error | Bonferroni correction, FDR control, or pre-specify one primary metric |
| **Network effects** | One user's treatment affects another user's outcome (social platforms) | Cluster randomization (randomize by geographic region or social cluster) |
| **Novelty/primacy effects** | Users react differently to new features initially | Run experiments long enough for the effect to stabilize (2-4 weeks) |
| **Sample ratio mismatch** | Unequal assignment ratios indicate implementation bugs | Check actual vs. expected ratio before analyzing results |
| **Simpson's paradox** | Aggregate result reverses when stratified by a confounder | Pre-stratify or check for compositional changes across groups |

## Randomization Strategies

| Strategy | Description | Use when |
|---|---|---|
| **Simple** | Coin flip per unit | Default; works for large samples |
| **Stratified** | Randomize within strata of important covariates | Known prognost