statistical-modeling

# Statistical Modeling

Statistical modeling is the practice of fitting mathematical structures to data in order to quantify relationships, test hypotheses, and make predictions. Unlike machine learning, which optimizes prediction, statistical modeling privileges interpretability and inference -- understanding *why* variables relate, not just *that* they do. Leo Breiman's "two cultures" paper (2001) crystallized this distinction. This skill covers the inferential tradition while acknowledging where the two cultures overlap.

**Agent affinity:** tukey (EDA and diagnostics), fisher (experimental design and ANOVA), breiman (model comparison)

**Concept IDs:** data-hypothesis-testing, data-confidence-intervals, data-correlation, data-normal-distribution

## The Modeling Workflow

| Stage | Goal | Key operations |
|---|---|---|
| 1. Formulation | Define the question as a model | Specify response variable, predictors, functional form |
| 2. Exploration | Understand data structure | Scatterplots, correlation matrices, distribution checks |
| 3. Fitting | Estimate parameters | OLS, MLE, MCMC, IRLS depending on model class |
| 4. Diagnostics | Check assumptions | Residual plots, Q-Q plots, leverage, VIF |
| 5. Inference | Draw conclusions | Confidence intervals, hypothesis tests, effect sizes |
| 6. Selection | Compare models | AIC, BIC, cross-validation, likelihood ratio tests |
| 7. Communication | Report results | Effect estimates with uncertainty, not just p-values |

## Linear Regression

### The Model

y = beta_0 + beta_1 * x_1 + beta_2 * x_2 + ... + beta_p * x_p + epsilon

where epsilon ~ N(0, sigma^2) independently. The betas are estimated by ordinary least squares (OLS), minimizing the sum of squared residuals.

### Assumptions (LINE)

| Assumption | Check | Violation consequence |
|---|---|---|
| **L**inearity | Residual vs. fitted plot -- no pattern | Biased estimates, meaningless coefficients |
| **I**ndependence | Study design, Durbin-Watson test | Underestimated standard errors, inflated significance |
| **N**ormality of residuals | Q-Q plot, Shapiro-Wilk test | Unreliable confidence intervals and p-values (less critical for large n by CLT) |
| **E**qual variance (homoscedasticity) | Scale-location plot, Breusch-Pagan test | Inefficient estimates, unreliable standard errors |

### Interpretation

- **beta_j:** The expected change in y for a one-unit increase in x_j, holding all other predictors constant.
- **R-squared:** Proportion of variance in y explained by the model. Not a measure of model quality alone -- a high R-squared with violated assumptions is meaningless.
- **Adjusted R-squared:** Penalizes for number of predictors. Always use this for model comparison.

### Multicollinearity

When predictors are highly correlated, coefficient estimates become unstable. Variance Inflation Factor (VIF) quantifies this: VIF > 5-10 indicates problematic collinearity. Remedies: drop a predictor, combine predictors via PCA, or use regularization (ridge regression).

## Logistic Regression

### The Model

For binary outcome y in {0, 1}:

log(p / (1 - p)) = beta_0 + beta_1 * x_1 + ... + beta_p * x_p

where p = P(y = 1 | x). The left side is the log-odds (logit). Parameters are estimated by maximum likelihood.

### Interpretation

- **exp(beta_j):** The odds ratio for a one-unit increase in x_j, holding other predictors constant. An odds ratio of 1.5 means 50% higher odds of the outcome.
- **Predicted probability:** p = 1 / (1 + exp(-(beta_0 + beta_1 * x_1 + ...))). The sigmoid function maps the linear predictor to [0, 1].
- **No R-squared analog that works well.** Use pseudo-R-squared measures (McFadden, Nagelkerke) with caution. Prefer ROC-AUC or calibration plots for assessing fit.

### Assumptions

- Observations are independent.
- The log-odds are a linear function of the predictors (check with partial residual plots).
- No perfect multicollinearity.
- No assumption of normality or equal variance -- this is not linear regression with a binary outcome.

## Generalized Linear Models (GLMs)

Logistic regression is one instance of the GLM framework. The general structure:

| Component | Role |
|---|---|
| **Random component** | Distribution of y (Normal, Binomial, Poisson, Gamma, ...) |
| **Systematic component** | Linear predictor eta = X * beta |
| **Link function** | g(mu) = eta, connecting the mean to the linear predictor |

### Common GLMs

| Response type | Distribution | Link | Model name |
|---|---|---|---|
| Continuous | Normal | Identity | Linear regression |
| Binary | Binomial | Logit | Logistic regression |
| Count | Poisson | Log | Poisson regression |
| Count (overdispersed) | Negative binomial | Log | Negative binomial regression |
| Positive continuous | Gamma | Log or inverse | Gamma regression |
| Proportion (not 0/1) | Beta | Logit | Beta regression |

### Poisson Regression

For count data: log(mu) = beta_0 + beta_1 * x_1 + ... Assumes the mean equals the variance (equidispersion). When variance > mean (overdispersion), use negative binomial or quasi-Poisson. Always check for overdispersion.

## Analysis of Variance (ANOVA)

### Purpose

ANOVA tests whether group means differ. It is a special case of linear regression where all predictors are categorical.

### One-Way ANOVA

- **Null hypothesis:** mu_1 = mu_2 = ... = mu_k (all group means are equal).
- **Test statistic:** F = (between-group variance) / (within-group variance).
- **Assumptions:** Independence, normality within groups, equal variances (Levene's test).
- **Post-hoc:** If the F-test rejects, pairwise comparisons identify which groups differ. Use Tukey's HSD or Bonferroni correction to control family-wise error rate.

### Two-Way ANOVA

Adds a second factor and their interaction. The interaction term tests whether the effect of one factor depends on the level of the other. Always plot the interaction (mean response by factor A, colored by factor B) before interpreting the F-test.

### ANOVA as Regression

One-