machine-learning-foundations

# Machine Learning Foundations

Machine learning is the practice of building systems that learn patterns from data and use those patterns to make predictions or decisions on new data. Where statistical modeling (the inference culture) asks "what is the relationship between X and Y?", machine learning (the prediction culture) asks "given X, what is the best prediction of Y?" This skill covers the foundational concepts, algorithms, and workflow of machine learning from the practitioner's perspective.

**Agent affinity:** breiman (algorithm selection, ensemble methods), tukey (feature engineering, EDA)

**Concept IDs:** data-correlation, data-distributions, data-measures-of-spread, data-hypothesis-testing

## The ML Workflow

| Stage | Goal | Key operations |
|---|---|---|
| 1. Problem framing | Define the task precisely | Classification vs. regression vs. clustering; define target variable and success metric |
| 2. Data collection | Assemble training data | Sources, sampling, labeling; ensure data represents the deployment population |
| 3. Feature engineering | Create informative inputs | Domain-driven features, transformations, encoding categoricals |
| 4. Train/test split | Prevent overfitting evaluation | Hold out 20-30% for testing; never touch test set during development |
| 5. Model selection | Choose algorithm family | Based on data size, interpretability needs, problem structure |
| 6. Training | Fit model parameters | Optimization (gradient descent, tree splitting, etc.) |
| 7. Validation | Tune hyperparameters | k-fold cross-validation on training set only |
| 8. Evaluation | Assess on held-out test set | Metrics appropriate to the problem (accuracy, F1, RMSE, etc.) |
| 9. Interpretation | Understand what the model learned | Feature importance, partial dependence, SHAP values |
| 10. Deployment | Put the model in production | Monitoring, drift detection, retraining schedule |

## Supervised Learning

### Classification

The task: given features X, predict a categorical label y.

**Key algorithms:**

| Algorithm | Strengths | Weaknesses | When to use |
|---|---|---|---|
| **Logistic regression** | Interpretable, fast, probabilistic | Linear decision boundary | Baseline; when interpretability matters |
| **k-Nearest Neighbors** | Non-parametric, no training phase | Slow at prediction, curse of dimensionality | Small datasets, low dimensionality |
| **Decision tree** | Interpretable, handles mixed types | Overfits easily, unstable | When interpretability is paramount; as building block for ensembles |
| **Random forest** | Robust, handles high dimensions | Less interpretable than single tree | Default for tabular data |
| **Gradient boosting** | State-of-the-art tabular performance | Prone to overfitting without tuning | Competition-grade tabular prediction |
| **SVM** | Effective in high dimensions | Slow on large datasets, kernel choice | Text classification, small-medium datasets |
| **Neural network** | Learns complex patterns, scales to huge data | Requires large data, expensive, black box | Images, text, sequences, very large datasets |

### Regression

The task: given features X, predict a continuous value y.

Same algorithms apply (linear regression, k-NN regression, decision tree regression, random forest regression, gradient boosting regression, neural network regression). The loss function changes from cross-entropy to squared error (or absolute error, Huber loss, etc.).

### Evaluation Metrics

**Classification:**

| Metric | Formula / Definition | When to use |
|---|---|---|
| **Accuracy** | Correct / Total | Balanced classes only |
| **Precision** | TP / (TP + FP) | Cost of false positives is high (spam detection) |
| **Recall** | TP / (TP + FN) | Cost of false negatives is high (cancer screening) |
| **F1 score** | 2 * Precision * Recall / (Precision + Recall) | Need balance between precision and recall |
| **ROC-AUC** | Area under ROC curve | Ranking quality across thresholds |
| **Log loss** | Negative log-likelihood of predicted probabilities | When calibrated probabilities matter |

**Regression:**

| Metric | Formula / Definition | When to use |
|---|---|---|
| **MSE** | Mean of (y - y_hat)^2 | Default; penalizes large errors |
| **RMSE** | sqrt(MSE) | Same scale as y; more interpretable |
| **MAE** | Mean of |y - y_hat| | Robust to outliers |
| **R-squared** | 1 - (SS_res / SS_tot) | Proportion of variance explained |
| **MAPE** | Mean of |y - y_hat| / |y| * 100 | Percentage interpretation; fails when y near 0 |

## The Bias-Variance Tradeoff

The expected prediction error decomposes into three components:

Error = Bias^2 + Variance + Irreducible noise

- **Bias:** Error from oversimplifying the model. A linear model fit to a quadratic relationship has high bias (underfitting).
- **Variance:** Error from model sensitivity to training data. A deep decision tree memorizes the training set and varies wildly across samples (overfitting).
- **Irreducible noise:** Inherent randomness in the data. No model can reduce this.

**The tradeoff:** Increasing model complexity reduces bias but increases variance. Decreasing complexity reduces variance but increases bias. The optimal model balances both.

**Regularization** controls this tradeoff by penalizing complexity:

| Method | Penalty | Effect |
|---|---|---|
| **Ridge (L2)** | Sum of beta_j^2 | Shrinks coefficients toward zero; keeps all predictors |
| **Lasso (L1)** | Sum of |beta_j| | Shrinks coefficients; sets some exactly to zero (feature selection) |
| **Elastic net** | Alpha * L1 + (1 - Alpha) * L2 | Combines ridge and lasso benefits |
| **Tree depth limit** | Max depth, min samples per leaf | Prevents tree from memorizing noise |
| **Dropout** | Randomly zero out neurons during training | Prevents neural network co-adaptation |
| **Early stopping** | Stop training when validation error increases | Universal; works for any iterative algorithm |

## Cross-Validation

Cross-validation estimates out-of-sample perfor