Generalization: Measuring What a Model Actually Learned

A supervised model is fitted by minimising its error on the training examples (the previous lesson in this course sets this up). That makes training error the one number guaranteed to look good: the fitting procedure chose the parameters specifically to make it small.

The quantity you care about is different. Generalization error is the expected error on new examples drawn from the same process as the training data. You can never compute it exactly, because it is an average over data you do not have. You can only estimate it, and every estimate needs examples the model did not use for fitting.

Key idea: The gap between training error and generalization error is the model's optimism. A decision tree grown to full depth routinely scores 100% on its own training set. On the scikit-learn breast cancer dataset, one does exactly that, then scores 91.7% under five-fold cross-validation. Only the second number says anything about the next patient.

The standard protocol divides the data before anything else happens.

The validation set exists because choosing is itself a form of fitting. Try fifty settings and keep the best validation score, and part of that score is luck: you selected the setting that happened to suit those particular validation examples. The test set is the protection against this. It stays locked until every decision is made.

Gotcha: A test set consulted twice is a validation set. If you look at the test score, change something, and look again, the final number is optimistic in exactly the way the test set was meant to prevent. There is no way to un-see it; the only fix is fresh data.

Here is overfitting measured rather than described. Twenty training points were drawn from a sine curve plus noise with standard deviation 0.3, and polynomials of rising degree were fitted to them, then scored on 1,000 fresh points.

Training error falls at every step. Test error falls until degree 3, reaching 0.322, barely above the noise level of 0.3 that no model can beat, then climbs back up. Past that point the extra flexibility is spent memorising the particular noise in twenty points.

The same experiment continues to degree 15: a polynomial with 16 coefficients fitted to 20 points.

This is the general shape to remember. Training error is almost always monotone in model flexibility. Held-out error is U-shaped, and its minimum sits wherever the data can support the flexibility, which depends on how much data you have and how noisy it is. Rerun with 2,000 training points and the degree-9 fit scores 0.301 on test data, essentially at the noise floor, because there are now far too many points to memorise.

For squared error, the expected test error at a point decomposes into three parts:

The expectations run over training sets you might have drawn. Bias is how far the average fitted model sits from the truth: a straight line fitted to a sine is wrong however much data you give it. Variance is how much the fit changes from one training sample to another: the degree-15 polynomial is a different wild curve every time. Noise is the floor.

Instead of restricting the model family, you can keep it flexible and charge for large coefficients. Ridge regression adds a penalty on the squared size of the weights:

Lasso uses absolute values, $\alpha \sum_j |w_j|$, which pushes some weights to exactly zero and so doubles as feature selection. The strength $\alpha$ is a hyperparameter: it is not learned by the fit, it is chosen on validation data.

from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures, StandardScaler
from sklearn.linear_model import RidgeCV

model = make_pipeline(
    PolynomialFeatures(degree=15),
    StandardScaler(),
    RidgeCV(alphas=[1e-4, 1e-3, 1e-2, 0.1, 1, 10]),  # picks alpha by internal cross-validation
)
model.fit(X_train, y_train)

The scaler matters: the penalty treats all weights alike, so features must be on comparable scales or the penalty falls unevenly.

A single validation split wastes data and gives a noisy estimate, especially on small datasets. k-fold cross-validation splits the data into $k$ folds, trains $k$ times, and holds out a different fold each time.

Every example is used for validation exactly once, and you get $k$ scores instead of one. Report their mean and spread; a model at 0.96 plus or minus 0.02 is not clearly better than one at 0.95.

Gotcha: If you tune hyperparameters with cross-validation and then report that same best score, it is optimistic for the reason given earlier: selection fits the folds. Keep a separate test set, or use nested cross-validation, with an inner loop for tuning and an outer loop for scoring. For time-ordered data, shuffled folds leak the future into the past; that case has its own lesson in the financial machine learning course.

Every step that learns from data, including scaling and feature selection, must learn from the training rows only.

Data leakage is any path by which information from the evaluation data reaches the fitted model. scikit-learn's own documentation demonstrates it on data that contains no signal at all: 200 examples, 10,000 random features, random labels. Selecting the 25 "best" features on the full dataset, then splitting, produces a model that looks useful.

The 76% is fiction: the selector had already seen the test labels and chose features that happened to correlate with them. Done correctly, accuracy is 50%, which is chance for two balanced classes. The fix is structural, not a matter of care:

from sklearn.pipeline import make_pipeline
from sklearn.feature_selection import SelectKBest
from sklearn.ensemble import HistGradientBoostingClassifier

pipe = make_pipeline(SelectKBest(k=25), HistGradientBoostingClassifier())
pipe.fit(X_train, y_train)          # selector sees training rows only
print(pipe.score(X_test, y_test))   # about 0.5, the truth

Leakage is not a beginner's mistake that experts outgrow. Kapoor and Narayanan's 2023 survey in Patterns collected earlier reviews across scientific fields and found leakage documented in at least 294 papers across 17 disciplines, in some cases producing wildly overoptimistic conclusions.

The common routes:

• Preprocessing on the full data: scaling, imputation, feature selection or oversampling fitted before the split.
• Duplicates across the split: the same patient, user or near-identical image in both training and test.
• Target leakage: a feature that is only known because of the outcome, like "days in hospital" when predicting admission.
• Temporal leakage: training on rows from after the period you test on, so the model effectively knows the future.

In practice: Suspect leakage whenever a score is surprisingly good. Check which features drive the predictions; a leak usually shows up as one feature that is far too informative. Then split by entity (patient, customer) or by time rather than by row whenever the deployed model will face new entities or future dates.