Cross-Validation

The problem with a single split is not that it is biased. Run it a thousand times and the average comes out right. The problem is that you run it once, and one draw from a wide range is a poor way to learn where the middle is.

There is a second cost, quieter and worse. Every student in the drawer is a student the model never learned from. Holding back 20% of a small dataset means training on 80% of what you collected, and you paid for all of it.

Cross-validation fixes both with one idea: stop choosing which examples go in the drawer, and use all of them, in turn.

Deal the sixty students into five blocks of twelve. Hold back block one, train on the other forty-eight, score. Put block one back, hold back block two, train, score. Five rounds, five models, five scores. Average them.

Every student is tested exactly once and trained on exactly four times. Nothing is wasted, and no student is ever tested by a model that studied them. That last clause is the rule from chapter five, and it survives intact — which is the only reason this is legitimate rather than clever.

Step through the five rounds and watch the individual scores: 3.44, 4.67, 3.89, 3.90, then 5.67. The average is 4.31, and that is the number you would report.

But look at the range those five came from. The kindest fold and the cruellest differ by 2.23 points. Same model, same procedure, same sixty students — the only difference is which twelve were being marked. Fold five got a batch this model handles badly.

Most people average the folds and throw the rest away. Do not. The spread across folds is a measurement you cannot get from a single split, and it answers a question the average never touches: how much does this score depend on luck? A model scoring 4.31 give or take 0.1 and a model scoring 4.31 give or take 2.2 are not the same result, and only one of them is worth defending.

Here are five folds on a small dataset bundled with scikit-learn: 178 wines, 13 chemical measurements each, and three grape varieties to tell apart.

import numpy as npfrom sklearn.datasets import load_winefrom sklearn.model_selection import KFold, cross_val_scorefrom sklearn.tree import DecisionTreeClassifier X, y = load_wine(return_X_y=True)   # 178 wines, 13 chemical measurements, 3 grape typesmodel = DecisionTreeClassifier(max_depth=3, random_state=0) folds = KFold(n_splits=5, shuffle=True, random_state=0)   # try n_splits=10scores = cross_val_score(model, X, y, cv=folds) print("each fold:", np.round(scores, 2))print(f"average {scores.mean():.2f}, spread {scores.min():.2f} to {scores.max():.2f}")

The folds score between 0.83 and 0.97, averaging 0.93. Set n_splits to 10 and the average barely moves while the spread widens to 0.72 to 1.00: smaller folds mean noisier individual scores.

The bottom panel makes the payoff concrete. Repeat the whole experiment forty times with forty different shuffles. Report from a single 20% holdout and you land somewhere across a 3.84-point range — as low as 2.59, as high as 6.43, entirely by luck. Report the five-fold average and you land within 0.69 points. The estimate is five and a half times tighter for five times the compute.

Cross-validation costs five times the training. That is the whole objection, and it is decisive more often than tutorials admit.

With a large dataset it stops earning its keep. Hold back 20% of two million rows and the test set has four hundred thousand examples in it — the lottery has already been averaged away, and a single split gives a stable number for a fifth of the work. The rule of thumb is that cross-validation matters most exactly when data is scarce, which is also when a single split is least trustworthy.

There is one more use for this that matters more than honest reporting. When you need to choose something — a depth for a tree, a k for a neighbour vote — you need to compare candidates without burning the test set. Cross-validating on the training data lets you compare as many as you like, and keeps the real test set sealed for the single measurement at the end.