Cross-Validation
Chapter five left you with a score that was partly luck. Rotate which slice is held back, average the five results, and watch a lottery ticket turn into a measurement.
Read first: Train/Test Split
Chapter five ended on an uncomfortable note. You hold data back, you score the model once, and the number you write down depends on which examples happened to land in the drawer. You accepted that because there was nothing else to do. There is something else to do.
One split is one opinion
The problem with a single split is not that it is biased. Run it a thousand times and the average comes out right. The problem is that you run it once, and one draw from a wide range is a poor way to learn where the middle is.
There is a second cost, quieter and worse. Every student in the drawer is a student the model never learned from. Holding back 20% of a small dataset means training on 80% of what you collected, and you paid for all of it.
Cross-validation fixes both with one idea: stop choosing which examples go in the drawer, and use all of them, in turn.
Using every row twice
Deal the sixty students into five blocks of twelve. Hold back block one, train on the other forty-eight, score. Put block one back, hold back block two, train, score. Five rounds, five models, five scores. Average them.
Every student is tested exactly once and trained on exactly four times. Nothing is wasted, and no student is ever tested by a model that studied them. That last clause is the rule from chapter five, and it survives intact — which is the only reason this is legitimate rather than clever.
The same sixty students from chapter five, dealt into five blocks of twelve. Each round holds one block back, trains on the other forty-eight, and scores. Every student is tested exactly once and trained on four times.
round 1 of 5 — holding back fold 1, scoring 3.44
Mean so far
3.44After 1 of 5 rounds. Keep going.
Fold to fold
2.23The gap between the kindest and cruellest fold. Not noise to hide — it is the model telling you how much it depends on which students it saw.
Tighter by
5.5×One split lands anywhere across 3.84 points. Five folds land within 0.69.
The spread is the point, not the average
Step through the five rounds and watch the individual scores: 3.44, 4.67, 3.89, 3.90, then 5.67. The average is 4.31, and that is the number you would report.
But look at the range those five came from. The kindest fold and the cruellest differ by 2.23 points. Same model, same procedure, same sixty students — the only difference is which twelve were being marked. Fold five got a batch this model handles badly.
Most people average the folds and throw the rest away. Do not. The spread across folds is a measurement you cannot get from a single split, and it answers a question the average never touches: how much does this score depend on luck? A model scoring 4.31 give or take 0.1 and a model scoring 4.31 give or take 2.2 are not the same result, and only one of them is worth defending.
The bottom panel makes the payoff concrete. Repeat the whole experiment forty times with forty different shuffles. Report from a single 20% holdout and you land somewhere across a 3.84-point range — as low as 2.59, as high as 6.43, entirely by luck. Report the five-fold average and you land within 0.69 points. The estimate is five and a half times tighter for five times the compute.
When not to bother
Cross-validation costs five times the training. That is the whole objection, and it is decisive more often than tutorials admit.
With a large dataset it stops earning its keep. Hold back 20% of two million rows and the test set has four hundred thousand examples in it — the lottery has already been averaged away, and a single split gives a stable number for a fifth of the work. The rule of thumb is that cross-validation matters most exactly when data is scarce, which is also when a single split is least trustworthy.
There is one more use for this that matters more than honest reporting. When you need to choose something — a depth for a tree, a k for a neighbour vote — you need to compare candidates without burning the test set. Cross-validating on the training data lets you compare as many as you like, and keeps the real test set sealed for the single measurement at the end.
Key takeaways
- Cross-validation splits the data into k blocks and takes turns holding each one back, then averages the scores.
- Every row is tested exactly once and trained on k-1 times, so nothing is wasted and the chapter-five rule is never broken.
- Here five folds scored 3.44, 4.67, 3.89, 3.90 and 5.67 — an average of 4.31 from scores spanning 2.23 points.
- Report the spread across folds, not only the average. It tells you how much the score depends on which rows were held back.
- Across forty repeats a single holdout ranged over 3.84 points and five-fold cross-validation over 0.69 — a five-and-a-half-fold tightening.
- It costs k times the training, and stops being worth it once the dataset is large enough that one split is already stable.
- Use it to choose settings without touching the test set, which stays sealed for one final measurement.
Quick check
Answer these to unlock the next chapter — 3 of 4 to pass. You can retake it anytime.
Answer every question to check.
Make a free account to read on
Every chapter is free — an account is how your progress, XP, and streak follow you from your laptop to your phone, and how you show up on the leaderboard. No payment, no trial.