Train/Test Split
A model graded on the data it studied will always flatter itself. Re-roll the split and watch the score you would have reported swing by six points.
There is one rule in machine learning that everything else leans on, and it is this: a model must never be graded on the examples it studied. This lesson is why that rule exists, and what it costs to follow it.
Grading your own homework
Imagine revising for an exam using a set of past questions, then being tested on those exact questions. You would score brilliantly, and the score would tell nobody anything about whether you understood the subject.
A model is worse than you at this, because it has a perfect memory. Give it enough capacity and it can store the answers outright — and then report a near-perfect score on them, honestly and uselessly.
Holding some back
The fix is simple enough to state in one sentence. Before training, put some of your data in a drawer. Train on the rest. Grade on the drawer.
The data you train on is the training set. The data in the drawer is the test set. The test set has to stay in the drawer — if you look at it, tune your model, and look again, you have started revising for those questions too, just more slowly.
The score is a lottery ticket
Now the part nobody tells beginners. Which examples go in the drawer is decided at random — and that choice changes your score.
Below, the same sixty students are split many different ways at each setting. You only ever perform one split in real life, so you only ever see one of those dots. Drag the slider and watch how much they disagree.
The strip at the top is the sixty students, coloured by whether this particular shuffle held them back. Below, the error you would have reported — the band covers every one of 12 different shuffles, and the dots are the individual tickets you could have drawn.
- trained on
- held back
holding back 10% — 54 to learn from, 6 to test on
Six students in the test set. The score you would report swings across a wide range depending only on which six — and you run this once, so you would never find out which you got.
If you had shuffled differently
2.4worst 5.7 − best 3.3 = 2.4 points, across 12 shuffles of the same 60 students
You run this once. Whichever ticket you drew is the number you would have written down and believed.
Training error
4.1Lower than the test score — the model flattering itself on work it has already seen.
Shuffles under 4 points
3/12How many shuffles would have let you claim you beat 4 points. With a small test set that is luck, not skill.
How much to hold back
At a 5% test set the reported score swings by nearly six points depending purely on which three students landed in the drawer. You would never learn that. You would run it once, write down whatever you got, and believe it.
Hold back more and the estimate steadies, because you are averaging over more people. By 70% the spread is about a fifth of what it was. That is the trade: a bigger test set buys you a more trustworthy number.
In practice people hold back somewhere between 20% and 30%, which is a compromise rather than a law. Now you know what is being traded on each side of it.
One more thing visible in the chart: the training error line sits below the test line at every single setting. That gap never closes, and it is the model flattering itself. Never quote a training score as though it were a measure of how good your model is.
Key takeaways
- Never grade a model on data it trained on. It has a perfect memory and will score brilliantly for no reason.
- Put some data in a drawer before training, and keep it there.
- Which examples land in the drawer is random, and that changes your score — a small test set makes the number a lottery.
- A bigger test set buys a more trustworthy estimate. That is the whole trade.
- Training error sits below test error at every split. It always flatters, so never quote it.
Quick check
Answer these to unlock the next chapter — 3 of 4 to pass. You can retake it anytime.
Answer every question to check.
Make a free account to read on
Every chapter is free — an account is how your progress, XP, and streak follow you from your laptop to your phone, and how you show up on the leaderboard. No payment, no trial.