Overfitting and Underfitting
The central failure of machine learning. Add bends to a curve until it passes through every training point — and gets dramatically worse at everything else.
Read first: Train/Test Split
This is the failure that catches everyone, and it is the reason the previous lesson insisted you hold data back. A model can get better and better at the examples it studied while getting steadily worse at everything else — and nothing in the training numbers warns you it is happening.
Too simple
Start at the easy end. A model that is too simple cannot represent the pattern even if you show it a million examples. Set the slider to zero bends and you get a flat line: the model has learned exactly one thing, the average score, and applies it to everybody.
That is underfitting. It is easy to spot, because the model is obviously bad at the data you trained it on — which means the training numbers tell you about it honestly.
Too complicated
Now the other end, which is where the trouble lives. Give the curve enough bends and it can wander through the training points almost exactly. Its error on them collapses toward zero.
But it did not learn the pattern. It learned the specific students, noise and all — every random high scorer and unlucky low one treated as a rule to be honoured. Show it someone new and it has nothing useful to say.
Sixty students: hours practised against test score. The model only gets to see the fourteen filled dots; the forty-six hollow ones are held back. Below, the error on each group at every complexity you could have chosen.
- 14 students it learns from
- 46 held back
- the real pattern, which the model never sees
3 bends allowed
The lowest error on students it has never seen. Notice it does not pass exactly through a single training dot, and that is fine.
Training error
4.1over the 14 students it studied
Test error
4.3over the 46 it has never seen
The gap
0.2The gap between these two numbers is overfitting. It has no other definition. Numbers the model had to learn: 4.
The gap is the definition
Drag from left to right and watch the two error lines separate. That separation is not a symptom of overfitting; it is overfitting. There is no other definition worth carrying around.
Underfitting
Both errors are high. The model is too simple to capture the pattern, and it is honest about that.
Overfitting
Training error is tiny and test error is large. The model looks excellent and is useless.
Notice the dashed line — the real pattern the model never gets to see. At three bends the fitted curve sits close to it. At twelve it is chasing individual dots and has wandered off the truth entirely, in places predicting scores that could not exist.
Why training error cannot help you
Here is the part that matters most, and it is worth stating carefully. As you add bends, training error drops from about 28 points to under 2 and essentially never stops improving. If you chose your model by that number alone, you would pick the most complicated one available, every single time.
There is no wobble in the training line that says “stop here.” The warning simply is not in that number, and no amount of staring at it will make it appear.
Key takeaways
- Underfitting: too simple to capture the pattern, and honest about it — both errors stay high.
- Overfitting: it memorised the examples, noise and all. Training error tiny, test error large.
- The gap between training and test error is not a symptom of overfitting. It is the definition.
- Training error essentially never stops improving, so choosing by it always picks the most complex model.
- Held-back data is the only instrument that can see the turn. That is what it is for.
Quick check
Answer these to unlock the next chapter — 3 of 4 to pass. You can retake it anytime.
Answer every question to check.
Make a free account to read on
Every chapter is free — an account is how your progress, XP, and streak follow you from your laptop to your phone, and how you show up on the leaderboard. No payment, no trial.