Overfitting and Underfitting

Start at the easy end. A model that is too simple cannot represent the pattern even if you show it a million examples. Set the slider to zero bends and you get a flat line: the model has learned exactly one thing, the average score, and applies it to everybody.

That is underfitting. It is easy to spot, because the model is obviously bad at the data you trained it on — which means the training numbers tell you about it honestly.

Now the other end, which is where the trouble lives. Give the curve enough bends and it can wander through the training points almost exactly. Its error on them collapses toward zero.

But it did not learn the pattern. It learned the specific students, noise and all — every random high scorer and unlucky low one treated as a rule to be honoured. Show it someone new and it has nothing useful to say.

Drag from left to right and watch the two error lines separate. That separation is not a symptom of overfitting; it is overfitting. There is no other definition worth carrying around.

Notice the dashed line — the real pattern the model never gets to see. At three bends the fitted curve sits close to it. At twelve it is chasing individual dots and has wandered off the truth entirely, in places predicting scores that could not exist.

Here is the slider as a loop: fifteen students to fit, two hundred new ones to test on, and polynomial curves with more and more bends.

import numpy as npfrom numpy.polynomial import Polynomial rng = np.random.default_rng(5) def students(n):    hours = rng.uniform(0, 10, n)    score = 40 + 8 * hours - 0.4 * hours ** 2 + rng.normal(0, 5, n)    return hours, score train_x, train_y = students(15)    # what the model studiestest_x, test_y = students(200)     # people it has never seen def error(curve, x, y):    return np.mean((curve(x) - y) ** 2) for bends in [0, 1, 2, 5, 9, 12]:    curve = Polynomial.fit(train_x, train_y, deg=bends)    print(f"degree {bends:2}:  training error {error(curve, train_x, train_y):7.1f}"          f"   test error {error(curve, test_x, test_y):9.1f}")

Training error falls at every step, from 195 down to 3.7. Test error bottoms out at degree 2, which is the shape the data was generated from, and by degree 12 it has exploded past 22,000. Same students, same code; only the number of bends changed.

Here is the part that matters most, and it is worth stating carefully. As you add bends, training error drops from about 28 points to under 2 and essentially never stops improving. If you chose your model by that number alone, you would pick the most complicated one available, every single time.

There is no wobble in the training line that says “stop here.” The warning simply is not in that number, and no amount of staring at it will make it appear.