Regularisation

Overfitting showed a model turning its coefficients up until it could chase every point in front of it, memorising the noise along with the pattern. This lesson is not another way to catch that happening — Cross-Validation already gave you the instrument for that. This is the fix.

Fitting a model, left alone, minimises one thing: error on the training data. Nothing in that minimisation cares how large a coefficient gets. If growing one by a hundred shaves even a sliver off the error on the noise sitting in front of it, an unpenalised fit takes that trade every single time, because nothing is charging it for the size of the coefficient itself.

Regularisation changes what gets minimised. Not error alone — error plus a penalty on the size of the coefficients.

minimise:  error(model)  +  λ · penalty(coefficients)

A coefficient now has to earn its keep. It is only allowed to grow if the error it saves is worth more than the rent it now owes on its own size. A coefficient chasing three noisy points rarely clears that bar. A coefficient carrying the real relationship almost always does.

“Penalty on the size of the coefficients” is doing a lot of work in that sentence, because there are two ordinary ways to measure size, and they punish a model in noticeably different ways.

L2 (ridge):  penalty = sum of coefficient²L1 (lasso):  penalty = sum of |coefficient|

Squaring, in L2, is severe on large coefficients and gentle on small ones — a coefficient of 10 costs a hundred times more than a coefficient of 1, so the penalty bites hardest exactly where it is biggest and eases off as it shrinks. It keeps easing off forever. A coefficient can get arbitrarily close to zero without the penalty ever fully letting go, so it shrinks towards zero and stops just short.

The absolute value, in L1, charges the same marginal rent no matter how small the coefficient already is. Shrinking a coefficient from 0.4 to 0.3 saves exactly as much penalty as shrinking it from 10.4 to 10.3. Once a coefficient is doing so little that the error it saves can no longer clear that flat, constant rent, the optimiser's best move is to stop paying rent on it at all — set it to exactly zero.

There is a compromise between the two — elastic net, which penalises with a blend of both sums at once — for when you want some sparsity without betting everything on it.

λ in that objective is not a fact about your data. It is a setting you choose, and dragging it changes the fit continuously between two named failure modes you have already met.

At λ near zero, above, nothing is charged for size at all and this is overfitting again, wearing a different name — validation error sits at $668 for ridge on this rental data, more than four times the $159 the model reports on the rows it trained on. Push λ all the way to 100 and every coefficient has been squeezed towards nothing; the model predicts close to the same rent for every listing, which is underfitting by another name. Its validation error, $592, lands almost exactly on the $593 you would get by ignoring the predictors entirely and guessing the average rent every time.

Between those two ends sits a dip. Ridge's validation error bottoms out at $382, a real improvement over doing nothing, at a modest λ of about 0.13. Lasso bottoms out at $371 at a much larger λ of roughly 42 — and at that setting, four of the nine predictors have been zeroed out entirely, leaving a model that openly admits it only trusts five of them. That U shape — high, dipping, climbing again — is the entire picture. Somewhere in the middle is better than either edge, and nothing about the training curve alone would have told you where.

Here is the part worth being careful about. You do not get to pick λ by trying a few values and reading off whichever gives the best score on your test set. The moment you choose a hyperparameter by looking at the test score, that score has stopped measuring how the model will do on data it has never seen — it is now measuring how well you searched. This is exactly the failure Data Leakage warned you about, arriving through a new door.

1. 1

   Hold out a validation set, separate from the test set.

   Or use the rotating folds from Cross-Validation instead of a single split.

2. 2

   Fit a model at each candidate λ, using only the training rows.

   Every value on the dial above is one full fit, not a tweak to one already made.

3. 3

   Score every candidate on the validation set, or the average across folds.

4. 4

   Pick the λ with the lowest validation error.

   That is the marked point on the error chart — a choice, not a discovery.

5. 5

   Touch the test set once, at the very end, with λ already fixed.

   Its score is now an honest estimate, because nothing about it influenced any choice you made.