Feature Scaling

Nearest-neighbour models, and a good number of others, decide everything with Euclidean distance: take the difference between two points in each column, square it, add the squares up, and take the root. That formula treats every column identically — it adds whatever numbers it is handed, in whatever units they arrived in.

Below, a loan applicant is described by two columns: years of credit history, which runs 2 to 30, and annual income, which runs $18,000 to $240,000. Credit history spans 28. Income spans 222,000. Square those spans before you add them and income’s contribution outweighs credit history’s by a factor of roughly 62 million to one. A gap of a few hundred dollars in income now counts for more than a decade of credit history — not because income is more important, but because it is measured in smaller units and therefore produces bigger numbers.

Nobody decided that income should matter sixty-two million times more. The columns were just typed into a spreadsheet using whatever units people normally use for them, and the distance formula took that literally.

The fix is to put every column on a common scale before distance ever gets computed. There are two ways to do it, and it is worth writing both out plainly rather than waving at “normalise your data”.

Min-max scaling squashes a column onto [0, 1]: x′ = (x − min) / (max − min). The smallest value in the column becomes 0, the largest becomes 1, and everything else lands proportionally between them.

Standardisation centres a column on its own mean and divides by its own spread: x′ = (x − mean) / standard deviation. The result has mean 0 and standard deviation 1, and it is not bounded — a value three standard deviations out still reads as 3.

The choice between them turns on one thing: outliers. Min-max is at the mercy of a single extreme value, because that value defines one end of the scale. One applicant reporting a $4,000,000 income stretches the whole column, and every other applicant gets crushed into the bottom sliver near 0. Standardisation does not do this. An outlier pulls the mean a little and the standard deviation a little, but it does not redefine the axis for everybody else. When you cannot rule out an extreme value — which, with real data, is most of the time — standardisation is the safer default. Min-max earns its place when the bounds are genuine and fixed, such as a pixel value that is always 0 to 255.

You just watched the same applicant get two different verdicts from the same five neighbours. In raw units, income decides essentially 100% of the distance and the nearest applicants are simply whoever earns close to $45,000 — three of whom were denied. Standardise both columns and credit history claims most of the distance back; the nearest applicants become the other long-tenured ones, and every one of them was approved. Nothing about the applicant changed. Only the arithmetic did.

That failure is specific to a family of models, not universal to machine learning.

Here is the part that goes wrong even after everyone agrees scaling matters: where the mean and standard deviation — or the min and max — get computed.

The tempting order is to scale the whole dataset once, then split it into training and test folds. It looks harmless; scaling is “just preprocessing”, not modelling. But computing a mean over the whole dataset means the test fold’s incomes and credit histories helped decide what “average” is — and that average gets baked into every training row. The test set has quietly influenced the training data before the model has seen a single example. That is exactly the shape of leak the data leakage chapter warned about: information that should only exist after the split leaking into the numbers computed before it.

The correct order fits the scaler on the training fold only, and reuses those two saved numbers — nothing else — on the test fold:

scaler = fit_scaler(train)train_scaled = scaler.transform(train)test_scaled  = scaler.transform(test) # Wrong: fits mean and std on the whole dataset, so the test fold has# already shaped the numbers the training fold is scaled with.scaler = fit_scaler(features)features_scaled = scaler.transform(features)train, test = random_split(features_scaled, 0.33)

The test fold never contributes to the mean, the standard deviation, the min, or the max. It only ever gets transformed by numbers that were fixed before it was looked at — which is the same rule the model itself has to follow, applied one step earlier in the pipeline.