Dimensionality Reduction

Every extra column feels like a small win. One more measurement, one more angle on the problem, surely a fuller picture of each example. It is not, and the reason is not simply that some columns turn out to be useless. Even useful ones can hurt once there are enough of them.

Picture a scatter of points on a page. Some sit close together, some sit far apart, and “close” means something — the near ones really do resemble each other more than the far ones do. Keep adding measurements and something strange happens to that idea. By the time you are tracking forty things about each example, the nearest point and the farthest point end up almost the same distance away. Every direction looks equally empty. Anything that leans on distance — nearest neighbours, clustering, anything that measures how alike two examples are — starts guessing.

There is a second cost, and it compounds the first. A modest number of examples is enough to see a pattern laid out along one axis. Add axes and the space you would need to fill grows exponentially: cover a line with ten points and it looks dense; cover a hundred-dimensional cube with ten points along every side and you would need far more examples than exist for anything anyone measures. Practically, more columns without more rows means a thinner picture of the same-sized dataset, not a richer one.

The columns rarely earn their keep individually, either. Height in centimetres and height in inches are the same fact wearing two names — keeping both teaches a model nothing new, and it has spent one of its columns on a repeat. Two exam scores that move together are a softer version of the same problem: a student strong in algebra is usually strong in geometry too, so a good chunk of what one score tells you, the other one already told you.

Reduction sounds like deletion — pick a column, throw it away, hope you did not need it. That is one way to do it, and it is the crude one. The better way keeps every column’s information and asks the data to stand at a different angle instead.

Below are sixty-four students, placed by their algebra and geometry marks. The two scores are correlated, so the cloud is not round — it leans, spreading further along one diagonal than the other. A projection line is a direction you flatten every point onto: walk from each point straight across to the line and mark where it lands.

Drag the angle and watch two things move together: the drop each point makes onto the line, and the one-dimensional strip of shadows collecting underneath it. Somewhere in that sweep is the angle that keeps the most of the original spread. Hunt for it before you read the next section.

The angle you were hunting for has a name. It is the first principal component: the single direction along which the cloud spreads out the most. Here that is fifty-two degrees, and it keeps 86% of the total variance in the two scores. Turn the line ninety degrees the wrong way and it keeps 14%. Delete geometry outright and keep only the raw algebra axis and you retain 42%; delete algebra and keep geometry and it is 58%. Both of those single-column deletions lose more than the rotation you found by eye.

A second component is defined the same way, with one restriction: it must sit at a right angle to the first. In two dimensions that leaves no freedom at all — once the best line is fixed, the second one runs perpendicular to it and mops up whatever variance the first missed. With forty original columns you keep going: a third component at right angles to the first two, a fourth at right angles to all three, each one required to explain less than the one before it.

None of these directions is one of your original columns. The one you found is 0.62 × algebra + 0.79 × geometry — a blend, weighted slightly toward geometry because geometry happened to spread out a little further. That blending is exactly why components resist a plain-English name. “Overall academic strength” is a reasonable gloss for this one, but it is a gloss, not a fact the data handed you. There is no column in the original table called that — only a formula for rebuilding it from the two you gave it.

With two columns nobody bothers choosing how many components to keep — you use one or you use both. With forty, the choice is real, and it is normally made by looking at a single curve rather than inspecting each direction by hand.

1. 1
   Compute each component's share of the variance

   Divide every component's variance by the total, and order them largest first. The first component's share is, by construction, the biggest any single direction can claim.

2. 2
   Add the shares up as you go

   A running total: after one component, after two, after three. It climbs fast at the start and flattens as the later components mop up smaller and smaller scraps.

3. 3
   Stop where the curve bends

   People commonly stop at the elbow — where one more component buys noticeably less than the last one did — or at a round threshold like 90% or 95% retained, whichever comes first.

The blending that makes a component powerful is the same thing that makes it opaque. Tell someone a loan application was declined because of feature 3 — a mixture of income, existing debt, and two other columns weighted 0.4, 0.3, and 0.2 — and you have not really answered the question they asked. Before reduction, “this decision was driven by income” is a sentence you can say and defend. After it, the honest sentence is closer to “this decision was driven by a direction we cannot name,” which satisfies nobody in a room asking to be told why.

There is a sharper version of the same problem sitting in the sixty-four students above. Twenty-one of them were flagged for extra support, and that flag is barely visible along the best angle — the two groups’ averages sit just 0.2 points apart on the fifty-two-degree line. Rotate ninety degrees off it, to the direction the dial liked least, and the gap widens to 6.8 points, the clearest separation on the whole dial. PCA never looks at the flag. It only ever asks which direction the cloud spreads out along, and nothing forces that to be the direction that mattered for the thing you actually wanted to predict.

The fix for the other risk is procedural, and it is the same fix as scaling and every other transform learned from the data: fit the rotation on the training set only, then apply that exact rotation to the test set. Fit it on everything first and the test set has quietly leaked into the axes the model is trained on — the same rule the data leakage chapter spent an entire lesson on, wearing a new transform.