Clustering
Every model up to here needed somebody to write the right answer down first. Step a k-means run one move at a time and watch the groups it settles on change with where you started it.
Worth reading first: Features and Labels
Every chapter before this one gave you something to check your answer against. A held-out label, a test score, a confusion matrix you could argue with. This one does not. You are about to hand an algorithm a pile of points with nothing written on them and ask it to find the groups, and it will find them — with total confidence, whether or not there was anything there to find.
When nobody labelled anything
Look back at Part 4. Train/test splitting, cross-validation, precision and recall — all of it assumed somebody had already written down the right answer for every example, so the model’s answer could be compared against it. That comparison is the entire honesty toolbox. Take away the right answers and every tool in it stops working, not because the problem got harder, but because the ingredient they all need is gone.
Nobody labelled which customers belong together, which genes switch on as a set, which documents are about the same thing. There is no drawer of correct groupings to hold back and grade against later, because no correct grouping was ever written down. This is unsupervised learning: structure found in the points themselves, with no answer key anywhere in the building.
The algorithm in this chapter, k-means, is the simplest and most common way to do it. You tell it how many groups to look for. It looks. It always reports back a grouping — never a shrug, never an “I don’t see anything here.” Whether that grouping means anything is a question the algorithm has no way to answer, and neither will you until you build the habit of asking it separately, every time.
Two moves, repeated until nothing changes
Start by dropping k points onto the plot — the centres. Everything else follows from two moves, repeated:
- 1
Assign
Every point joins whichever centre is currently closest to it, measured in plain straight-line distance.
- 2
Move
Each centre slides to the mean position of the points that just joined it — its own little average.
A centre nobody joined does not move. There was nothing to average.
Repeat both steps until an assign round changes nobody’s group. That has to happen eventually: each move can only shrink or hold the total squared distance from every point to its centre — never grow it — and there are only finitely many ways to split a fixed set of points into k groups. A quantity that keeps shrinking and can only take finitely many values has to stop shrinking. That is the whole termination argument, and it says nothing about where it stops.
That squared-distance total has a name: inertia. It is the only thing k-means is minimising. Not how sensible the groups look to you, not whether they match anything you care about — just that one number, and it will happily settle for a bad answer as long as moving further would not lower it.
Ninety-eight points, no colours, no labels. Pick how many groups to look for and where the centres begin, then press Step to watch assign and move alternate until nothing changes.
- group 1 — circle
- group 2 — square
- group 3 — triangle
The 3 centres are placed. Nothing has been assigned to them yet.
—
points assigned
—
points assigned
—
points assigned
Inertia
—The total squared distance from every point to the centre it belongs to. This is the only number k-means is trying to shrink — lower means tighter groups, and nothing else about the grouping is being judged.
Choosing k is your problem, not the algorithm's
Run k = 3 on the data above with a centre dropped near each group you can actually see, and it converges in two rounds to an inertia of about 88,639. Drop all three centres together in one corner instead, and it takes three rounds to land at roughly 255,560 — nearly three times worse — with one centre never claiming a single point. Both runs finish. Both print a number with the same confidence. Nothing about the output tells you which one to trust; you have to already know, or go find out.
There is a second assumption buried in the method itself, and it is easy to miss because nothing in the algorithm checks it. Distance to a single centre is only a sensible way to describe a group if the group is roughly round and roughly the same size as the others.
What k-means is built for
Round, roughly equal-sized blobs, spread out enough that the gaps between them are wider than the blobs themselves. Every group has one honest centre, and distance to it is a fair summary of belonging.
Where it quietly breaks
A crescent with a small round cluster tucked into its concave side. One huge diffuse group next to one small tight one. Two rings, one inside the other. k-means will draw straight boundaries through all of them and report an inertia number as if nothing were wrong, because nothing in the maths checks whether a straight boundary was the right kind of boundary to draw.
When the groups mean nothing
Here is the part that should worry you more than a bad elbow chart. Feed k-means pure noise — points scattered with no structure in them at all — and ask for three groups. It will not tell you there is nothing to find. It will draw three regions, hand you three centres, and report an inertia number exactly as confidently as it would on real structure. Clusters always come back. The algorithm has no mechanism for reporting “this is meaningless,” because meaning was never part of what it computes.
That confidence is manufactured, not earned, and it is the single most dangerous thing about this chapter’s algorithm. A classifier that is wrong will eventually meet a test set that says so. A clustering that is wrong meets nothing — there is no label to contradict it, so a meaningless grouping and a real one look identical in the output. The only thing that tells them apart is a check you choose to run.
Before you act on a clustering
None of these are computed by the algorithm. All of them are yours to do.
- Do the groups differ on something you did not feed the algorithm — spend, tenure, outcome — or only on the coordinates it was given?
- Does someone who actually knows this data recognise the groups, or do they look like an arbitrary cut?
- Do the same groups show up again with a different k, or a different starting position, or does the story change every time?
- Would you make a decision — a budget, a diagnosis, a policy — on this grouping, or only put it in a slide?
None of that turns a hypothesis into proof. It turns a number the algorithm was obligated to produce into a claim you are willing to stand behind, which is a different and much smaller thing.
Key takeaways
- Clustering has no answer key. Every honesty tool from Part 4 needed one, and none of them apply here.
- k-means repeats two moves — assign to the nearest centre, move the centre to the mean — until an assign round changes nobody. It has to stop; it does not have to stop somewhere good.
- It minimises inertia and nothing else. A grouping that looks wrong to you can still have the lowest inertia the run ever found.
- k is a choice you make, not a fact the data hands you. The elbow method is a heuristic for making that choice, not a proof you made it correctly.
- A cluster is a hypothesis, not a finding. The algorithm will hand you one from pure noise without blinking — checking whether it means anything is entirely on you.