Image Classification

Classification’s contract is exactly this: one label, for the entire image, and nothing else. Not where the dog is. Not how many dogs. Not whether there is also a person in frame. Feed it a photo with three dogs and one cat and it still returns a single answer, and if that answer is “dog”, the model has done its job correctly by its own rules — the cat and the count were never part of the question.

That narrowness is the whole design, not a limitation someone forgot to fix. Detection and segmentation, later in this track, exist specifically to answer the questions classification refuses to.

Compare the two problems directly. Classification needs one number out of the model per photo — really, one probability per class it knows about, and it reports whichever is highest. Detection needs an unknown number of boxes, each with its own label and its own confidence, and the model does not even know in advance how many objects it is about to find. That single difference — a fixed-size answer versus a variable-size one — is most of why classification came first and detection took longer to get right.

One label per photo is also, not incidentally, much easier to grade. A classifier is either right or wrong about the whole image. A detector can be right about the label and wrong about the box, by a little or a lot, which is a harder thing to score — the subject of a later chapter on its own.

A well-known demonstration from research on explaining classifiers: a model was trained to tell huskies from wolves on photos where the wolves were nearly always pictured against snow. It then labelled a husky standing in snow as a wolf. When the researchers asked which parts of the image drove the decision, the answer was mostly the snow, not the animal. The model had learned to detect snow, not wolves, and on most of its test photos that shortcut happened to give the right answer — which is exactly why nobody would have noticed from the accuracy alone.

A confidence score describes how sure the model is, not how right it is. Those are the same number only when the training data taught the model the thing you actually wanted it to learn.

Two benchmark numbers get quoted for the same classifier, and they are not interchangeable. Top-1 accuracy asks whether the single best guess is correct. Top-5 accuracy asks whether the correct answer appears anywhere among the model’s five best guesses. A model can score 76% top-1 and 93% top-5 on the same test set, and both numbers are true at once — they are simply answering different questions.

A benchmark that reports only the flattering one is not lying, exactly, but it is choosing which promise to advertise. Ask which figure you are looking at before comparing two models, because the gap between them tends to be largest on exactly the fine-grained categories — breeds of dog, species of bird — where the correct answer and the model’s honest second guess are easy to confuse.

Four photos, six classes, and a model’s probabilities for each. Rank the guesses and score them both ways.

import numpy as np classes = ["husky", "wolf", "malamute", "fox", "coyote", "cat"]# The model's probabilities for four photos (one row each).probs = np.array([    [0.50, 0.30, 0.10, 0.05, 0.04, 0.01],    [0.35, 0.40, 0.15, 0.04, 0.05, 0.01],    [0.20, 0.10, 0.60, 0.04, 0.05, 0.01],    [0.05, 0.40, 0.03, 0.02, 0.30, 0.20],])truth = ["husky", "husky", "husky", "cat"] k = 3   # benchmarks usually report k=5; this model only knows 6 classesfor row, answer in zip(probs, truth):    ranked = [classes[i] for i in np.argsort(row)[::-1]]    print(f"truth {answer:6} top-1 {ranked[0]:9} top-{k} {ranked[:k]}") top1 = np.mean([classes[np.argmax(row)] == t for row, t in zip(probs, truth)])topk = np.mean([t in [classes[i] for i in np.argsort(row)[::-1][:k]]                for row, t in zip(probs, truth)])print(f"top-1 accuracy {top1:.0%}, top-{k} accuracy {topk:.0%}")

Only the first photo’s best guess is right, so top-1 accuracy is 25%. Every correct answer is somewhere in the top three, so top-3 accuracy is 100%. Same model, same probabilities.