Image Classification

Classification’s contract is exactly this: one label, for the entire image, and nothing else. Not where the dog is. Not how many dogs. Not whether there is also a person in frame. Feed it a photo with three dogs and one cat and it still returns a single answer, and if that answer is “dog”, the model has done its job correctly by its own rules — the cat and the count were never part of the question.

That narrowness is the whole design, not a limitation someone forgot to fix. Detection and segmentation, two chapters ahead, exist specifically to answer the questions classification refuses to.

Compare the two problems directly. Classification needs one number out of the model per photo — really, one probability per class it knows about, and it reports whichever is highest. Detection needs an unknown number of boxes, each with its own label and its own confidence, and the model does not even know in advance how many objects it is about to find. That single difference — a fixed-size answer versus a variable-size one — is most of why classification came first and detection took longer to get right.

One label per photo is also, not incidentally, much easier to grade. A classifier is either right or wrong about the whole image. A detector can be right about the label and wrong about the box, by a little or a lot, which is a genuinely harder thing to score — the subject of a later chapter on its own.

A well-known example: a model trained to tell huskies from wolves reported 97% confidence on a photo of a wolf standing in a grassy field — correctly, this time, identifying it as a wolf, but for the wrong reason entirely. In earlier tests the same model had called wolves “husky” whenever the photo had snow in the background, because most of its husky training photos happened to have snow in them and most of its wolf photos did not. The model had learned to detect snow, not dogs, and reported startling confidence while doing it.

A confidence score describes how sure the model is, not how right it is. Those are the same number only when the training data taught the model the thing you actually wanted it to learn.

Two benchmark numbers get quoted for the same classifier, and they are not interchangeable. Top-1 accuracy asks whether the single best guess is correct. Top-5 accuracyasks whether the correct answer appears anywhere among the model’s five best guesses. A model can score 76% top-1 and 93% top-5 on the same test set, and both numbers are true at once — they are simply answering different questions.

A benchmark that reports only the flattering one is not lying, exactly, but it is choosing which promise to advertise. Ask which figure you are looking at before comparing two models, because the gap between them tends to be largest on exactly the fine-grained categories — breeds of dog, species of bird — where the correct answer and the model’s honest second guess are easy to confuse.