Confusion Matrix, One Class at a Time

Imagine a five-class animal classifier — cat, dog, fox, wolf, coyote — evaluated on 250 held-out photos, 50 per class. Report one number, and 78.4% is what comes back. That is a respectable score for a five-way problem with random chance at 20%.

It is also compatible with wildly different failure modes. A model that is a little wrong on every class could land at 78.4%. So could a model that is essentially flawless on cats, dogs and foxes and badly, specifically wrong about wolves and coyotes. Same headline number, two completely different engineering problems, and nothing about “78.4% accuracy” tells you which one you are looking at.

Every accuracy score is an average over a matrix, and the matrix is what actually happened. Rows are the truth — what the photo really shows. Columns are the prediction — what the model said. Cell (row, column) is a count: how many photos of the row’s class the model called the column’s class.

The diagonal — row equals column — is where the prediction matched reality. Everything off the diagonal is a mistake, and unlike the accuracy number, it is a nameable mistake: not just “wrong”, but wrong in one specific, countable direction. 19 wolves called coyotes is a different fact from 19 wolves called cats, even though both would show up identically in the overall score.

Click through the matrix above and one pair dominates. Cats, dogs and foxes each lose only three or four photos out of fifty, scattered one or two at a time across the other classes — noise, essentially. Wolves and coyotes are a different story: 19 wolves called coyotes, 21 coyotes called wolves, out of 50 of each. That is 40 of the 54 total errors in the entire matrix, from one pair of classes out of five.

That number changes what you would do next. Improving every class’s accuracy by a point each is five separate pieces of work for one point of overall gain. Fixing the wolf/coyote confusion alone — better ear and muzzle features, more training photos of exactly this pair side by side — recovers up to 40 of the 54 errors on its own, which would push accuracy from 78.4% to as high as 94.4%. The matrix tells you where to spend the next month, and the single number never could.

Try it on real images: a deliberately weak digit classifier, then the off-diagonal cells sorted by count.

import numpy as npfrom sklearn.datasets import load_digitsfrom sklearn.metrics import confusion_matrixfrom sklearn.model_selection import train_test_splitfrom sklearn.neighbors import KNeighborsClassifier X, y = load_digits(return_X_y=True)X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5, random_state=0)# A deliberately weak model: one neighbour, trained on only 60 images.model = KNeighborsClassifier(1).fit(X_train[:60], y_train[:60])predicted = model.predict(X_test) print(f"accuracy: {np.mean(predicted == y_test):.1%}")matrix = confusion_matrix(y_test, predicted)print(matrix)                       # rows: true digit, columns: predicted digit mistakes = matrix.copy()np.fill_diagonal(mistakes, 0)       # keep only the errorsfor _ in range(3):    true, said = np.unravel_index(mistakes.argmax(), mistakes.shape)    print(f"{mistakes[true, said]} photos of a {true} were called a {said}")    mistakes[true, said] = 0

Overall it scores 84.4%. The biggest single error is 21 eights called ones, then 16 threes called nines. Change the 60 training images to 300 and see which confusions survive.

Suppose the test set above had 50 cats, 50 dogs and 50 birds instead of wolves and coyotes — three classes nobody, model or human, seriously confuses. The diagonal would look close to perfect, accuracy would land somewhere near 98%, and it would prove almost nothing about whether the model has learned to see.

A clean diagonal on an easy test set is not evidence the model handles hard cases well. It is evidence the test set did not contain any hard cases. The wolf/coyote pair above is hard because the two animals look alike at the pixel level — similar coat colour, similar build, overlapping habitats in photos. A benchmark that never pairs visually similar classes together will never surface that failure, no matter how good the model’s overall score looks.

The honest question is not “what is the accuracy” but “which confusable pairs did this test set actually contain”. A model can be excellent at telling a cat from a bird and still fail constantly at the one distinction its deployment will actually need.