From Pixels to a Prediction

Take a small image — say 6 by 6, 36 pixels — and unroll it into a flat list of 36 numbers. The simplest possible classifier assigns one learned weight to each of those 36 positions, entirely independently. Pixel number 14 gets its own weight. Pixel number 15, its immediate neighbour, gets a completely unrelated one. Nothing in the model knows they are next to each other.

score = sum(pixel[i] * weight[i] for i in range(36))

Multiply every pixel value by its weight and add the 36 results together, and you get a single number: the score. Above some threshold, predict “yes” — this is the shape the model was trained to spot. Below it, predict “no”. That is the entire inference step, and it is trainable: adjust the 36 weights until the score comes out above threshold for positive examples and below it for negative ones.

The training is not the problem here. The problem is what this particular shape of model — one independent weight per pixel — can and cannot learn, no matter how well it is trained.

Below, four sliders control the weight for each quarter of a 6-by-6 grid, and a fixed bright 2-by-2 square sits in the top-left quarter — the “shape” this toy model is scoring. Set the weights, note the score, then press Shift pattern one cell right. The square moves one cell over. Nothing about its size or brightness changed. Watch the score anyway.

It moves, often by a lot, because two of the four bright cells now sit under a different region’s weight than they did before. The model has no concept of “a bright square,” only of “this exact pixel, at this exact position.” A shape it has correctly learned to recognise in one spot is, as far as its weights are concerned, an entirely different pattern one pixel over.

The same failure shows up on real handwriting. This model gives every pixel of an 8-by-8 digit its own weight, then is tested on digits nudged one or two pixels to the right.

import numpy as npfrom sklearn.datasets import load_digitsfrom sklearn.linear_model import LogisticRegressionfrom sklearn.model_selection import train_test_split digits = load_digits()X_train, X_test, y_train, y_test = train_test_split(    digits.images, digits.target, test_size=0.3, random_state=0) # One weight per pixel per digit: each 8x8 image becomes a flat list of 64 numbers.model = LogisticRegression(max_iter=2000)model.fit(X_train.reshape(len(X_train), -1), y_train) def accuracy(images):    return model.score(images.reshape(len(images), -1), y_test) print(f"test digits as drawn:     {accuracy(X_test):.0%}")for shift in [1, 2]:   # move every digit right; the empty column wraps round    moved = np.roll(X_test, shift, axis=2)    print(f"shifted {shift} pixel(s) right: {accuracy(moved):.0%}")

It reads 95% of the digits as drawn, 40% after a one-pixel shift, and 9% after two. Nothing about the digits changed except where they sit.

A convolution kernel, from the earlier chapter on filters, fixes exactly this failure. Instead of one independent weight per pixel position, a kernel is a small set of weights — say nine, for a 3-by-3 kernel — reused at every position in the image by sliding it across. The same nine numbers that detect a vertical edge in the top-left corner detect a vertical edge in the bottom-right corner too, because it is the same nine numbers doing the looking, wherever they happen to be looking.

That single change — share the weights across positions instead of giving every pixel its own — is the entire reason a convolutional network can recognise a shape wherever it appears in the frame, rather than only in the one spot it happened to appear during training. It is also, not coincidentally, the direct motivation for stacking that operation into the deep networks the next chapter builds.