Convolution and Filters

A kernel is a small grid of numbers — 3-by-3 is the size you will see most — with one job: describe how to combine a pixel with the pixels immediately around it. That is the whole definition. It carries no code, no conditionals, no idea of what a face or an edge is. It is nine numbers.

What those nine numbers are decides everything. Nine equal small numbers average a neighbourhood together. A large number in the centre with negative numbers around it exaggerates the difference between a pixel and its neighbours. Same size grid, same operation applied to it, opposite result.

Convolution is the procedure that puts the kernel to work: stop at every position in the image, multiply each of the kernel’s nine numbers by the pixel currently underneath it, add the nine products together, and write that single sum into the output at that position. Then move one pixel over and do it again.

Try it below. An 8-by-8 input, a 3-by-3 kernel, and a button that moves the kernel one position at a time — nine multiplications, one sum, one output pixel, repeated 36 times until the entire 6-by-6 output is built. Nothing here is simulated or approximated; the arithmetic shown is exactly what produces the number in the highlighted output cell.

Switch the preset above between blur and sharpen and watch what actually changed: not the sliding, not the multiply-and-sum, not the number of stops. Only the nine numbers in the kernel changed.

Blur’s nine values are all 1/9 — every neighbour, including the centre, counted equally, which is just an average. Averaging a bright anomaly in with its darker neighbours pulls it toward them; that is what a blur is, arithmetically. Sharpen keeps a single large positive number in the centre and subtracts its four direct neighbours. Where the centre already matches its neighbours, the subtraction cancels out to roughly the same value. Where the centre is a genuine anomaly, the subtraction exaggerates it instead of averaging it away.

A hand-designed kernel only ever looks for the one pattern somebody built it to find. The sharpen kernel above always looks for “a pixel different from its four neighbours”, forever, on every image you ever give it. It has no way to notice that a particular photo would be better served by looking for something else — a diagonal line, a curve, a texture — because nobody wrote a kernel for that and handed it over.

It also cannot improve. Feed a hand-designed kernel a thousand more photos and it performs identically on the thousand-and-first, because there is nothing in it that changes in response to data. A trained convolutional network’s kernels are the same nine-number grids, run through the exact same sliding-and-summing procedure — the difference is that its numbers are not chosen by a person in advance. They are adjusted automatically until the network’s own predictions improve, which is exactly how it ends up with kernels for patterns nobody thought to hand-design. That is the subject of a later chapter; for now, the sliding-multiply-sum arithmetic you just ran by hand is the same arithmetic running, unchanged, inside it.