Convolutional Neural Networks

You already have the operation. Convolution: a small grid of numbers, slid across every position in an image, multiplying and summing as it goes. A convolutional neural network does not introduce anything new here. It takes that exact operation and repeats it, dozens of times, with a different set of kernels learned at each layer.

One layer might learn 32 kernels, each 3×3, each one a tiny pattern detector. Feed its output — 32 new grids, one per kernel — into a second layer of 64 more kernels, and each of those looks at all 32 inputs at once. Stack thirty layers like this and you are several million learned numbers deep before a single fully connected layer gets involved.

Nothing about a CNN is a new idea. It is the old idea, repeated, with the output of one repetition becoming the input to the next.

Open up the first layer of almost any trained image network and look at what its kernels actually detect. You will not find anything exotic. You will find edges — vertical ones, horizontal ones, diagonal ones at several angles — and a handful of kernels that respond to a colour contrast rather than a shape at all: orange next to blue, green next to red.

Nobody wrote “detect a 45-degree edge” into the training code. Every one of those kernels started as random noise and was pushed, gradient by gradient, toward numbers that reduced the network’s error on millions of labelled photos. Edges are simply what a network with a 3×3 window and one layer of depth is capable of noticing, and so edges are what it converges on.

A second layer does not see raw pixels. It sees the first layer’s edge map, and it can combine several edges at once. Two edges meeting at an angle is a corner. A corner repeated in a ring is close to a wheel’s rim. An edge that curves is close to the outline of an eye.

By the third or fourth layer, kernels stop responding to any single visual property and start responding to arrangements of parts — a headlight-shaped blob next to a wheel-shaped blob, roughly where a car would put them. Nothing forces this hierarchy to happen. It falls out of stacking simple operations and training on enough examples that “part in the right place” is the pattern that reduces error fastest.

Step through a small trained network below and see what a kernel at each depth actually fires on, and how much of the original photo each one is even looking at.

Depth buys you two things, and they are related. The first is composability: a later layer can only detect a wheel because an earlier layer already handed it edges and corners to combine. Skip the early layers and there is nothing left for a later one to combine.

The second is reach. A single 3×3 kernel in the first layer only ever looks at nine pixels. Stack five layers of 3×3 kernels and the fifth layer’s output at any position is influenced by roughly an 11×11 patch of the original image — its receptive field. Stack thirty and that patch can cover most of the photo. Depth is how a network built from small, local operations ends up seeing something global.

Depth is not free. More layers means more parameters to learn, more computation for every photo you show it, and a model that needs far more labelled examples to train from a random start than a shallow one does — training a serious network like this from nothing typically wants a dataset in the millions of images, not the thousands. The next lesson is about what you do instead of paying that price every time.