Neural Networks

Start with a single unit — one circle in any diagram you have ever seen of a network. Inside it: a weighted sum of its inputs, plus a bias — z = w1·x1 + w2·x2 + … + b. That is the exact shape of the model you built in How a Model Learns, just admitting the one term that example deliberately left out. Nothing about a single unit is neural in any sense a biologist would recognise. It is a straight-line fit, drawn as a circle instead of a graph.

A network is many of these, arranged in layers, each layer’s output feeding the next layer’s input. So try the obvious question: what happens if you stack two of these weighted sums, with nothing in between?

layer 1:  h = W1x + b1layer 2:  y = W2h + b2        = W2(W1x + b1) + b2        = (W2W1)x + (W2b1 + b2)

The last line is a weighted sum and a bias. Nothing else survives. Layer two just relabelled layer one’s numbers — W2W1 is one matrix, W2b1 + b2 is one bias — and ten stacked layers collapse the same way, in one more step each. A network built entirely from this circle, however deep, can compute nothing that a single weighted sum could not.

Which means the activation function — the one piece not yet mentioned — is not a detail bolted onto the design. It is the entire reason stacking layers does anything at all.

The activation most networks reach for by default is called ReLU, and the name is more intimidating than the definition. The whole of it: negatives become zero. Feed a unit’s weighted sum through ReLU and anything above zero passes through untouched; anything below zero is clamped to exactly zero.

One unit, run through that, gives you one fold in an otherwise straight line — flat on one side, sloped on the other. That alone breaks the collapse from the last section: a fold is not a weighted sum, so relabelling stops working, and stacking starts buying you something.

One fold is not a curve. But a handful of them, each oriented and positioned differently by training, cut the input into a patchwork of flat pieces that — looked at from far enough away — trace something that bends. A network never has a circle or a curve operation available to it. It only ever has straight pieces, glued together at a growing number of folds. Watch that happen below: an inner ring and an outer ring that no straight line can separate, and a decision boundary built entirely out of ReLU folds, one hidden unit at a time.

At zero hidden units the widget above is exactly the straight-line model from the first section, and it shows: 62.5% on the points it trained on, 61.7% on the ones it did not — barely past guessing, because half of each ring sits on either side of any line you could draw. One or two units bend that line slightly and buy almost nothing (69.2%, then 85.0% training accuracy, with validation trailing well behind at 63.3% and 68.3%). By three units the fold count catches up with the shape: 99.2% training accuracy, 93.3% held back. Five is where this particular ring stops needing help — 98.3% on data it never trained on, the best any width reaches here. Pushing on to eight buys nothing further: training accuracy tops out at 100%, and validation sits exactly where it sat at five. The extra capacity is not fitting the ring any better. It is chasing individual training points that happened to land where they did.

That is width — more units in the same layer, each one a fresh fold available at once. Depth is the other axis: more layers, stacked, so a fold in a later layer can bend a fold the earlier layer already made.

None of the above is an argument against networks. It is the case for reading this section before reaching for one, because everything that made the models earlier in this track pleasant to work with, a network mostly gives up.

The next chapter does not touch the model at all. It assumes you already have one that scores well and asks the harder question: what happens to that number on Tuesday’s data, six months from now, once nobody is watching it. The model was never the hard part. Keeping it working is.