Gradient Descent

Try every value of one number at a resolution of a hundred steps and that is a hundred tries — an afternoon, and more or less what chapter three’s bowl already was: a curve you could have plotted point by point and read the bottom off with your eyes.

Add a second number — a slope and a starting height, say — and every value of the first now needs testing against every value of the second. A hundred becomes ten thousand. A model with ten numbers at that same resolution needs a hundred raised to the tenth power tries. A small neural network has on the order of a million numbers to set, which puts the count at a hundred raised to the millionth power — a figure with no name worth learning, because the point is not the size of the number, it is that grid search does not merely slow down as parameters increase. It stops being an option at all.

The shape of the search has to change. Instead of trying values, you need a way to be told, from wherever you currently stand, which direction is better — without checking everywhere else first.

Stand at one point on the error curve and ask a single question: if this number went up a little, would the error go up or down? That question has an exact answer, and the answer is the derivative — the slope of the error exactly where you are standing, nothing more, and nothing about the rest of the curve required.

Chapter three’s widget already computed this every time you dragged its slider. Its “which way is downhill?” panel told you which way to nudge the number, and that panel was reading the sign of the derivative — positive or negative, downhill left or downhill right. Gradient descent is that panel with nobody left to read it: the sign alone chooses a direction, and once there is more than one number to move, the same idea applied to all of them at once gets a new name, the gradient.

Turn “which way” into “how far” and you have the entire algorithm. Multiply the gradient by a step size and subtract it from where you are:

new = old − rate × gradient

That line, run in a loop until it stops moving meaningfully, is the whole of how a model with one number and a model with a hundred billion numbers both learn. Nothing gets added on the way from one to the other. Only more numbers pass through the identical line, once each, every step.

Before you try the widget, run one step by hand.

Now set a rate below and press Step. Nobody is dragging anything this time — watch what the loop does on its own.

The rate is the only knob in the whole loop, and it does more work than its size suggests. At 0.001 the direction is right every single step and the pace is not: after fifty steps the parameter has crept from 0.5 to 4.54, a little over half the distance to the target of 7.8, with the error still sitting at 85.37 against a floor of 0.4. Each step closes the same small fraction of whatever gap remains, so fifty more steps would close only about the same share again — the gap shrinks, but never in a hurry.

At 0.03 the same loop reaches 7.761 — within the widget’s own tolerance of the target — by step eight. Nothing about the algorithm changed between these two runs. Only the size of the step did.

At 0.3 the step overshoots the bottom entirely: step one lands at 35.54, past the target on the far side, and step two overshoots the overshoot, landing at −97.61. The error that started at 426.72 is 88,893.92 by then and still climbing. At 1.02 one step is enough — the parameter jumps from 0.5 to 119.64 and the error goes from 426.72 to 100,058.73 before a second step is ever taken.

Here is the loop with two numbers instead of one: a slope and a starting height, fitted to fifty generated points whose true values are 3 and 1.

import numpy as np rng = np.random.default_rng(0)x = rng.uniform(-2, 2, 50)y = 3 * x + 1 + rng.normal(0, 0.5, 50)   # the true slope is 3, the true start is 1 slope, start = 0.0, 0.0rate = 0.1   # try 0.01, 0.5 and 0.8 for step in range(1, 41):    miss = (slope * x + start) - y    # The gradient of the mean squared error, one entry per number.    grad_slope = 2 * np.mean(miss * x)    grad_start = 2 * np.mean(miss)    slope -= rate * grad_slope              # new = old - rate x gradient    start -= rate * grad_start    if step in (1, 2, 5, 10, 20, 40):        print(f"step {step:2}: slope {slope:8.3f}  start {start:8.3f}"              f"  error {np.mean(miss ** 2):10.3f}")

At 0.1 it reaches a slope of 3.065 and a start of 1.010 within twenty steps. Try 0.01 and after forty steps it is still crawling. Try 0.8 and every step overshoots further than the last until the numbers run away.

In practice nobody knows the right rate in advance. It gets found by trying a handful of values on a small run and watching for exactly the three behaviours above — which is why you now know what each one looks like before it costs you a real training run to find out.

The bowl in this lesson has exactly one bottom, chosen deliberately so every run has a single honest answer to arrive at. Real error surfaces are rarely that considerate. A model with enough numbers folds its error into a landscape with several low points, and gradient descent has no way to tell, from where it is standing, whether the valley it just walked into is the deepest one on the map or simply the first one it fell into. It only ever knows the slope under its own feet.

There is a second requirement, easy to miss because the last lesson already met it: the error has to have a slope everywhere, which is why accuracy — the percentage of cases you got right — is never the number being minimised directly. Accuracy jumps in whole steps as one example flips from wrong to right; it has nothing to take a derivative of. Models are trained against a stand-in that does have a slope everywhere — squared error, or its cousins — and accuracy is only measured afterwards, as a report card, never handed the wheel.

You will meet five more words attached to this exact loop as soon as you read about training real models. Each one changes a single part of it and none of them replace the update rule above:

Every one of those is a variation on the one-line rule you already have. Two chapters from now, the neural networks lesson stacks that same loop underneath something with vastly more numbers to move, and the loop itself does not change. Only the size of the bowl does.