From Notebook to Production

A fitted model is a function. Put a system around it and the function is now the smallest part of the job. Something has to receive a request. Something has to find the features that function expects, in milliseconds, from wherever they actually live. Something has to answer inside a time budget, and something has to be certain which version of the model just answered. None of that is machine learning, and all of it is why the notebook score stops being the interesting number the day you deploy.

“Find its features” is the one that surprises people. In the notebook, every column sat in one tidy table, computed once, with time to check it. In production, the same feature — say, days since the last order — has to be recomputed live, from a database that may lag by minutes, by code that a different engineer wrote from a different specification on a different day. It is not the same computation with a different name. Very often it is quietly a different computation wearing the same name.

Nothing here is about whether the maths was right. The model can be exactly as good as the notebook said, and still fail, for reasons the notebook was never in a position to test.

A trained model is frozen the moment training finishes. Everything it will ever know about the world is whatever was true in that training window. The world keeps going. The gap between the two has a name — drift — and it comes in two flavours worth telling apart, because they break a model in different ways and get caught by different tools.

Input drift is the population changing while the underlying relationship stays put. A subscription model trained before a price rise has never seen the more price-sensitive customers who show up after it. Nothing about how price predicts churn has changed — the model just spends its days on people who look less and less like the ones it studied.

Concept drift is worse, because the relationship itself changes. A fraud model trained before a new payment method launched is now scoring transaction patterns that used to mean one thing and now mean another. The inputs can look identical to what the model saw in training. The honest answer behind them has moved.

A third failure hides between the two, and it is the quietest of the three. The transform in the notebook and the transform in the live service are supposed to be the same function, and they disagree by a hair — a missing value filled with zero here and left as null there, a date computed in one time zone in training and another at serving time. This is training/serving skew, and it is the leakage chapter's discipline running forward instead of back. There, the question was whether a column would exist at prediction time. Here it is whether the column at prediction time is computed the same way it was in training. Skip either question and the score you trusted was never describing the system you shipped.

Above, the world changes in month 9 and accuracy falls from 90.2% to 81.8% in a single month — over eight points, overnight. If you could watch accuracy directly, you would catch that immediately. Almost nobody can. Ground truth for most real systems arrives weeks after the prediction, if it arrives at all — a loan defaults or does not months later, a customer churns or does not next quarter, a fraud case gets confirmed by a human eventually or never gets reviewed. In the chart above, the labelling delay alone pushes discovery to month 13, and by then four months and 47,474 predictions have already gone out, quietly worse than the number on anyone's dashboard.

So you do not monitor accuracy. You monitor what you actually have the moment a prediction is made, none of which needs a label. The distribution of each input against the distribution it had in training, using a drift score like the one above — a population-stability-style number where roughly 0.2 is the usual line for “this is no longer a rounding error.” The distribution of the model's own predictions, since a sudden shift in how often it says yes is informative even blind. The flag rate, if there is a downstream action attached to a prediction. Latency and error rates, because a model that times out is a model that is wrong for every request it drops.

And you keep the baseline from chapter fourteen running, permanently, in parallel. It cost half an hour to build and nothing to keep running. When labels do eventually land, score it alongside the real model on the same window and watch the gap between the two rather than either number on its own. Both numbers drift with the population, so a falling accuracy alone is ambiguous — a harder population would do that too. A shrinking gap is not ambiguous. The baseline cannot overfit and cannot adapt; if it is closing in on the model that is supposed to be smarter than it, the model has lost ground the baseline never had to earn.

The moment drift shows up, the instinct is to retrain immediately and move on. Resist it. A retrain decided on a hunch, at the moment someone is anxious, is exactly how a model with a real, structural problem gets a fresh coat of paint instead of a fix. Retrain on a schedule — monthly, say, regardless of whether anything looks wrong — or on a trigger defined in advance, such as the input drift score crossing its alert line. Either is a decision made in advance, by a process, rather than a decision made in a panic by whoever is watching the dashboard that day.

A retrained model is a candidate, not a replacement. Evaluate it against the model currently live, and against the trivial baseline, on the same recent window — the same discipline as chapter fourteen, repeated every time, because a candidate that beats last quarter's incumbent by a point may just be fitting a population that shifted in its favour. If it cannot also clear the baseline by a comfortable margin, it has not earned the extra complexity, no matter how it compares to the model it is replacing.

1. 1

   Trigger the retrain on a schedule or a defined threshold

   Never on a feeling, and never mid-incident.

2. 2

   Score the candidate against the incumbent and the trivial baseline

   Same recent window, all three, so the comparison is fair.

3. 3

   Roll out behind a shadow or a canary

   A shadow scores live traffic without acting on it; a canary acts on a small slice. Neither lets the candidate decide anything on its own yet.

4. 4

   Watch it earn the rest of the traffic, and keep the rollback live

   The ability to go back to the incumbent in minutes is not optional.

None of this is exotic. It is the same honesty this whole track has been asking for — hold data back, compare against something stupid, never grade yourself on your own homework — applied to a model that keeps running after the notebook closes. A score tells you how the model did on one moment in time. Monitoring tells you whether the world it is still answering questions about resembles the one it was trained on.