From Notebook to Production
A model that scored well on Tuesday's data is not a system, and the gap between the two is where most projects quietly die. Age a deployed model month by month and watch the score decay before anyone files a bug.
Read first: Baselines, Data Leakage
Somewhere there is a notebook cell that printed 0.94, and the team called that the model. It never was. It was one number, from one run, on data that was already ageing the moment anyone read it. Everything below is about the gap between that number and a system — and about how quietly that gap swallows projects that were, by every score that mattered on the day, working.
The score was never the product
A fitted model is a function. Put a system around it and the function is now the smallest part of the job. Something has to receive a request. Something has to find the features that function expects, in milliseconds, from wherever they actually live. Something has to answer inside a time budget, and something has to be certain which version of the model just answered. None of that is machine learning, and all of it is why the notebook score stops being the interesting number the day you deploy.
“Find its features” is the one that surprises people. In the notebook, every column sat in one tidy table, computed once, with time to check it. In production, the same feature — say, days since the last order — has to be recomputed live, from a database that may lag by minutes, by code that a different engineer wrote from a different specification on a different day. It is not the same computation with a different name. Very often it is quietly a different computation wearing the same name.
Nothing here is about whether the maths was right. The model can be exactly as good as the notebook said, and still fail, for reasons the notebook was never in a position to test.
The world moves, your model does not
A trained model is frozen the moment training finishes. Everything it will ever know about the world is whatever was true in that training window. The world keeps going. The gap between the two has a name — drift — and it comes in two flavours worth telling apart, because they break a model in different ways and get caught by different tools.
Input drift is the population changing while the underlying relationship stays put. A subscription model trained before a price rise has never seen the more price-sensitive customers who show up after it. Nothing about how price predicts churn has changed — the model just spends its days on people who look less and less like the ones it studied.
Concept drift is worse, because the relationship itself changes. A fraud model trained before a new payment method launched is now scoring transaction patterns that used to mean one thing and now mean another. The inputs can look identical to what the model saw in training. The honest answer behind them has moved.
A third failure hides between the two, and it is the quietest of the three. The transform in the notebook and the transform in the live service are supposed to be the same function, and they disagree by a hair — a missing value filled with zero here and left as null there, a date computed in one time zone in training and another at serving time. This is training/serving skew, and it is the leakage chapter's discipline running forward instead of back. There, the question was whether a column would exist at prediction time. Here it is whether the column at prediction time is computed the same way it was in training. Skip either question and the score you trusted was never describing the system you shipped.
A model deployed at month 1, aged one month at a time. Watch its true accuracy, how far its inputs have drifted from the training data, and how many predictions it served, all move together. Labels take 3 months to arrive, so accuracy is only ever known for months that far in the past.
- true accuracy
- input drift from training data
- predictions served
month 13 of 24
Accuracy now
78.5%Only ever knowable 3 months after the fact, once labels arrive.
Months since the change
4How long the model has been scoring a population it was never shown.
Served while degraded
47474Predictions made in the window nobody could yet see was bad.
By month 13, the labelling delay meant a human would only just be noticing what the model has known was wrong since month 9. 4 months and 47474 predictions had already gone out, quietly worse than the number on the dashboard.
Retraining resets the model to the world as it is now. It recovers most of the accuracy, not all of it — the underlying problem may just be harder than it was.
Monitoring when the labels arrive late
Above, the world changes in month 9 and accuracy falls from 90.2% to 81.8% in a single month — over eight points, overnight. If you could watch accuracy directly, you would catch that immediately. Almost nobody can. Ground truth for most real systems arrives weeks after the prediction, if it arrives at all — a loan defaults or does not months later, a customer churns or does not next quarter, a fraud case gets confirmed by a human eventually or never gets reviewed. In the chart above, the labelling delay alone pushes discovery to month 13, and by then four months and 47,474 predictions have already gone out, quietly worse than the number on anyone's dashboard.
So you do not monitor accuracy. You monitor what you actually have the moment a prediction is made, none of which needs a label. The distribution of each input against the distribution it had in training, using a drift score like the one above — a population-stability-style number where roughly 0.2 is the usual line for “this is no longer a rounding error.” The distribution of the model's own predictions, since a sudden shift in how often it says yes is informative even blind. The flag rate, if there is a downstream action attached to a prediction. Latency and error rates, because a model that times out is a model that is wrong for every request it drops.
And you keep the baseline from chapter fourteen running, permanently, in parallel. It cost half an hour to build and nothing to keep running. When labels do eventually land, score it alongside the real model on the same window and watch the gap between the two rather than either number on its own. Both numbers drift with the population, so a falling accuracy alone is ambiguous — a harder population would do that too. A shrinking gap is not ambiguous. The baseline cannot overfit and cannot adapt; if it is closing in on the model that is supposed to be smarter than it, the model has lost ground the baseline never had to earn.
Retraining without fooling yourself
The moment drift shows up, the instinct is to retrain immediately and move on. Resist it. A retrain decided on a hunch, at the moment someone is anxious, is exactly how a model with a real, structural problem gets a fresh coat of paint instead of a fix. Retrain on a schedule — monthly, say, regardless of whether anything looks wrong — or on a trigger defined in advance, such as the input drift score crossing its alert line. Either is a decision made in advance, by a process, rather than a decision made in a panic by whoever is watching the dashboard that day.
A retrained model is a candidate, not a replacement. Evaluate it against the model currently live, and against the trivial baseline, on the same recent window — the same discipline as chapter fourteen, repeated every time, because a candidate that beats last quarter's incumbent by a point may just be fitting a population that shifted in its favour. If it cannot also clear the baseline by a comfortable margin, it has not earned the extra complexity, no matter how it compares to the model it is replacing.
- 1
Trigger the retrain on a schedule or a defined threshold
Never on a feeling, and never mid-incident.
- 2
Score the candidate against the incumbent and the trivial baseline
Same recent window, all three, so the comparison is fair.
- 3
Roll out behind a shadow or a canary
A shadow scores live traffic without acting on it; a canary acts on a small slice. Neither lets the candidate decide anything on its own yet.
- 4
Watch it earn the rest of the traffic, and keep the rollback live
The ability to go back to the incumbent in minutes is not optional.
None of this is exotic. It is the same honesty this whole track has been asking for — hold data back, compare against something stupid, never grade yourself on your own homework — applied to a model that keeps running after the notebook closes. A score tells you how the model did on one moment in time. Monitoring tells you whether the world it is still answering questions about resembles the one it was trained on.
Before a model is allowed to make a real decision
Not a wish list. Every item here is something that has already gone quietly wrong on a real system.
- →The feature transform is one piece of code, called from both training and serving — not two implementations that are supposed to agree.
- →Every column has a documented answer to: would this value exist, computed this same way, at the moment the prediction is needed?
- →Input distributions, prediction distributions, flag rate, latency and error rate are all monitored, with alert lines set before launch rather than discovered after.
- →The labelling delay is known in months, written down, and every metric inside that window is treated as provisional.
- →The trivial baseline from chapter fourteen runs in parallel, permanently, and someone is watching the gap, not just the model's own score.
- →A retrain trigger is defined in advance — a schedule or a threshold, never a hunch.
- →A rollback to the previous model has actually been exercised, not just documented as possible.
You can now build a model, split it honestly, measure it without fooling yourself, baseline it against something stupid, and watch it for two years after the demo ends. That is the complete loop, and most introductions to this subject stop somewhere in the middle of it. What you do not yet have is the surrounding engineering — a feature store that agrees with itself, an alert that pages a real person, a retraining job that runs without someone copying files by hand. Those are real jobs, done by real teams, and no twelve minutes were ever going to teach them. What this track has taught you is what to ask for when you join one: where a score comes from, what it quietly stops meaning, and how to notice before your users do.
Key takeaways
- A notebook score is one number from one run on data that was already ageing. A system also has to receive a request, find its features live, answer in time, and know which version of itself replied.
- Input drift is the population changing while the relationship holds; concept drift is the relationship itself changing. They break a model differently and need different fixes.
- Training/serving skew is the leakage chapter's discipline running forward — the same transform, computed the same way, in both places, or the score you trusted was never describing the system you shipped.
- You almost never get accuracy live. Monitor the proxies you do have — input and prediction distributions, flag rate, latency, error rate — and know your labelling delay in months.
- Retrain on a schedule or a trigger, evaluate the candidate against the incumbent and the trivial baseline on the same window, ship behind a shadow or a canary, and always keep the rollback live.
Quick check
Answer these to unlock the next chapter — 3 of 4 to pass. You can retake it anytime.
Answer every question to check.
Make a free account to read on
Every chapter is free — an account is how your progress, XP, and streak follow you from your laptop to your phone, and how you show up on the leaderboard. No payment, no trial.