From Prototype to Production

Every metric this track has built up — accuracy, the confusion matrix, mAP — describes performance on a fixed test set, measured once, under conditions that held still for the duration of the measurement. That number is real and it is useful for comparing models. It is not a description of what happens when the same model faces a continuous stream of camera frames arriving from the actual world, which does not hold still.

Production input is not a bigger version of the test set. It is a different kind of thing entirely: unbounded, arriving in real time, drawn from conditions the test set could only sample a slice of. A 94% test accuracy is a fact about the test set, not a promise about tomorrow’s camera feed.

A deployed camera drifts, in ways a training set fixed at one point in time cannot anticipate. A security camera’s lens picks up a scratch or a light film of dust over months of outdoor exposure. Its white balance shifts gradually with the seasons, as the colour temperature of daylight itself changes. A phone released two years after the training photos were captured carries a different camera sensor, a different default sharpening pipeline, sometimes a visibly different colour response out of the box.

None of this shows up as a dramatic failure. It shows up as a slow, unannounced decline — the model still runs, still returns confident-looking numbers, and is quietly working against slightly different input than the one it was tuned against.

A live video feed at 30 frames per second hands the model a new frame every 33 milliseconds, whether or not it has finished with the last one. Take a large, highly accurate model that needs 200 milliseconds per frame, running on a single worker. That worker cannot keep up with the frame rate, regardless of how the model scored offline: it has to skip five frames in every six, and whatever happens in those gaps goes unseen.

Two numbers are tangled together here. Throughput is how many frames per second the system can get through; latency is how long any one frame waits for its answer. Running six or seven copies of the model side by side, each taking every sixth or seventh frame, can bring throughput back up to 30 frames per second, at the cost of that much more hardware. It does nothing for latency. Every answer still arrives at least 200 milliseconds after its frame, which for a robot arm or a car can already be too late.

A three-second simulation of a 30 fps feed and a 200 ms model, with one worker and then seven.

frame_gap = 1000 / 30     # ms between frames at 30 fpsinference = 200           # ms per frame, on one workerframes = 90               # three seconds of video for workers in [1, 7]:    # try 3 or 5    free_at = [0.0] * workers          # when each worker can next start    processed = 0    for i in range(frames):        arrives = i * frame_gap        w = min(range(workers), key=lambda j: free_at[j])        if free_at[w] > arrives:       # every worker is busy, so this frame is skipped            continue        free_at[w] = arrives + inference        processed += 1    print(f"{workers} worker(s): {processed} of {frames} frames processed, "          f"each answer {inference} ms after its frame")

One worker processes 15 of the 90 frames. Seven process all 90. Either way, every answer arrives 200 ms after its frame.

This is a genuine trade-off against accuracy, not an implementation footnote to solve later. A smaller, faster, slightly less accurate model that fits inside 33 milliseconds per frame on the hardware you actually have is often the only real option; a bigger model that cannot keep up does not get partial credit for its offline score.

A benchmark had labels: every image had a known right answer, so accuracy could be computed directly. Production traffic has no such thing — nobody is sitting behind the live camera feed manually labelling every frame the moment it arrives, and a clean accuracy number requires exactly that.

Monitoring in practice means watching proxies instead of the real metric. A model’s confidence-score distribution drifting lower over weeks, even without any single dramatic failure, is a signal something in the input has shifted. A sudden spike in low-confidence predictions on a specific camera is a signal worth investigating before it becomes a user complaint. User-reported corrections, sparse as they are, are often the closest thing to ground truth a production system gets in real time.