Vision in Self-Driving Cars

A car moving at highway speed covers roughly 30 metres every second. Convert that to something more concrete: a vision pipeline that takes 200 milliseconds to decide “object ahead, brake” has already let the car travel about 6 metres before the decision even exists, before the brakes have engaged at all.

Every one of the metrics built up across this track — accuracy, mAP, IoU thresholds — was computed offline, with no clock running, no consequence to being slow. In a car, latency is not a performance detail to optimise later. It is subtracted directly from the stopping distance the vehicle has left, before a single millisecond of thinking about model accuracy even starts.

A camera is genuinely excellent at classifying and reading — it can tell a stop sign from a yield sign, read a speed limit, recognise a pedestrian’s posture. It is also exactly as blind as your own eyes in direct low sun, heavy rain, or full darkness, because it is measuring the same thing your eyes measure: visible light bouncing off surfaces.

That is why no production self-driving system trusts a camera by itself. Radar keeps working in rain, fog and darkness that would blind a camera, and measures distance and closing speed directly and precisely — but it is weak at telling you what kind of object it found, a plastic bag and a small dog can look similar to it. Lidar gives precise 3D distance to every surface it hits regardless of lighting, at real hardware cost, and can struggle with dark or reflective surfaces that absorb or scatter its laser pulses.

Combining sensor readings is called sensor fusion, and the hard part is not the combining — it is what to do the moment the camera says one thing and radar says another. A tempting answer is to average the two readings. It is also close to the worst option available: averaging a correct radar distance with an incorrect camera-based depth guess produces a number that is wrong in a new way, confidently.

Production systems instead generally weight each sensor by which one is more reliable for the specific kind of measurement and the specific condition. Radar’s distance reading is trusted over a camera’s depth estimate, because depth from a single camera is an inference, not a direct measurement. A camera’s read of an object’s class is trusted over radar’s, because radar was never built to answer that question. Fusion is a policy about which sensor gets believed for which question, not an average of everything the car currently hears.

None of this makes the problem solved, and it is worth being honest about where it still fails. An object with no real equivalent in the training data — an overturned trailer lying across a lane, a mattress that fell off a truck — can be missed entirely or misclassified, because the model has nothing in its experience to match it against.

Construction zones are a standing problem: lane markings that contradict the permanent ones underneath, temporary signage, cones arranged in a pattern no dataset fully anticipated. And adverse weather remains the case that degrades every sensor at once — heavy snow scatters lidar, fog attenuates radar less than camera but still measurably, and a camera in blowing snow can lose the road entirely. These are not edge cases in the abstract; each of them has caused real, documented incidents in deployed systems.