Image Segmentation
A box around a dog still includes fifteen pixels of fence behind it. Segmentation labels every pixel individually, and the moment two dogs overlap, decides whether it still knows they are different dogs.
Worth reading first: Object Detection
A rectangle around a dog is never just the dog. It is the dog plus however much fence, grass, or empty air happened to be inside the smallest box that contained it.
A box still includes the background
Take a photo of a dog with one leg stretched forward and its tail out behind it. The smallest rectangle that contains the whole animal also contains the gap under its raised leg, the triangle of grass behind its tail, and a strip of fence on either side. None of that is dog.
Boxes are cheap to draw and cheap to grade with IoU, which is exactly why the last few lessons used them. But for some questions a box’s precision is the whole point, and a rectangle around an irregular shape structurally cannot deliver it.
A label for every pixel
Segmentation asks a different question entirely: not “where is a box that contains the object”, but “for every single pixel in the photo, what does it belong to”. The output is not four numbers — it is one label per pixel, for every pixel in the image.
For a 640×480 photo that is 307,200 individual decisions, not one. The result is a mask that follows the actual outline of the dog: the gap under its leg is correctly labelled background, the strip of fence is correctly labelled fence, and only the pixels that are actually dog are labelled dog.
Semantic versus instance, and why the difference matters
There are two real versions of this question, and they are not the same question wearing different names. Semantic segmentation labels every dog pixel in the photo “dog” — full stop. If there are two dogs standing side by side, every pixel belonging to either of them gets the identical label, and the output has no idea there are two animals rather than one oddly shaped one.
Instance segmentation keeps the pixel-level precision and adds the count back: each individual dog gets its own separate mask, so the two overlapping animals are known, correctly, to be two distinct objects rather than one. Which version you need depends entirely on the question: counting cars in a parking lot needs instance segmentation; measuring what fraction of a photo is sky needs only semantic.
What you pay for that precision
Producing a segmentation label needs a human to trace the actual outline of every object in every training photo, not draw a rectangle around it or type one word describing the whole image. Labelling a single photo this way routinely takes many times longer than boxing it, and boxing it already takes longer than giving the whole photo one label.
The models pay a cost too. Predicting one label per pixel is a larger, slower prediction than four box coordinates, so a segmentation model is typically slower to run than a comparable detector on the same photo. Precision bought at the pixel level is bought with real hours of labelling time and real milliseconds of runtime, not a free upgrade over drawing a box.
Key takeaways
- A rectangular box around an irregular object always includes some background — the gap under a raised leg, the fence behind a tail — that a mask would correctly exclude.
- Segmentation's actual output is a label for every pixel in the image, not four box coordinates, producing a mask that follows the object's real outline.
- Semantic segmentation labels every pixel of a class the same way and has no idea how many separate objects it is looking at.
- Instance segmentation keeps that precision and adds the count back, giving each individual object — including two overlapping ones — its own separate mask.
- That precision is paid for twice: pixel-level labels take far longer for a human to produce than a box or a single tag, and segmentation models are typically slower to run than detectors.
Quick check
Answer these to unlock the next chapter — 3 of 4 to pass. You can retake it anytime.
Answer every question to check.
Make a free account to read on
Every chapter is free — an account is how your progress, XP, and streak follow you from your laptop to your phone, and how you show up on the leaderboard. No payment, no trial.