Image Segmentation

Take a photo of a dog with one leg stretched forward and its tail out behind it. The smallest rectangle that contains the whole animal also contains the gap under its raised leg, the triangle of grass behind its tail, and a strip of fence on either side. None of that is dog.

Boxes are cheap to draw and cheap to grade with IoU, which is exactly why the last few lessons used them. But for some questions a box’s precision is the whole point, and a rectangle around an irregular shape structurally cannot deliver it.

Segmentation asks a different question entirely: not “where is a box that contains the object”, but “for every single pixel in the photo, what does it belong to”. The output is not four numbers — it is one label per pixel, for every pixel in the image.

For a 640×480 photo that is 307,200 individual decisions, not one. The result is a mask that follows the actual outline of the dog: the gap under its leg is correctly labelled background, the strip of fence is correctly labelled fence, and only the pixels that are actually dog are labelled dog.

There are two real versions of this question, and they are not the same question wearing different names. Semantic segmentation labels every dog pixel in the photo “dog” — full stop. If there are two dogs standing side by side, every pixel belonging to either of them gets the identical label, and the output has no idea there are two animals rather than one oddly shaped one.

Instance segmentation keeps the pixel-level precision and adds the count back: each individual dog gets its own separate mask, so the two overlapping animals are known, correctly, to be two distinct objects rather than one. Which version you need depends entirely on the question: counting cars in a parking lot needs instance segmentation; measuring what fraction of a photo is sky needs only semantic.

A small semantic mask with two round dogs. First, how much of a box around one of them is not dog; then, counting the dogs by finding connected blobs.

import numpy as npfrom scipy import ndimage # A 20x20 semantic mask: 1 = dog, 0 = background. Two round dogs.rows, cols = np.mgrid[0:20, 0:20]dog_a = (rows - 6) ** 2 + (cols - 5) ** 2 <= 16dog_b = (rows - 12) ** 2 + (cols - 14) ** 2 <= 16mask = dog_a | dog_bfor row in mask:    print("".join("#" if dog else "." for dog in row)) # How much of dog A's bounding box is not dog?r, c = np.where(dog_a)box = mask[r.min():r.max() + 1, c.min():c.max() + 1]print(f"box around dog A: {box.size} pixels, {box.size - dog_a.sum()} of them background") # Semantic labels say "dog" but not how many. Count the separate blobs._, count = ndimage.label(mask)print("separate dogs found:", count)# Try centring dog_b at (8, 11) instead: the dogs touch and become one blob.

32 of the 81 pixels in the box are background. The blob count finds two dogs, but only while they do not touch. Move them together and the same mask reports one.

Producing a segmentation label needs a human to trace the actual outline of every object in every training photo, not draw a rectangle around it or type one word describing the whole image. Labelling a single photo this way routinely takes many times longer than boxing it, and boxing it already takes longer than giving the whole photo one label.

The models pay a cost too. Predicting one label per pixel is a larger, slower prediction than four box coordinates, so a segmentation model is typically slower to run than a comparable detector on the same photo. Precision bought at the pixel level is bought with real hours of labelling time and real milliseconds of runtime, not a free upgrade over drawing a box.