What Is Computer Vision?

A digital photograph is a rectangle, and the rectangle is made of pixels, and every pixel is a small handful of integers. A modest phone photo might be 4032 pixels wide and 3024 tall — over twelve million pixels, each one carrying three numbers between 0 and 255 for red, green and blue. That is the entire file. There is no separate layer where “dog” or “grass” or “jumping” is recorded.

The computer receives a wall of numbers with no idea what a face is. It does not receive a slightly confused version of the photo you see. It receives a grid, and every ounce of meaning — where the edges are, which pixels belong to the same object, what that object is called — has to be built back up from those numbers by something else. That something else is the entire subject of this track.

You can look at that wall of numbers yourself. scikit-learn ships with a set of small handwritten digits; here is the first one.

import matplotlib.pyplot as pltfrom sklearn.datasets import load_digits digits = load_digits()image = digits.images[0]     # one handwritten digit, 8 pixels by 8 print(type(image).__name__, image.shape)print(image.astype(int))     # this grid is everything the computer receives plt.figure(figsize=(2.5, 2.5))plt.imshow(image, cmap="gray_r")plt.axis("off")plt.show()                   # the same 64 numbers, drawn as squares

The array is the image. The picture underneath is drawn from those same 64 numbers, and nothing in them says “zero”.

“Computer vision” sounds like one skill, the way “seeing” feels like one thing to you. It is not one job. Ask a system to look at a photo and you could be asking it to answer three different questions, and a solution to one does not hand you the other two for free.

People say “computer vision can recognise faces now” as though that settles it, and it settles almost nothing — recognising that a face is present, finding where it is in a crowd, and outlining exactly which pixels are eyebrow are three different engineering problems with three different failure modes.

The obvious first move is to write the rule down yourself, the way you would for ordinary software. A face has two dark ovals above a horizontal line — eyes above a mouth. Write that as code, point it at a folder of photos, and it falls apart almost immediately.

Take one photograph: someone photographed from a three-quarter angle, backlit by a window behind them, wearing sunglasses. The eyes are not dark ovals — they are hidden. The mouth is not a clean horizontal line — it is foreshortened by the angle and half in shadow. Nothing about the rule was wrong on the photo it was written for. The photo changed underneath it.

That is the actual shape of the problem, not a one-off edge case: lighting changes which pixels are bright without changing what is in the scene. Angle changes an oval into a sliver. Occlusion — a hand, a scarf, another person walking past — deletes part of the thing you are looking for entirely. Scale means the same face is forty pixels wide in one photo and four hundred in another, and a rule tuned to one size misses the other completely.

If rules do not survive contact with real photos, the rest of this track is about what does. The shape of the argument is always the same: start from the grid of numbers, build something that survives the four problems above, and check honestly whether it actually did.

1. 1

   Turn a photo into numbers you can actually compute on

   Pixels, channels, resolution — what is really inside the file.

2. 2

   Find the structure hiding in the noise

   A small grid of numbers, slid across the image, is enough to find edges and corners without a single hand-written rule about faces.

3. 3

   Teach a model to name what it sees

   Not by writing the rule, but by showing it enough labelled examples that it works the boundary out for itself.

4. 4

   Teach it to point, and to count

   Where an object is, how many there are, and which pixels belong to it.

5. 5

   Make the score honest

   A number on a test set can flatter a model in several specific, checkable ways. Learn to catch them.

6. 6

   Survive contact with the world

   A phone's camera, a car's latency budget, and a stranger's face are not a notebook.