What Is Machine Learning?

Ordinary programming is telling a computer exactly what to do. If you want to know whether someone can vote, you write the rule:

def can_vote(person):    return person.age >= 18 and person.is_registered

This is a good program. It is fast, it is exactly right, anyone can read it, and when the law changes you change one line. Do not use machine learning for this. If you can write the rule down, write the rule down.

Now try: is this message spam?

You start writing rules. Lots of links is suspicious — unless it is a newsletter. ALL CAPS is suspicious — unless it is a short excited message from a friend. The word “free” is suspicious — unless the email is about free shipping on something you ordered.

Every rule you write needs an exception, and the exceptions need exceptions. Try it below: each step adds the single best rule that can be added, chosen automatically, so you are seeing hand-written rules at their absolute best.

Two things matter here. The second matters more.

First, the returns collapse. The first rule buys you thirty percentage points. By rule six the gains are down to fractions of a point, and rules seven and eight buy literally nothing — there is no rectangle left to add. The effort per unit of improvement keeps climbing.

Second, look at the shape. Each rule is a rectangle, because that is what an if-statement about two numbers draws. But the real boundary between spam and not-spam runs diagonally — three links in a short message is suspicious, three links in a long newsletter is not. A staircase of rectangles can get closer and closer to a diagonal and never become one.

The dashed line is what a model learns from the same data. It is one line rather than eight rules, nobody wrote it, and it does better than the rules ever manage.

Here is the voting rule from the top of the page again, except this time nobody writes it. The model gets ten example people, their ages, and whether each one could vote, and has to find the rule itself.

from sklearn.tree import DecisionTreeClassifier, export_text # Examples instead of a rule: ages, and whether that person could vote.# (Everyone here is registered, so age is the only thing that matters.)ages = [[12], [15], [16], [17], [18], [19], [25], [40], [67], [80]]could_vote = [0, 0, 0, 0, 1, 1, 1, 1, 1, 1] model = DecisionTreeClassifier(max_depth=1).fit(ages, could_vote)print(export_text(model, feature_names=["age"])) # Nobody told it "18". It put the cut halfway between the# closest examples it saw: 17 (no) and 18 (yes).print(model.predict([[17], [18], [30]]))

It prints age <= 17.50: close to the real rule, and found only because the examples happened to include a 17-year-old and an 18-year-old. Delete the 17 from both lists and run it again, and the cut moves. That is the trade in miniature: the learned rule is only as good as the examples it saw.

This trade is not free. Know the price before you start.

You need examples, and lots of them. The rule for voting needed nobody’s data. A spam filter needs thousands of messages that somebody already labelled.

You lose the ability to read it. You can read age >= 18 and know exactly what it does. A learned model is a pile of numbers, and explaining any single decision is its own hard problem.

It is never exactly right. Notice that even the learned line does not get everything — some of these messages are ambiguous, and no method recovers those. Machine learning trades certainty for coverage.