Features and Labels

Nearly all of supervised machine learning is described by two words, and they are both simpler than they sound.

Learning means finding a pattern that gets from the features to the label reliably enough to be useful on examples you have not seen.

Features are not handed down. Somebody chooses them, and a different choice produces a different model.

Take bread. You have two measurements for every loaf: how many minutes it baked, and how hot the oven was. Neither one is the answer on its own — a loaf can be in a long time at a low temperature, or briefly at a high one, and come out the same.

So invent a third measurement out of the two you have. Slide between them and watch both the direction you are measuring in and the error at the best possible cut.

Time alone is not useless — it gets you most of the way, because longer usually does mean more cooked. Temperature alone is much worse. But a mix of the two roughly halves the error of the better one, and it is a number nobody handed you. You made it up, and it corresponds to something real: roughly thirty minutes at 200 °C.

That is feature engineering, and for most of the history of machine learning it was where the actual work happened. Modern systems can sometimes discover combinations like this themselves, which is a large part of why they are impressive — but somebody still chose what to measure in the first place.

The same experiment in code. Each column gets the single best cut a one-question decision tree can find, and then so does an invented column that mixes the two.

import numpy as npfrom sklearn.tree import DecisionTreeClassifier rng = np.random.default_rng(0)n = 400minutes = rng.uniform(15, 45, n)oven = rng.uniform(160, 240, n) # A loaf is cooked when time and heat together cross a line:# roughly thirty minutes at 200 degrees, with a little noise.heat_dose = minutes / 30 + oven / 200cooked = heat_dose + rng.normal(0, 0.05, n) > 2 def best_cut_accuracy(feature):    # A depth-1 tree finds the single best cut on one column.    stump = DecisionTreeClassifier(max_depth=1).fit(feature.reshape(-1, 1), cooked)    return stump.score(feature.reshape(-1, 1), cooked) print("minutes alone:   ", round(best_cut_accuracy(minutes), 2))print("oven alone:      ", round(best_cut_accuracy(oven), 2))print("invented feature:", round(best_cut_accuracy(heat_dose), 2))

Minutes alone gets about 91%, oven temperature alone about 59%, and the invented feature 97%. One honest caveat: this data was generated from that exact mix, so here the invented feature wins by construction. With real loaves you would not know the recipe, and finding a good mix is the work.