Random Forests

A deep tree is not wrong on average. It is wrong erratically. Move three flats in the training set and whole branches rearrange themselves, because one different split near the root changes everything beneath it. That instability is a nuisance if you want one reliable tree. It is the raw material if you want a crowd.

Here is the bet. If you can build many trees whose mistakes are unrelated to each other, then averaging them cancels the mistakes and keeps the signal. Each tree is worse than the careful single tree you would have built. The average of them is better than any of them. That only holds if the errors really are unrelated — which is why the next section is about forcing the trees apart rather than about trees at all.

The data has changed slightly for this chapter, and deliberately. The previous chapter's flats were arranged in rectangles, which is the shape a tree draws naturally. These 250 flats follow the honest rule: a flat lets quickly when the rent plus the cost of travelling to campus is low. That boundary is a diagonal, and a diagonal is precisely what an axis-aligned tree cannot express.

Given identical data and the same greedy procedure, every tree you grow is the same tree. Two hundred copies of one opinion is still one opinion. So a random forest handicaps each tree in two independent ways.

Bootstrapping. Each tree gets its own training set, drawn from the original with replacement and the same size. Some flats appear three times, some not at all — roughly a third are missing from any given tree. Every tree therefore sees a slightly different world.

Feature subsetting. At every split, a tree may only consider a random subset of the features rather than all of them. With just two features here, that means each split is decided by a coin flip and the best cut on whichever feature it landed on. This is the more important of the two, and the less obvious: without it, one dominant feature would be chosen at the root of nearly every tree no matter how the rows were resampled, and the trees would stay correlated.

At one tree the shading is blocky and absolute: solid or nothing, with a staircase edge where the tree tried to trace the diagonal. It scores 72.5% on the held-back flats, the same as the single unrestricted tree it is a copy of in spirit.

Add trees and a soft band appears along the true boundary. Those are the cells where the trees disagree, and their position is not arbitrary — the disagreement collects exactly where the answer is actually uncertain. By 120 trees the staircase has become a diagonal, drawn by an ensemble of models none of which can draw a diagonal.

The score climbs to 82.5%. Ten points above the unrestricted tree, and six and a half above the best pruned single tree, which was found by cheating slightly — trying all fourteen depths and keeping whichever scored best on the very data it was being measured on.

Notice where the gain stops. Between 32 and 120 trees the score only wobbles within a point: 82% at 32, 81% at 50, back to 82.5% at 120. Adding trees does not make a forest systematically worse. The vote just gets less noisy, and on a test set this small a single flat or two changing sides moves the score up or down a little. But the returns flatten early. Practitioners pick a few hundred and stop thinking about it, because the parameter that matters is how the trees are made, not how many.

Try it on a different dataset. One unrestricted tree first, then forests of increasing size.

from sklearn.datasets import make_classificationfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.model_selection import train_test_splitfrom sklearn.tree import DecisionTreeClassifier X, y = make_classification(n_samples=600, n_features=8, n_informative=4,                           flip_y=0.1, random_state=2)X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0) tree = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)print(f"one unrestricted tree: {tree.score(X_test, y_test):.1%}") for n_trees in [1, 5, 25, 100, 200]:    forest = RandomForestClassifier(n_estimators=n_trees, random_state=0)    forest.fit(X_train, y_train)    print(f"forest of {n_trees:3} trees:  {forest.score(X_test, y_test):.1%}")

The single tree scores 72.8%. A forest of one tree does worse, 60.6%, because that lone tree was trained on a bootstrap sample with a limited choice of features at each split. By 25 trees the forest reaches 80.6%, and after that the score only wobbles: 80.0% at 100 and 79.4% at 200.

The previous chapter sold you the tree on the strength of being able to read it. A forest throws that away completely. There is no flowchart to print. The answer is a tally across 120 disagreeing flowcharts, and no sentence summarises it.

What you get back is a partial substitute: the forest can tell you which features mattered, by measuring how much impurity each one removed across every split in every tree. That is useful and it is not the same thing. “Rent mattered most” does not tell a landlord why their flat was predicted to sit empty.

It also costs more of everything. Training is a hundred-odd times the work of one tree, prediction asks every tree before it can answer, and the model is a hundred times the size on disk. All three are usually acceptable, and all three are the reason a single tree still ships in places where a forest would not fit.

The honest summary is that random forests are the strongest thing in this track that you can understand completely, and they are a reasonable first attempt on almost any table of rows and columns. They will not read a sentence or recognise a photograph. Nothing here will. What they do is take a model that was too unstable to trust and make its instability the point.