Baselines

Every number in this track has been a comparison in disguise. 82.5% for the forest meant something because a single tree managed 72.5%. The 98.1% in the last chapter meant nothing, and you could only tell because you knew that flagging nothing at all scored 98.1% too.

A baseline is the stupidest thing that could possibly work, built first and deliberately. Not a simpler model — a rule so trivial that beating it proves nothing, and failing to beat it proves something rather serious.

Building it first matters more than the number it produces. A baseline forces you to load the data, split it honestly, run a prediction and score it before you have written a model, which means the whole apparatus is working and debugged while there is still nothing to blame it on.

Say the commoner answer. Look at which outcome is more frequent and predict it every time, ignoring all input. For a classifier this is the floor.

Guess at the right rate. Predict at random in the proportions the training data showed. This is always worse than the majority rule, and running it once is the fastest way to understand why chance is not the bar to clear.

One rule on one column. Try every feature at every cut-off, keep the single best if-statement. This is a decision stump: a decision tree that was stopped after one question. It is the baseline that will embarrass you.

Predict the last value. For anything measured over time, predict that tomorrow equals today. It is devastating on forecasting problems and beats a surprising number of published models.

The regression equivalents are the same idea with the arithmetic changed: predict the mean of the training targets, or predict the previous value. If your model cannot beat the mean, it has learned nothing at all.

The dataset is deliberately ordinary. Six columns were recorded because recording them was easy — hours revised, hours slept, commute, siblings, height, shoe size — and exactly one of them has anything to do with passing.

The majority rule scores 56.0%. Random guessing at the right rate scores 47.3%, which is worse, as it always is. Then the one-rule baseline finds “revised at least 8.2 hours” and scores 87.3%.

Now the two real models. The five-neighbour vote gets 78.0%. The depth-4 decision tree gets 76.0%. Both lose to the if-statement, by nine and eleven points.

Neither is broken, and this is not a trick. The neighbour vote measures distance across all six scaled columns, so five-sixths of every distance it computes is noise — the exact failure from chapter eight, arriving through a different door. The tree does pick the right column first, then spends its remaining three levels splitting on shoe size and commute times, carving out little regions that fit the training set and mean nothing. It overfits into the noise the stump was too simple to reach.

The same race on fresh data: one column that matters, five that do not, and two of scikit-learn’s ready-made dummy models as the floor.

import numpy as npfrom sklearn.dummy import DummyClassifierfrom sklearn.model_selection import train_test_splitfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.pipeline import make_pipelinefrom sklearn.preprocessing import StandardScalerfrom sklearn.tree import DecisionTreeClassifier rng = np.random.default_rng(7)n = 500revised = rng.uniform(0, 16, n)passed = revised + rng.normal(0, 2, n) > 8noise = rng.normal(size=(n, 5))        # sleep, commute, siblings, height, shoe sizeX = np.column_stack([revised, noise])  # only column 0 mattersX_train, X_test, y_train, y_test = train_test_split(X, passed, test_size=0.3, random_state=0) models = {    "majority class": DummyClassifier(strategy="most_frequent"),    "random at the right rate": DummyClassifier(strategy="stratified", random_state=0),    "one rule (a stump)": DecisionTreeClassifier(max_depth=1),    "5 nearest neighbours": make_pipeline(StandardScaler(), KNeighborsClassifier(5)),    "depth-6 tree": DecisionTreeClassifier(max_depth=6, random_state=0),}for name, model in models.items():    model.fit(X_train, y_train)    print(f"{name:26} {model.score(X_test, y_test):.1%}")

The majority rule scores 54.7% and guessing at the right rate 48.7%. The stump gets 89.3%. The neighbour vote (88.0%) and the depth-6 tree (87.3%) come close but do not beat it.

It happens more than the literature suggests, and it is information rather than failure. When a trivial rule matches your model, one of four things is true.

The problem may be easy, and one column carries it — as here. Your features may not contain the answer, in which case no algorithm will find it and the work is to collect better data, not to try a bigger model. The model may be misconfigured, which is worth ruling out before the more interesting explanations. Or the extra capacity is finding noise, which is chapter six wearing a hat.

In every one of those cases the right move is the same: ship the baseline. It is faster, it is readable, it will not drift, and somebody can maintain it after you have left. Complexity has to be earned against a number, and here the number says it has not been.