Train/Test Split

Imagine revising for an exam using a set of past questions, then being tested on those exact questions. You would score brilliantly, and the score would tell nobody anything about whether you understood the subject.

A model is worse than you at this, because it has a perfect memory. Give it enough capacity and it can store the answers outright — and then report a near-perfect score on them, honestly and uselessly.

The fix is simple enough to state in one sentence. Before training, put some of your data in a drawer. Train on the rest. Grade on the drawer.

The data you train on is the training set. The data in the drawer is the test set. The test set has to stay in the drawer — if you look at it, tune your model, and look again, you have started revising for those questions too, just more slowly.

Now the part nobody tells beginners. Which examples go in the drawer is decided at random — and that choice changes your score.

Below, the same sixty students are split many different ways at each setting. You only ever perform one split in real life, so you only ever see one of those dots. Drag the slider and watch how much they disagree.

At a 5% test set the reported score swings by nearly six points depending purely on which three students landed in the drawer. You would never learn that. You would run it once, write down whatever you got, and believe it.

Hold back more and the estimate steadies, because you are averaging over more people. By 70% the spread is about a fifth of what it was. That is the trade: a bigger test set buys you a more trustworthy number.

In practice people hold back somewhere between 20% and 30%, which is a compromise rather than a law. Now you know what is being traded on each side of it.

One more thing visible in the chart: the training error line sits below the test line at every single setting. That gap never closes, and it is the model flattering itself. Never quote a training score as though it were a measure of how good your model is.

Both effects show up on a real dataset that ships with scikit-learn: 569 breast tumours, each described by 30 measurements. An unrestricted decision tree is graded on what it studied, then on the drawer, and then the drawer is redrawn ten times.

from sklearn.datasets import load_breast_cancerfrom sklearn.model_selection import train_test_splitfrom sklearn.tree import DecisionTreeClassifier X, y = load_breast_cancer(return_X_y=True)   # 569 tumours, 30 measurements each # Put a quarter in the drawer, train on the rest.X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)model = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)print(f"graded on what it studied: {model.score(X_train, y_train):.1%}")print(f"graded on the drawer:      {model.score(X_test, y_test):.1%}") # Same model, same data, ten different drawers.test_size = 0.05   # try 0.3scores = []for seed in range(10):    X_train, X_test, y_train, y_test = train_test_split(        X, y, test_size=test_size, random_state=seed)    model = DecisionTreeClassifier(random_state=0).fit(X_train, y_train)    scores.append(model.score(X_test, y_test))print(f"test size {test_size}: scores from {min(scores):.0%} to {max(scores):.0%}")

The tree scores 100% on its own training rows and 88% on the drawer. With a 5% drawer the ten scores run from 79% to 100%; change test_size to 0.3 and they tighten to between 89% and 95%.