Class Imbalance

Four thousand card transactions. Seventy-five are fraudulent, which is one in fifty-three. Train a perfectly ordinary logistic regression on them, take the usual probability cut-off of 0.5, and it flags nothing at all. Not one transaction. It scores 98.1% accuracy.

The model is not broken and it has not failed to learn. It has learned correctly that under the loss it was handed, guessing “legitimate” every time is the best available strategy. Every fraud costs it a little; every false alarm would cost it a little; there are fifty-three times as many chances to be wrong on the second. The arithmetic is unarguable and the result is useless.

This is what imbalance does. The rare class is always the one you care about — the fraud, the tumour, the failing part — and it is always the one the default loss function is happiest to sacrifice.

The standard advice is to fix the data. There are three ways, and you should try all three in the panel above before reading on.

Undersampling keeps every fraud and throws away legitimate transactions until the classes are level. It works, and it means learning the legitimate half of the problem from a few dozen rows out of two and a half thousand. You paid for that data.

Oversampling keeps everything and copies each fraud until the counts match. Nothing is discarded, and nothing is added either — the model sees the same forty-eight frauds fifty times each and becomes confident about their particular quirks rather than about fraud.

Class weighting leaves the rows alone and charges a mistake on a fraud fifty-three times what it charges a mistake on a legitimate transaction. Nothing is duplicated or deleted, which is why it is usually the right one of the three to reach for.

Now the finding that should bother you. All three produce the same result: 59% recall, 4% precision, about 413 transactions flagged. Not similar — the same, to the row. Three different interventions, one outcome.

If all three are really moving the cut-off, then move the cut-off. Take the original model, trained on the untouched imbalanced data, and flag anything above a probability of 0.025 rather than 0.5.

That reproduces the rebalanced result almost exactly: 59% recall, 4% precision, 372 flagged. Same answer, no retraining, no discarded data, no duplicated rows.

And once the cut-off is a dial rather than an accident, you can put it where the problem actually wants it. At 0.076 the model catches 30% of frauds at 14% precision, flagging 56 transactions instead of 413. By F1 that is 0.19 against the 0.07 all three rebalancing strategies managed — nearly three times better, from changing one number after training.

Here is the same comparison on a fresh set of generated transactions, about one in fifty fraudulent.

import numpy as npfrom sklearn.linear_model import LogisticRegressionfrom sklearn.metrics import precision_score, recall_scorefrom sklearn.model_selection import train_test_split rng = np.random.default_rng(1)n = 6000fraud = rng.random(n) < 0.02                       # about 1 in 50amount = rng.normal(0, 1, n) + 1.2 * fraud         # fraud skews larger...night = rng.normal(0, 1, n) + 0.8 * fraud          # ...and later, with heavy overlapX = np.column_stack([amount, night])X_train, X_test, y_train, y_test = train_test_split(    X, fraud, test_size=0.5, stratify=fraud, random_state=0) print(f"flag nothing: accuracy {np.mean(~y_test):.1%}, recall 0%") def report(name, flagged):    print(f"{name:28} recall {recall_score(y_test, flagged):.0%}"          f"   precision {precision_score(y_test, flagged):.0%}   flagged {flagged.sum()}") plain = LogisticRegression().fit(X_train, y_train)report("plain model, cut-off 0.5", plain.predict(X_test)) weighted = LogisticRegression(class_weight="balanced").fit(X_train, y_train)report("class weights, cut-off 0.5", weighted.predict(X_test)) # No retraining: just lower the plain model's cut-off.for cutoff in [0.1, 0.04, 0.02]:   # try others    report(f"plain model, cut-off {cutoff}", plain.predict_proba(X_test)[:, 1] >= cutoff)

Flagging nothing scores 98.1% accuracy. The plain model at 0.5 catches 5% of frauds; class weighting catches 80% by flagging 707 transactions. Lowering the plain model’s cut-off to 0.02, with no retraining, lands next to it: 77% recall, 6% precision, 684 flagged.

Which threshold is right is not a question the data can answer. A review team that can process sixty cases a day sets it one way; an automatic block on a customer's card sets it far more cautiously, because the cost of a false positive is a stranded customer. Pick the operating point from the cost of each mistake, then report where you picked it and why.

Accuracy is the first thing to drop. At 98.1% for a model that flags nothing, it is not merely uninformative here — it actively rewards the failure. Any imbalanced problem where somebody quotes accuracy is a problem where nobody has looked closely.

Report precision and recall as a pair, at a stated threshold, and say how many cases that threshold sends to a human. F1 combines the two into one number, which is convenient for comparing models and hides the trade-off you actually need to argue about, so quote it alongside its parts rather than instead of them.

The uncomfortable summary: for a rare class, most of what you can do is choose which kind of mistake to make. Fifty-three to one is a hard problem, the model here tops out around 14% precision at 30% recall, and no amount of resampling changes the information in the data. Recognising that early is worth more than a better sampler.