40  Class Imbalance

In many classification problems one class is much less frequent than the other: deaths among admitted patients, fraudulent among all transactions, cases of a rare disease among those screened. Class imbalance is a property of the population, and in itself requires no correction. It does, however, affect how performance must be measured, how predictions are converted to decisions, and whether some algorithms learn anything about the minority class. This chapter sets out these effects, the common interventions, and their consequences for the predicted probabilities.

40.1 What imbalance changes

Let \(\pi\) be the prevalence of the positive class, the minority. Imbalance has four effects.

1. Metrics. Accuracy is dominated by the majority class, and a constant prediction of the majority class achieves accuracy \(1 - \pi\) (Section 8.2). Balanced accuracy, AUC and proper scoring rules compared with their baselines describe performance on both classes (Table 8.3).
2. The default threshold. A well-calibrated model predicts probabilities near \(\pi\) for most cases. With \(\pi\) = 0.12, few cases reach a predicted probability of 0.5, and a threshold of 0.5 classifies few cases as positive, even when the probabilities rank cases well.
3. Learning. What a model learns about the minority class comes from the minority cases, so the number of minority cases, not the ratio, limits what can be estimated (Section 6.7). Some fitting procedures respond to few minority cases by not separating them at all: a tree whose splits must reduce impurity by a minimum amount may not split, a penalized model whose penalty is chosen conservatively may shrink every coefficient to zero, and a nearest-neighbor model with small \(k\) finds few minority neighbors. Such a model predicts nearly the same probability, close to \(\pi\), for every case, and has an AUC close to 0.5.
4. Resampling. Folds of a cross-validation contain few minority cases, and their estimates vary widely; stratification keeps the number of minority cases equal across folds (Section 10.4).

40.2 The threshold is a decision

A probability is converted to a class by a threshold, and the threshold follows from the consequences of the two errors, not from the data. If a false positive costs \(c_{\mathrm{FP}}\) and a false negative \(c_{\mathrm{FN}}\), and the predicted probability \(\hat p\) is calibrated, the expected cost of predicting positive is \((1 - \hat p)\, c_{\mathrm{FP}}\) and of predicting negative \(\hat p \, c_{\mathrm{FN}}\). Predicting positive has the lower expected cost when

\[ \hat p > t^* = \frac{c_{\mathrm{FP}}}{c_{\mathrm{FP}} + c_{\mathrm{FN}}} \tag{40.1}\]

(Elkan 2001). A threshold of 0.5 corresponds to equal costs. When missing a death costs nine times as much as a false alarm, \(t^* = 0.1\). Changing the threshold changes sensitivity and specificity along the ROC curve (Section 8.3), and leaves the probabilities and the AUC unchanged. This use of the threshold requires calibrated probabilities.

40.3 Interventions on training

Several interventions change the training data or the loss so that the minority class weighs more:

• Case weights. Each case’s contribution to the loss is multiplied by a weight. Inverse frequency weighting gives each class a weight inversely proportional to its frequency, so that both classes contribute equally to the loss.
• Resampling the training data. Undersampling removes majority cases; oversampling duplicates minority cases; SMOTE creates synthetic minority cases by interpolating between neighboring minority cases (Chawla et al. 2002). All three are applied to the training data only, and within each training fold of a resampling.

All of these change the class balance that the model sees, from \(\pi\) to a value near 0.5. A model trained on a balanced sample estimates probabilities for a population with balanced classes, and therefore overestimates the probability of the minority class in the actual population. Its predictions are miscalibrated: among cases assigned a probability of 0.4, the proportion of positives is far below 0.4 (van den Goorbergh et al. 2022). The ranking of cases, and therefore the AUC, is typically little changed. The interventions reproduce, through the probabilities, the effect of lowering the threshold, at the cost of calibration.

For logistic regression the effect on the probabilities is known exactly. Weighting the positive class by \(w_1\) and the negative class by \(w_0\) adds \(\log(w_1/w_0)\) to the intercept in large samples, and the prior correction

\[ \operatorname{logit} \hat p_{\text{corrected}} = \operatorname{logit} \hat p_{\text{weighted}} - \log \frac{w_1}{w_0} \tag{40.2}\]

restores probabilities on the scale of the original prevalence (King and Zeng 2001). For other algorithms the distortion has no closed form, and a separate recalibration step, fitted on held-out cases, is needed (Chapter 42).

40.4 What to consider, in order

1. Measure performance with metrics that are not dominated by the majority class: AUC, balanced accuracy, and the Brier score compared with its baseline, \(\pi(1 - \pi)\).
2. Train without weights or resampling first. A model trained on the data as they are estimates probabilities on the scale of the population, which is the requirement for interpreting them as risks and for choosing a threshold by Equation 40.1.
3. Check whether the model learned the minority class. If its probabilities vary across cases and its AUC exceeds 0.5 clearly, it has. Low sensitivity at a threshold of 0.5 is a property of the threshold, and is addressed by choosing the threshold, not by retraining.
4. If the model failed to learn, predicting nearly the same probability for every case, use case weights, a different algorithm, or a less conservative regularization. Weights restore learning at the cost of calibration.
5. Recalibrate any model trained with weights or resampling before its probabilities are used, by prior correction for logistic regression or by a calibration model fitted on held-out cases (Chapter 42).

40.5 Summary

• Class imbalance affects metrics, the default threshold, the learning of some algorithms, and the variability of resampling estimates.
• The threshold is chosen from the costs of the two errors and requires calibrated probabilities.
• Case weights and resampling of the training data make some algorithms learn the minority class, and shift the predicted probabilities toward the minority class, which miscalibrates them.
• Models are first trained without correction; weights are used when the model otherwise fails to learn, followed by recalibration.

40.6 Further reading

• AMDS: Bayes’ theorem (prevalence and predictive values).
• van den Goorbergh et al. (2022) on the harm of imbalance corrections to calibration; King and Zeng (2001) on rare events and prior correction; Elkan (2001) on cost-sensitive thresholds; He and Garcia (2009) on learning from imbalanced data; Chawla et al. (2002) on SMOTE.
• Wikipedia: Oversampling and undersampling in data analysis, Cost-sensitive machine learning.

Chawla, N. V., K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. 2002. “SMOTE: Synthetic Minority over-Sampling Technique.” Journal of Artificial Intelligence Research 16: 321–57. https://doi.org/10.1613/jair.953.

Elkan, Charles. 2001. “The Foundations of Cost-Sensitive Learning.” Proceedings of the 17th International Joint Conference on Artificial Intelligence, 973–78.

He, Haibo, and E. A. Garcia. 2009. “Learning from Imbalanced Data.” IEEE Transactions on Knowledge and Data Engineering 21 (9): 1263–84. https://doi.org/10.1109/tkde.2008.239.

King, Gary, and Langche Zeng. 2001. “Logistic Regression in Rare Events Data.” Political Analysis 9 (2): 137–63. https://doi.org/10.1093/oxfordjournals.pan.a004868.

van den Goorbergh, Ruben, Maarten van Smeden, Dirk Timmerman, and Ben Van Calster. 2022. “The Harm of Class Imbalance Corrections for Risk Prediction Models: Illustration and Simulation Using Logistic Regression.” Journal of the American Medical Informatics Association 29 (9): 1525–34. https://doi.org/10.1093/jamia/ocac093.