16  Bias–Variance and Learning Curves

The test error of a model fitted to a finite sample has two sources besides the noise in the outcome: the model’s systematic deviation from the true function, and its sensitivity to the particular training sample. This chapter defines the two components, shows how model complexity trades one for the other, and introduces learning curves, which show how each component responds to more data.

16.1 The decomposition

Let the outcome be \(Y = f(X) + \varepsilon\), where \(f\) is the true regression function and \(\varepsilon\) is noise with mean zero and variance \(\sigma^2\). A learning procedure applied to a random training set \(D\) produces a fitted function \(\hat f_D\). At a fixed point \(x_0\), the expected squared error of a prediction, over training sets and over the noise of a new outcome, decomposes as

\[ \mathbb{E}\left[(Y - \hat f_D(x_0))^2\right] = \underbrace{\sigma^2}_{\text{irreducible error}} + \underbrace{\left(\mathbb{E}_D[\hat f_D(x_0)] - f(x_0)\right)^2}_{\text{squared bias}} + \underbrace{\mathbb{E}_D\left[\left(\hat f_D(x_0) - \mathbb{E}_D[\hat f_D(x_0)]\right)^2\right]}_{\text{variance}} \tag{16.1}\]

(Geman et al. 1992; Hastie et al. 2009). The bias is the difference between the average prediction, over all training sets the procedure could have received, and the true value. The variance is the spread of the predictions around that average. The same decomposition of the mean squared error of an estimator appears in AMDS (The bias-variance tradeoff).

Averaging Equation 16.1 over the distribution of \(x_0\) gives the decomposition of the expected test error. The irreducible error cannot be reduced by any model. Bias and variance depend on the learning procedure and on the size of the training set.

16.2 Complexity and the tradeoff

A procedure with a restricted model class, such as a linear model of a nonlinear relationship, has high bias: on average its fits miss the true function in the same way. Its variance is low, since there is little in the model for a particular sample to change. A procedure with a flexible model class can approximate the true function closely on average, so its bias is low, but it adapts to the noise of each training set, so its variance is high.

Figure 16.1 estimates both components for the polynomial regressions of Figure 4.1. For each degree, 500 training sets of 20 cases are simulated, a polynomial is fitted to each, and the squared bias and variance of the predictions are averaged over a grid of points.

Figure 16.1: Squared bias, variance, and expected test error (their sum plus the noise variance, 0.09) of polynomial regressions fitted to 20 cases, by degree. Each point averages 500 simulated training sets.

Squared bias falls sharply at degree 3, when the polynomial can represent the shape of the sine function, and changes little after. Variance rises with every added degree. Their sum, plus the noise variance, is smallest at degree 3. Every model-building procedure has some such point of balance, and much of this book concerns methods to find it, by penalizing complexity (Chapter 18) and choosing the penalty by resampling (Chapter 20).

16.3 Bias and variance in classification

The decomposition of Equation 16.1 holds for squared error. For a binary outcome it applies to the predicted probability: the squared error of the probability of the positive class, which is the Brier score, decomposes into the same three terms. Zero–one loss has no additive decomposition, and several incompatible definitions of bias and variance for it exist. The qualitative relationship holds in every case: restrictive procedures err consistently, flexible procedures err inconsistently.

16.4 Learning curves

A learning curve plots training error and test error against the size of the training set. As the training set grows, the variance of a procedure decreases, since each fit rests on more data; its bias does not decrease, since bias is a property of the model class. Training error typically rises with \(n\), since a fixed model class can fit a small sample more closely than a large one, and test error falls; the two converge to a level set by the bias and the irreducible error.

Figure 16.2 shows simulated learning curves for two procedures: a straight line, with high bias and low variance, and a polynomial of degree 9, with low bias and high variance.

Figure 16.2: Learning curves of a linear fit (degree 1) and a degree-9 polynomial, averaged over 100 simulated training sets per size. Dashed lines show training error and solid lines test error; the gray horizontal line marks the noise variance, 0.09.

The two patterns diagnose the two sources of error:

  • High bias. For the straight line, training and test error converge quickly, to a level well above the noise variance. More data does not help; a more flexible model class does.
  • High variance. For the degree-9 polynomial, test error is far above training error at small sizes, and the gap closes as data are added. More data helps, and so does restricting or penalizing the model.

16.5 Beyond the classical tradeoff

In some highly overparameterized models, including large neural networks and some ensembles, test error rises as the number of parameters approaches the number of training cases and then falls again as the number of parameters grows further, a pattern called double descent (Belkin et al. 2019). The classical tradeoff describes the regime in which the number of parameters is below the number of cases, which includes most models fitted to tabular data of the size in this book.

16.6 Summary

  • The expected squared error of a prediction is the sum of irreducible error, squared bias and variance.
  • Bias decreases and variance increases with the flexibility of the model class; their sum is smallest at an intermediate flexibility.
  • For classification, the decomposition applies to the squared error of predicted probabilities.
  • Learning curves distinguish high bias, where training and test error converge at a high level, from high variance, where a gap persists and more data helps.

16.7 Further reading