10  Resampling

The performance of a predictive model on new cases is estimated on cases not used to fit it (Section 2.4). With a single dataset, this requires dividing the cases into training and test sets. Resampling repeats the division, so that every case contributes to the estimate and the variability of the estimate is observed. This chapter describes the resampling designs in common use, what they estimate, and how the structure of the data constrains the choice among them.

10.1 What is estimated

Let \(\hat f_D\) denote the model obtained by applying a learning procedure to a dataset \(D\) of \(n\) cases. Two quantities can be of interest:

• the conditional error of the specific model fitted to the available data, \(\mathrm{Err}_D = \mathbb{E}\left[L(Y, \hat f_D(X))\right]\), where the expectation is over new cases \((X, Y)\) and \(D\) is fixed;
• the expected error of the procedure, \(\mathrm{Err} = \mathbb{E}_D\left[\mathrm{Err}_D\right]\), averaged also over training sets of size \(n\) drawn from the population.

The model that is eventually used is usually \(\hat f_D\), fitted to all available cases, so \(\mathrm{Err}_D\) is the quantity of practical interest. Resampling methods fit many models to subsets of \(D\), and their estimates are closer to the expected error \(\mathrm{Err}\) of the procedure, for training sets somewhat smaller than \(n\) (Hastie et al. 2009; Bates et al. 2024). A resampling estimate is therefore read as the performance to be expected from applying the same procedure, including any preprocessing and tuning, to data of this size from this population.

10.2 The holdout split

The simplest design divides the cases once, for example 75% for training and 25% for testing. Its estimate has two weaknesses. It is pessimistically biased, since the model is fitted to fewer cases than are available. It is variable, since it depends on which cases fall in the small test set: with 290 cases and 54 events, a test set of 73 patients contains about 14 deaths, and a different random division can give a substantially different estimate (Section 11.3). A single holdout split is appropriate when the dataset is large, or when the test set is defined by the study design, for example by time or by site.

10.3 k-fold cross-validation

k-fold cross-validation divides the cases into \(k\) folds of nearly equal size. Each fold serves once as the test set, with the remaining \(k - 1\) folds as the training set (Stone 1974). Every case is predicted exactly once, by a model that did not use it (Figure 10.1).

Figure 10.1: Five-fold cross-validation of 20 cases. Each column is one resample; light cells are training cases, dark cells test cases. Each case is a test case exactly once.

The cross-validation estimate is the average of the test-fold metrics, or a metric computed on the pooled out-of-fold predictions (Section 9.2). The choice of \(k\) balances two effects. Larger \(k\) gives training sets closer in size to \(n\), reducing pessimistic bias, and more model fits. Values of 5 and 10 are common and perform well in practice (Kohavi 1995; Hastie et al. 2009).

Leave-one-out cross-validation (LOOCV) is the case \(k = n\): each case is the test set once. Its training sets are nearly the full dataset, but the \(n\) fitted models are nearly identical, and a metric such as AUC cannot be computed on a single test case.

Repeated k-fold cross-validation repeats the procedure with different random fold assignments and averages the results. It reduces the variability due to the particular assignment, though not the variability due to the particular dataset.

10.4 Stratification

Random fold assignment can produce folds whose outcome distribution differs from that of the full data, especially for rare classes. Stratified resampling assigns cases to folds separately within each outcome class, so that every fold has close to the overall class proportions. For a numeric outcome, stratification is performed within quantile bins of the outcome. Stratification reduces the variability of the estimate at no cost, and is the default in rtemis for k-fold cross-validation, subsampling and the stratified bootstrap.

10.5 Repeated subsampling

Repeated random subsampling, also called Monte Carlo cross-validation, draws many independent training sets of a fixed proportion, for example 75%, and uses the remaining cases as the test set each time. Unlike k-fold cross-validation, the number of resamples and the training proportion are chosen independently, and a case can be a test case in several resamples or in none.

10.6 The bootstrap

A bootstrap resample draws \(n\) cases from the \(n\) available, with replacement (Efron 1979). A given case is left out of a bootstrap sample with probability \((1 - 1/n)^n \approx e^{-1} \approx 0.368\), so a bootstrap training set contains about 63.2% of the distinct cases, some repeated, and the remaining 36.8%, the out-of-bag cases, form the test set (Figure 10.2).

Figure 10.2: Five bootstrap resamples of 20 cases. Each cell shows how many times a case was drawn into the training set; cases drawn zero times are the out-of-bag test cases of that resample.

The out-of-bag estimate is pessimistically biased, since models are fitted to about 63% of the distinct cases; the .632 and .632+ estimators combine it with the training error to correct the bias (Efron and Tibshirani 1997). Because a bootstrap training set contains duplicated cases, it should not be used as the outer resampling of a procedure that itself divides its training data, as in hyperparameter tuning: copies of one case can fall on both sides of an inner split (Chapter 20).

10.7 Grouped resampling

The cases of a dataset are not always independent. Repeated measurements of the same patient, several patients from the same family, or images from the same scan share characteristics that do not carry over to a new patient, family or scan. If cases from one group fall in both the training and the test set, the model can be assessed on a group it has already seen, and the estimate reflects within-group rather than between-group generalization.

Grouped resampling assigns whole groups to folds, so that all cases of a group are on the same side of every split. The grouping variable is the unit of independence of the question (Section 6.1): to estimate performance on new patients, the patient is the group. In the Parkinson’s data, with 42 patients and over 100 recordings each, the difference between assigning recordings and assigning patients is large (Section 11.6).

10.8 Temporal resampling

When a model will be applied to cases that occur after the training data, for example to patients admitted next year, random assignment ignores changes over time in the population, in measurement practice or in treatment. A temporal split trains on earlier cases and tests on later ones, reproducing the conditions of use. With repeated measurements over time, it also prevents a model from using later measurements of a case to predict earlier ones.

10.9 The variability of resampling estimates

The metrics of the \(k\) test folds vary, and their standard deviation describes how much a single fold’s estimate can differ from another’s. It is not a standard error of the cross-validation estimate: the training sets of different folds overlap in \((k - 2)/(k - 1)\) of their cases, so the fold estimates are correlated, and no unbiased estimator of the variance of the k-fold cross-validation estimate exists (Bengio and Grandvalet 2004). Bates et al. (2024) propose a nested cross-validation procedure for confidence intervals. rtemis reports the standard deviation across resamples as a description of variability and does not compute confidence intervals from it.

10.10 Resampling the whole procedure

A resampling estimate applies to the complete procedure that is repeated within each training set. Any step that uses the outcome or the distribution of the data, such as feature selection, imputation, scaling or the choice of hyperparameters, must be performed within each training set and applied to the corresponding test set; otherwise information from the test cases enters the model and the estimate is optimistic (Chapter 12). When resampling is used both to choose hyperparameters and to assess performance, two levels of resampling are nested: an inner level for selection, within each training set of an outer level for assessment (Chapter 20).

After assessment, the final model is fitted to all available cases using the same procedure. The resampling estimate describes the expected performance of this model; the models fitted to the resamples are instruments of the estimate.

10.11 Summary

• Resampling estimates the expected performance of a modeling procedure on new cases from the same population.
• A single holdout split is simple but variable; k-fold cross-validation with \(k\) of 5 or 10, stratified on the outcome, uses every case once for testing.
• The bootstrap tests on out-of-bag cases and duplicates training cases.
• Grouped resampling keeps related cases together; temporal resampling tests on later cases.
• The standard deviation across folds describes variability; it is not a standard error.
• Every data-dependent step of the procedure is repeated within each training set.

10.12 Further reading

• AMDS: The bootstrap.
• Stone (1974) on cross-validation; Efron (1979) and Efron and Tibshirani (1997) on the bootstrap and its error estimators; Kohavi (1995) on the choice between them.
• Bengio and Grandvalet (2004) and Bates et al. (2024) on the variance of cross-validation estimates and what cross-validation estimates.
• Hastie et al. (2009), chapter 7.
• Wikipedia: Cross-validation (statistics), Bootstrapping (statistics), Resampling (statistics).

Bates, Stephen, Trevor Hastie, and Robert Tibshirani. 2024. “Cross-Validation: What Does It Estimate and How Well Does It Do It?” Journal of the American Statistical Association 119 (546): 1434–45. https://doi.org/10.1080/01621459.2023.2197686.

Bengio, Yoshua, and Yves Grandvalet. 2004. “No Unbiased Estimator of the Variance of K-Fold Cross-Validation.” Journal of Machine Learning Research 5: 1089–105.

Efron, B. 1979. “Bootstrap Methods: Another Look at the Jackknife.” The Annals of Statistics 7 (1). https://doi.org/10.1214/aos/1176344552.

Efron, Bradley, and Robert Tibshirani. 1997. “Improvements on Cross-Validation: The 632+ Bootstrap Method.” Journal of the American Statistical Association 92 (438): 548–60. https://doi.org/10.1080/01621459.1997.10474007.

Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning. In Springer Series in Statistics. Springer New York. https://doi.org/10.1007/978-0-387-84858-7.

Kohavi, Ron. 1995. “A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection.” Proceedings of the 14th International Joint Conference on Artificial Intelligence, 1137–45.

Stone, M. 1974. “Cross-Validatory Choice and Assessment of Statistical Predictions.” Journal of the Royal Statistical Society Series B: Statistical Methodology 36 (2): 111–33. https://doi.org/10.1111/j.2517-6161.1974.tb00994.x.