44 Conformal Prediction
A point prediction carries no statement of its uncertainty. Conformal prediction converts the predictions of any model into intervals, for regression, or sets of classes, for classification, that contain the true outcome with a chosen probability. The guarantee holds in finite samples and for any model, including a misspecified one, under a single assumption about the data. This chapter presents split conformal prediction, the meaning and limits of its guarantee, and methods that adapt the interval width or use the data more efficiently.
44.1 Split conformal prediction
The available cases are divided into a proper training set, used to fit a model \(\hat f\), and a calibration set of \(n\) cases. For each calibration case, a nonconformity score measures how unusual its outcome is given the model’s prediction; for regression, the absolute residual
\[ s_i = \lvert y_i - \hat f(x_i) \rvert, \qquad i = 1, \dots, n . \tag{44.1}\]
Let \(\hat q\) be the \(\lceil (n + 1)(1 - \alpha) \rceil\)-th smallest of the \(n\) scores. For a new case \(x\), the prediction interval is
\[ C(x) = \left[ \hat f(x) - \hat q, \; \hat f(x) + \hat q \right] . \tag{44.2}\]
If the calibration cases and the new case are exchangeable, meaning that their joint distribution does not change when their order is permuted, as for independent draws from the same distribution, then
\[ P\left(Y \in C(X)\right) \ge 1 - \alpha \tag{44.3}\]
(Vovk et al. 2005; Lei et al. 2018). The argument is short: the score of the new case is equally likely to fall at any rank among the \(n + 1\) scores, so the probability that it exceeds the \(\lceil (n + 1)(1 - \alpha) \rceil\)-th smallest of the others is at most \(\alpha\) (AMDS: Permutation tests). Nothing about the model enters the argument: a poor model produces wide intervals, not invalid ones. The calibration set must be large enough for the rank to exist, \(n \ge 1/\alpha - 1\).
44.2 What the guarantee covers
The probability in Equation 44.3 is over the draw of the calibration set and of the new case. Three consequences follow.
- Coverage is marginal. The guarantee holds on average over new cases, not for each case or each subgroup. An interval of constant width covers more than \(1 - \alpha\) of the cases where the outcome varies little, and fewer where it varies much (Figure 44.1).
- Coverage varies between calibration sets. For a given calibration set, the coverage on future cases is a random quantity whose mean is at least \(1 - \alpha\); with a small calibration set it varies substantially.
- Exchangeability is required. Under a change of population, such as a different hospital or a later period, the guarantee does not hold.
Over 5000 new cases, the interval covers 91.7% of outcomes: 99.2% where \(x < 0.5\), and 84.1% where \(x \ge 0.5\).
44.3 Adaptive intervals
An interval of constant width reflects only the average size of the errors. Conformalized quantile regression (CQR) fits models of the lower and upper conditional quantiles of the outcome, \(\hat q_{\alpha/2}(x)\) and \(\hat q_{1-\alpha/2}(x)\), for example with a quantile regression forest (Section 30.3), and uses the score
\[ s_i = \max\left( \hat q_{\alpha/2}(x_i) - y_i, \; y_i - \hat q_{1-\alpha/2}(x_i) \right), \tag{44.4}\]
the distance by which \(y_i\) falls outside the estimated quantile interval (Romano et al. 2019). The conformal correction widens or narrows that interval by the same amount everywhere, and keeps the guarantee of Equation 44.3, while the width follows the estimated spread of the outcome at \(x\).
44.4 Using all the data
Split conformal prediction spends cases on calibration that could have been used for fitting. CV+ uses the models of a k-fold cross-validation: each case’s score is computed from the model that did not see it, and an interval for a new case is formed from the predictions of all fold models and all out-of-fold scores (Barber et al. 2021). Its worst-case guarantee is \(1 - 2\alpha\), and its coverage is usually close to \(1 - \alpha\). The jackknife+ is the case of leave-one-out cross-validation.
44.5 Prediction sets for classification
For a classification model with class probabilities \(\hat p_k(x)\), a conformal prediction set contains every class whose score does not exceed the calibrated threshold:
- the LAC score, \(1 - \hat p_y(x)\), gives the smallest sets with valid marginal coverage (Sadinle et al. 2019);
- the APS score accumulates the probabilities of the classes ranked above the true class, with a randomization, and gives larger sets with better coverage on uncertain cases (Romano et al. 2020).
A set with one class is a confident prediction; a set with several classes expresses ambiguity; an empty set, possible with LAC, indicates a case unlike the calibration cases. For a binary outcome, the possible sets are {No}, {Yes}, {No, Yes} and the empty set.
44.6 Summary
- Split conformal prediction calibrates the residuals of any model on held-out cases, and gives intervals or sets with coverage at least \(1 - \alpha\) when calibration and new cases are exchangeable.
- The guarantee is marginal, varies with the calibration set, and requires exchangeability.
- Interval width measures the usefulness of the model; coverage measures the validity of the procedure.
- CQR adapts the width to the case; CV+ uses all cases for both fitting and calibration.
- For classification, conformal prediction produces sets of classes.
44.7 Further reading
- AMDS: Permutation tests.
- Vovk et al. (2005) on conformal prediction; Lei et al. (2018) on distribution-free predictive inference; Romano et al. (2019) on CQR; Barber et al. (2021) on jackknife+ and CV+; Sadinle et al. (2019) and Romano et al. (2020) on prediction sets; Angelopoulos and Bates (2023) for an introduction.
- Wikipedia: Conformal prediction.