36  Stacking

The previous chapters fitted many algorithms to the same data, and their cross-validated performance differed by less than its variation across folds. Instead of choosing one, stacking combines their predictions, with weights learned from cross-validated predictions so that the combination generalizes. This chapter presents stacked generalization and the Super Learner, and two variants: a combination whose weights depend on the case, and a combination of models fitted to different groups of features.

36.1 Stacked generalization

Let \(\hat f_1, \dots, \hat f_K\) be \(K\) learning procedures, the library or base learners. Stacking fits a second-level model, the meta learner, that predicts the outcome from the base learners’ predictions (Wolpert 1992). The meta learner cannot be fitted on the base learners’ predictions for their own training cases, which are optimistic (Section 2.4) and would give most weight to the most flexible learner. Instead:

  1. The training data are divided into \(V\) folds. Each base learner is fitted on \(V - 1\) folds and predicts the held-out fold, which gives a cross-validated prediction \(\tilde f_k(x_i)\) for every case \(i\) and learner \(k\).
  2. The meta learner is fitted to the outcome using the \(K\) cross-validated predictions as features.
  3. Each base learner is refitted on all the training data. A prediction for a new case is the meta learner applied to the refitted base learners’ predictions.

Breiman (1996) showed that, for regression, a linear meta learner with non-negative coefficients performs well, and that the constraint is necessary: the cross-validated predictions of different learners are highly correlated, and an unconstrained regression on them is unstable (Section 14.5).

36.2 The Super Learner

The Super Learner formalizes stacking with a cross-validated loss (van der Laan et al. 2007). With a convex combination as the meta learner, the ensemble prediction is

\[ \hat f_{\mathrm{SL}}(x) = \sum_{k=1}^{K} \alpha_k \hat f_k(x), \qquad \alpha_k \ge 0, \quad \sum_k \alpha_k = 1, \tag{36.1}\]

with weights \(\alpha\) that minimize the cross-validated loss of the combination, computed by non-negative least squares (Section 14.6) and normalized. The oracle inequality of the Super Learner states that, asymptotically and for a bounded loss, its risk exceeds that of the best combination in the class, the oracle, by at most a term of order \(\log(K)/n\) for \(K\) candidates and \(n\) cases (van der Laan et al. 2007). The discrete Super Learner selects the single learner with the lowest cross-validated risk, which is model selection by cross-validation.

Because the weights are estimated by cross-validation within the training data, the performance of the ensemble is assessed, like that of any tuned procedure, by an outer resampling (Section 20.4). A library can include the same algorithm with different hyperparameters, each a separate entry, so that the weights perform the selection that tuning would.

36.3 Conditional combinations

A convex combination applies the same weights to every case. The best learner can differ between regions of the feature space: a linear model where the relationship is linear, a tree ensemble where it is not. The Conditional Super Learner fits an oracle, a classifier that assigns each case to one of the base learners, the experts, and predicts each case with its assigned expert (Valdes et al. 2022). Oracle and experts are fitted by alternation: with the experts fixed, the oracle is fitted to assign each case to the expert with the lowest cross-validated loss on it; with the oracle fixed, each expert is refitted on the cases assigned to it. The fitted oracle describes which features determine which model applies.

36.4 Stacking feature groups

When the features form groups measured by different instruments, such as imaging, genomic and clinical variables, each group can be modeled separately, with an algorithm suited to it, and the group-specific predictions combined by a meta learner. Each base learner sees only its own group, so the meta learner weighs the predictive information of each modality. A convex combination averages the modalities’ predictions; a non-negative combination without the sum-to-one constraint lets complementary modalities add their contributions.

36.5 Properties

  • A stacked ensemble is at least as accurate as its best member, asymptotically, and more accurate when members err differently.
  • The members’ errors must differ for the combination to help; a library of similar algorithms produces correlated predictions and a combination close to its best member.
  • The cost is the sum of the costs of all base learners, multiplied by the number of folds.
  • The weights describe the contribution of each learner’s predictions given the others, not the quality of each learner alone; a strong learner correlated with a stronger one can receive zero weight.

36.6 Summary

  • Stacking fits a meta learner on the cross-validated predictions of a library of base learners.
  • The Super Learner uses a convex combination with weights minimizing the cross-validated loss, and performs asymptotically as well as the best combination.
  • The Conditional Super Learner assigns each case to one expert through a fitted oracle.
  • Stacking by feature group combines models fitted to separate modalities.
  • The ensemble is assessed by outer resampling, like any tuned procedure.

36.7 Further reading