2  Questions and Generalization

Every analysis of data is an attempt to answer a question, and the kind of question determines what is estimated, how the estimate generalizes beyond the data at hand, and how its quality is judged. This chapter distinguishes descriptive, predictive and causal questions, and contrasts the two forms of generalization that underlie them: from a sample to the population it was drawn from, and from a set of observed cases to new cases.

2.1 Three kinds of question

Hernán et al. (2019) classify the tasks of data science into three kinds.

Description summarizes features of a population, using data from a sample of it. Examples are the proportion of heart failure patients who die within 60 days of baseline, the mean ejection fraction among those who die and those who survive, or the association between serum creatinine and death after accounting for age. The target is a population quantity, such as a proportion, a mean, or a regression coefficient.

Prediction maps the measurements available for an individual to the value of an outcome for that individual. An example is the probability that a patient admitted today will die within 60 days, computed from the patient’s age, ejection fraction and laboratory values. The target is a function, and its quality is measured by how close its outputs are to the outcomes of new individuals.

Causal inference estimates what would happen under an intervention. An example is the change in 60-day mortality that would result from lowering serum creatinine by a given amount. The target is a contrast between potential outcomes under different interventions, only one of which is observed for any individual (Hernán and Robins 2020; Pearl 2009).

Leek and Peng (2015) give a finer classification of six types (descriptive, exploratory, inferential, predictive, causal and mechanistic), in which exploratory and inferential questions extend description to the search for patterns and to formal generalization from a sample, and mechanistic questions extend causal ones to the mechanism by which a cause produces its effect. The three-way classification suffices for this book, which is concerned with prediction.

Table 2.1: Three kinds of question

	Descriptive	Predictive	Causal
Target	A population quantity	A function from features to outcome	The effect of an intervention
Example	Association of creatinine with death	Probability of death for a new patient	Effect of lowering creatinine on death
Judged by	Bias and precision of the estimate	Error on new cases	Bias and precision of the effect estimate, given causal assumptions
Requires	A representative sample	Cases from the population of intended use	A causal model and its assumptions, e.g. no unmeasured confounding

2.2 One model, three questions

The same statistical model can serve all three purposes, and the purpose determines how its output is read. Consider a logistic regression of 60-day mortality on baseline measurements \(x_1, \dots, x_p\):

\[ \log \frac{P(Y = 1 \mid x)}{1 - P(Y = 1 \mid x)} = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p . \tag{2.1}\]

• As a description, the coefficient \(\beta_j\) is the difference in the log odds of death associated with a one-unit difference in \(x_j\) among patients with equal values of the other features, in the population from which the sample was drawn. The quantity of interest is \(\beta_j\) itself, together with a measure of its uncertainty.
• As a prediction, the fitted equation is a function that takes a new patient’s measurements and returns a probability. The individual coefficients are of secondary interest; what matters is how well the probabilities match the outcomes of new patients.
• As a causal estimate, \(\beta_j\) would be read as the effect on the log odds of death of intervening to change \(x_j\) by one unit. This reading requires assumptions that the data cannot verify: that the other features in the model include every common cause of \(x_j\) and \(Y\), that none of them is itself affected by \(x_j\), and that the intervention is well defined. Reporting every coefficient of a multivariable model as if each were a causal effect is a recognized error, since a set of covariates that identifies the effect of one feature generally does not identify the effects of the others (Westreich and Greenland 2013).

A model that predicts accurately therefore need not have causally interpretable coefficients, and a model built to estimate a causal effect need not predict accurately (Shmueli 2010; Breiman 2001).

2.3 Generalization from a sample to a population

A descriptive or inferential analysis treats the \(n\) observed cases as a sample from a population and asks what the sample says about a population quantity \(\theta\), such as a coefficient \(\beta_j\). An estimator \(\hat\theta\), computed from the sample, varies from sample to sample. Its distribution over repeated samples, the sampling distribution, describes how far an estimate is likely to fall from \(\theta\). The standard deviation of the sampling distribution is the standard error. For maximum likelihood estimators, including the coefficients of a generalized linear model, the sampling distribution is approximately normal in large samples, and an approximate 95% confidence interval for \(\beta_j\) is

\[ \hat\beta_j \pm 1.96 \, \widehat{\mathrm{SE}}(\hat\beta_j), \tag{2.2}\]

where the standard error is derived from the curvature of the log-likelihood at its maximum, the Fisher information (AMDS: Estimation).

This form of generalization is analytic: the uncertainty is computed from a probability model of how the sample arose and how the outcome depends on the features. The interval is valid to the extent that the sample is representative of the population, the cases are independent, and the model is correctly specified (AMDS: Inference).

2.4 Generalization to new cases

A predictive analysis asks a different question: how well will a function fitted to these \(n\) cases perform on a new case drawn from the same population? Let \(\hat f\) denote the function fitted to the sample and \(L\) a loss function that measures the discrepancy between an outcome \(y\) and a prediction \(\hat f(x)\), for example the squared error \((y - \hat f(x))^2\). The quantity of interest is the expected prediction error, or risk, of \(\hat f\) on a new case \((X, Y)\) from the population:

\[ \mathrm{Err} = \mathbb{E}_{(X, Y)} \left[ L\big(Y, \hat f(X)\big) \right]. \tag{2.3}\]

The average loss on the cases used to fit \(\hat f\), the apparent or training error,

\[ \overline{\mathrm{err}} = \frac{1}{n} \sum_{i=1}^{n} L\big(y_i, \hat f(x_i)\big), \tag{2.4}\]

is a biased estimate of \(\mathrm{Err}\). The fitting procedure chooses \(\hat f\) to make the loss small on exactly these cases, so the training error is systematically lower than the error on new cases. The difference is the optimism of the training error (Efron 1986). The more flexible the model relative to the number of cases, the larger the optimism.

Predictive generalization is therefore assessed empirically: the function is applied to cases that played no part in fitting it, and its loss on those cases is measured. With a single dataset, this is done by dividing the cases into a training set, used to fit the model, and a test set, used only to measure its error. Resampling repeats the division many times, fitting the model on each training set and measuring its error on the corresponding test set, so that every case contributes to the estimate of error and the variation across divisions is observed (Stone 1974). Chapter 10 develops these methods.

Resampling estimates the performance of a model-building procedure on new cases from the same population as the sample. It does not address performance in a different population or setting, such as another hospital, a later time period, or patients selected by different criteria. Assessing that, called external validation, requires data from the new setting (Steyerberg and Vergouwe 2014).

2.5 Two forms of generalization compared

The two forms of generalization differ in their target, their source of uncertainty and their tools (Table 2.2).

Table 2.2: Generalization in descriptive and predictive analysis

	Sample to population	Training cases to new cases
Question	What is the value of a population quantity?	How well will the fitted function predict for new cases?
Target	A parameter, e.g. \(\beta_j\)	The expected prediction error, \(\mathrm{Err}\)
Source of uncertainty	Sampling variation of the estimate	Sampling variation of the fitted function and of the new cases
Tool	Sampling distribution, standard errors, confidence intervals	Held-out test data and resampling
Validity depends on	Representative sample, correct model specification	Test cases independent of fitting, drawn from the population of intended use
Role of model correctness	Central: a misspecified model biases \(\hat\beta_j\)	Secondary: a misspecified model can still predict well, and its error is measured directly

The last row explains much of the difference between statistical modeling and machine learning practice. Because predictive performance is measured directly on held-out data, a predictive analysis can compare models with very different structures, including models whose parameters have no individual interpretation, by their error on new cases alone. Chapter 3 fits one logistic regression model to the heart failure data and examines it in both ways.

2.6 Summary

• Descriptive, predictive and causal questions estimate different targets: a population quantity, a function, and the effect of an intervention.
• The same model can serve all three purposes; the question determines how its output is read and how it is evaluated.
• Descriptive analysis generalizes from a sample to a population through the sampling distribution of an estimator.
• Predictive analysis generalizes from the training cases to new cases, and its performance is estimated on cases not used for fitting.
• The training error is an optimistic estimate of the error on new cases.

2.7 Further reading

• AMDS: Inference, in particular From inference to prediction; Estimation.
• Hernán et al. (2019) on the classification of data science tasks; Leek and Peng (2015) on types of data analysis question.
• Breiman (2001) and Shmueli (2010) on the distinction between explanatory and predictive modeling.
• Efron (1986) on the optimism of the apparent error rate.
• Wikipedia: Statistical inference, Predictive modelling, Causal inference, Generalization error.

Breiman, Leo. 2001. “Statistical Modeling: The Two Cultures (with Comments and a Rejoinder by the Author).” Statistical Science 16 (3). https://doi.org/10.1214/ss/1009213726.

Efron, Bradley. 1986. “How Biased Is the Apparent Error Rate of a Prediction Rule?” Journal of the American Statistical Association 81 (394): 461–70. https://doi.org/10.1080/01621459.1986.10478291.

Hernán, Miguel A., John Hsu, and Brian Healy. 2019. “A Second Chance to Get Causal Inference Right: A Classification of Data Science Tasks.” CHANCE 32 (1): 42–49. https://doi.org/10.1080/09332480.2019.1579578.

Hernán, Miguel A., and James M. Robins. 2020. Causal Inference: What If. Chapman & Hall/CRC.

Leek, Jeffery T., and Roger D. Peng. 2015. “What Is the Question?” Science 347 (6228): 1314–15. https://doi.org/10.1126/science.aaa6146.

Pearl, Judea. 2009. Causality: Models, Reasoning, and Inference. Cambridge University Press. https://doi.org/10.1017/cbo9780511803161.

Shmueli, Galit. 2010. “To Explain or to Predict?” Statistical Science 25 (3): 289–310. https://doi.org/10.1214/10-sts330.

Steyerberg, E. W., and Y. Vergouwe. 2014. “Towards Better Clinical Prediction Models: Seven Steps for Development and an ABCD for Validation.” European Heart Journal 35 (29): 1925–31. https://doi.org/10.1093/eurheartj/ehu207.

Stone, M. 1974. “Cross-Validatory Choice and Assessment of Statistical Predictions.” Journal of the Royal Statistical Society Series B: Statistical Methodology 36 (2): 111–33. https://doi.org/10.1111/j.2517-6161.1974.tb00994.x.

Westreich, D., and S. Greenland. 2013. “The Table 2 Fallacy: Presenting and Interpreting Confounder and Modifier Coefficients.” American Journal of Epidemiology 177 (4): 292–98. https://doi.org/10.1093/aje/kws412.