6  Data

A predictive model is fitted to a table of cases, but the table is the end of a process of measurement, recording and selection that determines what the model can learn and to whom it applies. This chapter covers the decisions made before any model is fitted: what a case is, how each variable is represented, how the outcome is defined, which features are admissible, and what to inspect.

6.1 Cases and variables

A dataset for supervised learning is a table in which each row is a case, the unit for which a prediction is made, and each column is a variable measured on that unit. This arrangement, with one observational unit per row and one variable per column, is called tidy data (Wickham 2014).

The unit of prediction must be decided from the question. In the heart failure data, a case is a patient. In the Parkinson’s telemonitoring data, a row is a voice recording, and each patient contributes between 101 and 168 recordings. If the question concerns individual recordings, a row is the right unit; but recordings from the same patient are not independent cases, and a model assessed on recordings from patients it was trained on is assessed on partly familiar data. The unit of independence, here the patient, determines how data must be divided for assessment (Chapter 10).

6.2 Variable types

Each variable has a type, which determines the operations that are meaningful on it and how a learning algorithm can use it.

Table 6.1: Variable types
Type Description Examples R class
Continuous Real-valued measurement Serum creatinine, BMI numeric
Count Non-negative integer Number of hospitalizations integer
Binary Two categories Diabetes (no, yes) factor with two levels
Nominal Unordered categories Blood type factor
Ordinal Ordered categories NYHA class I–IV ordered factor
Date and time A point in time Admission date Date, POSIXct
Text Free text Clinical notes character

Categorical variables are often stored as integer codes, such as 0 and 1 for absence and presence of a condition, or 1 and 2 for sex. A code is a label, not a quantity: the difference between codes 2 and 1 has no meaning, and for a variable with more than two categories an algorithm that treats codes as numbers imposes an order and spacing that the categories do not have. Converting categorical variables to factors makes their type explicit, and lets each algorithm encode them appropriately, for example as indicator columns in a linear model (Chapter 12).

A data dictionary records, for every variable, its meaning, type, units, the meaning of each code, the codes used for missing values, and how and when it was measured. Without one, a dataset can be analyzed but not interpreted. The diabetes data illustrate the cost: sex is coded 1 and 2, and the source does not state which code is which.

6.3 The outcome

The outcome must be defined so that its value is unambiguous for every case. Three properties require attention.

Time horizon. An event outcome, such as death or hospital readmission, is defined over a period. “Death within 60 days of baseline” is a well-defined binary outcome; “death during follow-up” is not, if follow-up lengths differ between cases, since a case followed for 10 days has had less opportunity for the event than one followed for 200 days.

Censoring. A case is censored when follow-up ends before the event is observed, for example because the study ended or the patient was lost to contact. For a fixed horizon \(h\), a case censored before \(h\) has unknown status at \(h\). Excluding such cases defines the outcome for the remaining cases, and is a reasonable choice when few cases are affected; if the reasons for early censoring are related to the outcome, exclusion biases the sample. Methods for time-to-event outcomes, survival analysis, use censored cases directly and are outside the scope of this book.

Measurement. An outcome measured with error, or by different methods in different cases, bounds the accuracy that any model can achieve. In the Parkinson’s data, the UPDRS scores were assessed by clinicians at scheduled visits during a six-month trial and linearly interpolated to the times of the voice recordings (Tsanas et al. 2010), so most outcome values are interpolations rather than direct assessments.

6.4 Admissible features

A feature is admissible only if its value is available at the time the prediction is made, for every case to which the model will be applied. A feature that violates this condition carries information about the outcome that will not be available in use, and produces leakage: a model that performs well in assessment and cannot be applied as assessed (Kaufman et al. 2012). Common forms are:

  • Future information. A measurement taken after the prediction time, such as a laboratory value from the day of discharge in a model meant for admission.
  • Proxies of the outcome. A variable that is recorded because the outcome occurred, or that is a component of the outcome. In the heart failure data, the follow-up time ends at death for patients who died, so short follow-up is a consequence of the outcome. In the Parkinson’s data, the motor UPDRS score is a component of the total UPDRS score.
  • Process artifacts. Identifiers, timestamps or data-collection codes that correlate with the outcome through the way the data were assembled, such as a hospital site code when sites differ in case mix.

Admissibility is decided from the question and the data dictionary, not from the data: a leaking feature typically improves every performance estimate.

6.5 Missing values

A value is missing when it was not measured or not recorded. The appropriate treatment depends on the mechanism that produced the missingness (Rubin 1976; Little and Rubin 2019):

  • Missing completely at random (MCAR): the probability of being missing is unrelated to any variable, observed or not.
  • Missing at random (MAR): the probability of being missing depends only on observed variables.
  • Missing not at random (MNAR): the probability of being missing depends on the unobserved value itself, for example when a test is ordered only for patients suspected of an abnormal result.

In predictive modeling, missingness can itself be informative, and the indicator of whether a value is missing can be a useful feature, provided the same process produces missing values when the model is used. Imputation and missingness indicators are covered in Chapter 12.

6.6 Inspection

Before modeling, each variable is examined for its type, range, distribution and missing values, and the dataset for duplicate rows and constant columns. Exploratory data analysis (Tukey 1977) serves two purposes here: detecting errors, such as implausible values, unit inconsistencies or miscoded categories; and understanding the data well enough to interpret the model. Inspection does not include selecting features by their association with the outcome in the full dataset. Any choice informed by the outcome is part of model fitting and must be repeated within each training set, or the performance estimate becomes optimistic (Chapter 12).

6.7 Sample size

The number of cases limits the complexity of the model that can be estimated reliably. For binary outcomes the limiting quantity is the number of cases in the less frequent class, since a model learns about the event mainly from the cases that experienced it. A traditional guideline for logistic regression calls for at least 10 events per estimated coefficient (Peduzzi et al. 1996); Riley et al. (2020) give sample size calculations based on the expected model performance and the outcome prevalence. The heart failure data, with 54 deaths within 60 days and 11 features, are small by these criteria, which is reflected in the variability of performance estimates across resamples (Chapter 3).

6.8 Summary

  • The case is the unit of prediction; the unit of independence determines how data are divided for assessment.
  • Each variable is represented according to its type; categorical codes are converted to factors.
  • The outcome is defined with an explicit time horizon; cases with unknown status at the horizon are identified.
  • Only features available at the time of prediction are admissible; features that encode future information or the outcome itself produce leakage.
  • Inspection detects errors and informs interpretation; outcome-informed feature selection belongs to model fitting.

6.9 Further reading