12  Preprocessing

Preprocessing transforms the features before a learning algorithm sees them: encoding categories as numbers, placing features on comparable scales, filling in missing values, or selecting a subset of features. Some of these transformations are fixed rules; others are estimated from the data. The second kind is part of the fitted model, and must be treated as such when performance is estimated. This chapter describes the common operations and the principle that governs where they belong.

12.1 Fixed and estimated transformations

A fixed transformation is determined without reference to the data: a unit conversion, a logarithm of a laboratory value, the ratio of two measurements, or an indicator of whether a value exceeds a clinically defined cutoff. Applying it to each case uses only that case’s values, so it can be applied to the whole dataset before resampling without affecting the assessment.

An estimated transformation has parameters computed from the data: centering subtracts a mean, scaling divides by a standard deviation, mean imputation fills in a mean, one-hot encoding uses the set of observed levels, and feature selection uses the association of each feature with the outcome. These parameters are estimated, like the coefficients of a model, and the transformation is part of the model-building procedure. In resampling, they are estimated on each training set and applied unchanged to the corresponding test set. When the model is used, the parameters estimated from the training data are applied to the new cases.

Estimating a transformation on the full dataset before resampling allows information from the test cases into the fitted model, which is a form of leakage (Section 6.4). The size of the resulting bias depends on how much the transformation uses: centering and scaling on the full data usually introduce little bias; feature selection on the full data can introduce a large one. Ambroise and McLachlan (2002) showed that selecting genes by their association with the outcome on all samples, and then cross-validating a classifier on the selected genes, produces near-perfect estimated accuracy from data with no predictive information. Chapter 13 reproduces the effect with features of pure noise.

12.2 Encoding categorical features

Most algorithms operate on numeric inputs. A categorical feature with \(K\) levels is encoded as indicator columns:

  • One-hot encoding creates \(K\) columns, one per level, each equal to 1 for cases at that level and 0 otherwise.
  • Dummy (treatment) coding creates \(K - 1\) columns, omitting a reference level, whose effect is absorbed by the intercept. Linear models with an intercept require it, since the \(K\) one-hot columns sum to the intercept column and the design matrix would be rank-deficient (AMDS: Rank).

Linear models in R construct the dummy coding of factors internally, and tree-based methods can split on factor levels directly. The set of levels is estimated from the training data; a level that appears only in new data has no column, and an encoding must define how to treat it.

12.3 Centering and scaling

Standardization replaces each numeric feature \(x_j\) by \((x_j - \bar x_j)/s_j\), where \(\bar x_j\) and \(s_j\) are its training mean and standard deviation. It changes no information in the data, but it matters for algorithms whose behavior depends on the scale of the features:

  • methods based on distances between cases, such as nearest neighbors, kernel methods and many clustering algorithms, in which a feature measured in larger units dominates the distance;
  • penalized models, in which a penalty on the size of the coefficients penalizes the coefficient of a feature in small units more than that of the same feature in large units (Chapter 18);
  • models fitted by gradient-based optimization, such as neural networks, in which features on very different scales slow convergence.

Unpenalized linear models and decision trees are unaffected by standardization, apart from the scale of the coefficients in the former.

12.4 Transforming skewed features

Strongly right-skewed features, such as serum creatinine or creatinine phosphokinase in the heart failure data (Section 7.2.4), place a few cases far from the rest. In a linear model these cases can dominate the fitted slope. A logarithm, or a power transformation such as the Box–Cox family (Box and Cox 1964), makes the distribution more symmetric and expresses multiplicative differences as additive ones. A logarithm with a fixed offset is a fixed transformation; a Box–Cox transformation whose power is estimated from the data is an estimated one.

12.5 Missing values

Most algorithms cannot use a case with a missing feature value. The options are:

  • Complete-case analysis removes cases with any missing value. It reduces the sample size, can bias the sample unless values are missing completely at random, and cannot be applied to new cases, which still need a prediction.
  • Single imputation replaces each missing value by an estimate: the training mean or median for a numeric feature, the most frequent level for a categorical one, or a prediction from a model of that feature given the others, for example by random forests (Stekhoven and Bühlmann 2012).
  • Multiple imputation creates several completed datasets to propagate the uncertainty of the imputed values into inference (van Buuren and Groothuis-Oudshoorn 2011). It serves descriptive and causal analysis; for prediction, a single imputation model fitted on the training data is applied to new cases.
  • Missingness indicators add, for each feature with missing values, a binary feature equal to 1 where the value was missing. When missingness is informative (Section 6.5), the indicator carries that information to the model.
  • Algorithms with native handling of missing values, such as some implementations of decision trees and gradient boosting, route missing values within the fitted model.

Every imputation model is an estimated transformation and is fitted within each training set.

12.6 Feature filtering and selection

Constant features carry no information and can be removed. A feature that is constant in a training set but not in the full data cannot be estimated from that training set, so the decision is best made once, on rules that do not use the outcome. Feature selection by association with the outcome, whether by univariate tests, by the importance measures of a model, or by stepwise search, is part of model fitting. When it is needed, it is performed within each training set, or replaced by algorithms that select or shrink features as part of their fit, such as the lasso (Chapter 18).

12.7 Operations by algorithm

Table 12.1: Preprocessing requirements by algorithm family
Algorithm family Encoding Scaling Missing values
Linear models (unpenalized) Dummy coding (internal) Not required Imputation
Penalized linear models Dummy or one-hot Required (often internal) Imputation
Nearest neighbors, kernel methods One-hot Required Imputation
Decision trees and tree ensembles Native or one-hot Not required Native in some implementations, or imputation
Neural networks One-hot or embeddings Required Imputation

12.8 Summary

  • Fixed transformations can be applied before resampling; estimated transformations are part of the model and are fitted within each training set.
  • Feature selection using the outcome on the full dataset can produce large optimistic bias.
  • Categorical features are encoded as indicators; standardization matters for distance-based, penalized and gradient-trained models.
  • Missing values are imputed by models fitted on the training data, and missingness indicators preserve informative missingness.

12.9 Further reading