34  Neural Networks

A neural network composes simple functions, each a linear map followed by a fixed nonlinearity, into a function that can approximate almost any relationship between features and outcome. Its parameters are fitted by gradient descent on a loss, with gradients computed by the chain rule. This chapter describes the multilayer perceptron, how it is trained and regularized, and how neural networks for tabular data compare with the tree ensembles of the previous chapters.

34.1 The multilayer perceptron

A unit, or neuron, computes a weighted sum of its inputs plus a bias and applies an activation function \(\sigma\): \(h = \sigma(w^\top x + b)\). A layer applies many units to the same inputs, \(h = \sigma(W x + b)\), with weight matrix \(W\) and bias vector \(b\). A multilayer perceptron (MLP) stacks \(L\) such hidden layers and a final output layer:

\[ h^{(1)} = \sigma\left(W^{(1)} x + b^{(1)}\right), \quad h^{(l)} = \sigma\left(W^{(l)} h^{(l-1)} + b^{(l)}\right), \quad \hat y = g\left(W^{(L+1)} h^{(L)} + b^{(L+1)}\right), \tag{34.1}\]

where \(g\) is the identity for regression, the logistic function for binary classification, and the softmax for multiclass classification (AMDS: A matrix as a linear map). With no hidden layer, the MLP is a generalized linear model: linear regression or logistic regression. The hidden layers compute new features, learned from the data, on which the output layer is a linear model.

The activation must be nonlinear, since a composition of linear maps is linear. Common choices are shown in Figure 34.1: the rectified linear unit \(\mathrm{ReLU}(z) = \max(0, z)\), the logistic sigmoid, and the hyperbolic tangent. A network of ReLU units computes a continuous piecewise linear function, related to the hinge functions of MARS (Section 22.4).

Figure 34.1: Common activation functions.

A network with one hidden layer and enough units can approximate any continuous function on a bounded region to any accuracy (Hornik et al. 1989). The result concerns what networks can represent, not how well a network fitted to finite data generalizes.

34.2 Training

The parameters are fitted by minimizing the training loss, squared error or cross-entropy (Table 4.1), by gradient descent. The gradient of the loss with respect to every weight is computed by backpropagation, the application of the chain rule from the output layer back to the first (Rumelhart et al. 1986) (AMDS: Backpropagation is the chain rule). The loss is not convex in the weights, and different initializations reach different solutions (AMDS: Non-convexity in deep learning).

Each step uses the gradient on a mini-batch of cases rather than the whole training set: stochastic gradient descent (AMDS: Mini-batches). An epoch is one pass over the training set. Adaptive methods such as Adam scale the step for each weight by running estimates of the first and second moments of its gradient (Kingma and Ba 2015) (AMDS: Adaptive methods). The learning rate, the size of each step, is the most influential optimization hyperparameter. Gradient-based training requires numeric features on comparable scales, so numeric features are standardized (Section 12.3).

34.3 Regularization

An MLP with many units has many more parameters than a tabular dataset has cases, and overfits without regularization:

  • Weight decay adds an \(L_2\) penalty on the weights, as ridge regression does (Section 18.2).
  • Dropout sets a random fraction of the units to zero at each training step, which prevents units from depending on specific other units (Srivastava et al. 2014).
  • Early stopping monitors the loss on held-out cases during training and keeps the weights of the epoch with the lowest validation loss, as for boosting (Section 32.3).
  • Batch and layer normalization standardize the inputs of each layer, which stabilizes and speeds training.

The architecture, the number of layers and of units per layer, is itself a hyperparameter, together with the learning rate, batch size, weight decay and dropout. The number of hyperparameters and the variability of training make neural networks more expensive to tune than tree ensembles.

34.4 Categorical features and embeddings

A categorical feature can be one-hot encoded, or represented by an embedding: a vector of a few learned values per level, trained with the rest of the network. Levels with similar effects acquire similar embeddings. Embeddings are useful for features with many levels.

34.5 Neural networks for tabular data

Neural networks are the most accurate methods for images, audio and text, in which the structure of the input, spatial or sequential, is built into the architecture, as in convolutional networks and transformers (LeCun et al. 2015; Goodfellow et al. 2016). For tabular data, in which each column has its own meaning, tree-based models outperformed deep learning models in a benchmark of medium-sized datasets (Grinsztajn et al. 2022).

TabNet is an architecture for tabular data that processes the features in a sequence of decision steps, each of which selects a sparse subset of features through an attention mechanism (Arik and Pfister 2021). The selection masks provide a measure of feature importance.

34.6 Summary

  • An MLP composes linear maps and nonlinear activations; with no hidden layer it is a generalized linear model.
  • One hidden layer with enough units can approximate any continuous function; generalization depends on training and regularization.
  • Networks are trained by stochastic gradient descent with gradients from backpropagation; inputs are standardized.
  • Weight decay, dropout, early stopping and normalization regularize training; the architecture is a hyperparameter.
  • On medium-sized tabular datasets, tree-based models have outperformed neural networks in benchmarks.

34.7 Further reading