set.seed(2026L)
f_true <- function(x) sin(2 * pi * x)
x_train <- seq(0.0125, 0.9875, length.out = 40L)
y_train <- f_true(x_train) + rnorm(40L, sd = 0.3)
x_test <- seq(0.05, 0.95, length.out = 100L)
poly_data <- function(degree) {
terms <- poly(c(x_train, x_test), degree)
colnames(terms) <- paste0("x", seq_len(degree))
list(
training = data.table(terms[1:40, , drop = FALSE], y = y_train),
test = data.table(terms[-(1:40), , drop = FALSE], y = f_true(x_test))
)
}17 Bias–Variance and Learning Curves in rtemis
bias_variance() estimates the squared bias and variance of a learning procedure by refitting it on many resamples of a training set and examining its predictions at a fixed test set. This chapter applies it to simulated data, where the true function is known, and to the diabetes data, where it is not. It then constructs learning curves for the linear models of Section 5.6.
17.1 Simulated data with a known function
As in Chapter 16, the outcome is \(\sin(2\pi x)\) plus noise with standard deviation 0.3. A training set of 40 cases and a test grid of 100 points are created, and the polynomial terms of each degree are computed as features, so that setup_GLM() fits a polynomial regression:
bias_variance() takes the training data, a fixed test set, the true function values at the test cases, a learner, and a resampling configuration that defines the training sets. Here each of 200 training sets is a random 75% subsample of the 40 cases:
bv <- rbindlist(lapply(c(1L, 3L, 6L, 9L), function(d) {
dat <- poly_data(d)
res <- bias_variance(
dat[["training"]],
dat_test = dat[["test"]],
true_values = f_true(x_test),
hyperparameters = setup_GLM(),
resampler_config = setup_StratSub(n_resamples = 200L, train_p = 0.75, seed = 2026L),
execution_config = setup_SerialExecution(seed = 2026L),
verbosity = 0L
)
data.table(
degree = d,
bias_squared = res@mean_bias_squared,
variance = res@mean_variance
)
}))
bv degree bias_squared variance
<int> <num> <num>
1: 1 0.18947 0.001911
2: 3 0.02524 0.002457
3: 6 0.02333 0.005574
4: 9 0.02646 0.013072Squared bias drops between degrees 1 and 3 and changes little after, and variance increases with degree, as in Figure 16.1. The resamples are drawn from one training set rather than from the population, so the average prediction approximates the fit to these 40 cases, and the squared bias includes the deviation of this particular sample from the true function. A simulation study, which draws new training sets from the population, separates the two; bias_variance() provides the estimate that is available when only one sample exists.
17.2 Observed data
For real data the true function is unknown. Without true_values, bias_variance() measures squared deviation from the observed outcomes, which adds the irreducible error to the squared bias. The sum remains comparable between procedures fitted to the same data, since they share the same irreducible error.
The linear models of the diabetes data with main effects and with all two-way interactions (Section 5.6) are compared on a fixed test set of 30% of the cases:
db <- read_diabetes()
expand_features <- function(dat, order) {
formula <- if (order == 1L) {
progression ~ .
} else {
as.formula(paste0("progression ~ (.)^", order))
}
data.table(model.matrix(formula, data = dat)[, -1], progression = dat[["progression"]])
}
split <- resample(db, setup_StratSub(n_resamples = 1L, train_p = 0.7, seed = 2026L))
bv_db <- rbindlist(lapply(1:2, function(k) {
dat <- expand_features(db, k)
res <- bias_variance(
dat[split[[1]]],
dat_test = dat[-split[[1]]],
hyperparameters = setup_GLM(),
resampler_config = setup_StratSub(n_resamples = 100L, train_p = 0.75, seed = 2026L),
execution_config = setup_SerialExecution(seed = 2026L),
verbosity = 0L
)
data.table(
model = c("Main effects", "Two-way interactions")[k],
bias_squared_plus_noise = res@mean_bias_squared,
variance = res@mean_variance
)
}))
bv_db model bias_squared_plus_noise variance
<char> <num> <num>
1: Main effects 3022 34.67
2: Two-way interactions 3867 333.97The variance of the interaction model is 9.6 times that of the main-effects model. In both models, the first column, which includes the irreducible error, is much larger than the variance; the decomposition does not separate how much of it is bias and how much is noise.
17.3 Learning curves
A learning curve requires models fitted to training sets of increasing size. setup_StratSub() with a given train_p draws training sets of that proportion and tests on the remaining cases. Repeating this for proportions from 20% to 90%, with 25 resamples each, gives the mean training and test RMSE at each training set size:
proportions <- seq(0.2, 0.9, by = 0.1)
lc <- rbindlist(lapply(1:2, function(k) {
dat <- expand_features(db, k)
rbindlist(lapply(proportions, function(p) {
mod <- train(
dat,
hyperparameters = setup_GLM(),
outer_resampling_config = setup_StratSub(
n_resamples = 25L, train_p = p, seed = 2026L
),
execution_config = setup_SerialExecution(seed = 2026L),
verbosity = 0L
)
data.table(
model = c("Main effects", "Two-way interactions")[k],
n_training = round(p * nrow(dat)),
training = mod@metrics_training@mean_metrics[["rmse"]],
test = mod@metrics_test@mean_metrics[["rmse"]]
)
}))
}))sizes <- lc[model == "Main effects", n_training]
draw_line(
sizes,
list(
`Main effects, training` = lc[model == "Main effects", training],
`Main effects, test` = lc[model == "Main effects", test],
`Interactions, training` = lc[model == "Two-way interactions", training],
`Interactions, test` = lc[model == "Two-way interactions", test]
),
palette = unname(rtemis_colors[c("orange", "orange", "blue", "blue")]),
line_style = c("dashed", "solid", "dashed", "solid"),
xlab = "Training set size",
ylab = "RMSE",
ylim = c(30, 140)
)The main-effects model shows the high-bias pattern: training and test RMSE are within 7.1 units of each other at every size, and the test RMSE changes little, from 57.4 to 54.3, between the smallest and the largest training set. More data of the same kind would improve this model little. The interaction model shows the high-variance pattern: its test RMSE falls from 129.9 to 56.4 as the training set grows, and remains above that of the main-effects model at every size in these data. The gap between its training and test error narrows with every increase in training set size. Penalizing its coefficients (Chapter 18) reduces the variance at a given size.