hp_spls <- setup_SPLS(
k = tune_over(1L, 2L, 3L, 4L),
eta = tune_over(0.1, 0.3, 0.5, 0.7, 0.9)
)
grid <- tuning_grid(hp_spls)
nrow(grid)[1] 20head(grid) k eta
1 1 0.1
2 2 0.1
3 3 0.1
4 4 0.1
5 1 0.3
6 2 0.321 Hyperparameter Tuning in rtemis
In rtemis, a search space is defined in the setup_*() call of a learner by wrapping candidate values in tune_over(). train() then tunes the hyperparameters by grid search, configured by setup_GridSearch() and passed as tuner_config, within each training set of the outer resampling. This chapter tunes sparse partial least squares on the diabetes data and the elastic net on the heart failure data, and examines the optimism of the inner estimates.
21.1 Defining a search space
tune_over() marks a set of candidate values. Hyperparameters without it are fixed. tuning_grid() lists the combinations a grid search evaluates:
21.2 Configuring the search
setup_GridSearch() sets the inner resampling, the search type, and the selection metric. With metric = NULL, the metric is set from the outcome type: mean squared error for regression and balanced accuracy for classification.
tuner <- setup_GridSearch(
resampler_config = setup_KFold(n_resamples = 5L, seed = 2027L)
)
tuner<GridSearch TunerConfig>
resampler_config:
<KFoldConfig>
n_resamples: 5
stratify_var: NULL
strat_n_bins: 4
id_strat: NULL
seed: 2027
search_type: <chr> exhaustive
randomize_p: <NUL> NULL
metrics_aggregate_fn: <chr> mean
metric: <NUL> NULL
maximize: <NUL> NULL
21.3 Nested resampling
Passing a search space, a tuner, and an outer resampling configuration to train() performs nested resampling: in each of the 10 outer training sets, the 20 combinations are evaluated by 5-fold cross-validation, the best is refitted on the outer training set, and the refitted model is tested on the outer test fold. The number of fits follows Equation 20.2:
10L * (nrow(grid) * 5L + 1L)[1] 1010db <- read_diabetes()
mod_spls <- train(
db,
hyperparameters = hp_spls,
tuner_config = tuner,
outer_resampling_config = setup_KFold(n_resamples = 10L, seed = 2026L),
execution_config = setup_SerialExecution(seed = 2026L),
question = diabetes_question
)<Resampled Regression Model>
SPLS (Sparse Partial Least Squares)
⚙ Tuned using exhaustive grid search.
⟳ Tested using 10 independent folds.
<Resampled Regression Training Metrics>
Showing mean (sd) across resamples.
MAE: 43.437 (0.508)
MSE: 2883.673 (55.224)
RMSE: 53.698 (0.515)
R²: 0.514 (0.009)
<Resampled Regression Test Metrics>
Showing mean (sd) across resamples.
MAE: 44.194 (4.725)
MSE: 2994.913 (547.165)
RMSE: 54.522 (4.969)
R²: 0.492 (0.091)
Each outer fold’s model stores the hyperparameters it selected:
fold k eta
<char> <int> <num>
1: Fold_1 3 0.5
2: Fold_2 4 0.9
3: Fold_3 3 0.7
4: Fold_4 3 0.5
5: Fold_5 3 0.7
6: Fold_6 3 0.7
7: Fold_7 3 0.7
8: Fold_8 3 0.7
9: Fold_9 4 0.7
10: Fold_10 4 0.7The selected values vary between folds, since each is chosen on a different training set. The outer estimate describes the procedure “choose k and eta by 5-fold cross-validation over this grid, then fit”, not any single combination.
21.4 The optimism of the inner estimate
The tuner of each outer fold stores the inner cross-validated metrics of every combination. The inner \(R^2\) of the selected combination, the one with the lowest inner mean squared error, can be compared with the \(R^2\) of the selected model on that fold’s outer test set:
inner_best <- vapply(mod_spls@models, function(m) {
validation <- m@tuner@tuning_results[["validation"]]
validation[which.min(mse), rsq]
}, numeric(1))
outer_test <- vapply(
mod_spls@metrics_test@res_metrics,
function(m) m@metrics[["rsq"]],
numeric(1)
)
optimism <- data.table(
fold = names(outer_test),
inner_best = inner_best,
outer_test = outer_test
)
optimism[, .(inner_best = mean(inner_best), outer_test = mean(outer_test))] inner_best outer_test
<num> <num>
1: 0.4962 0.4917The mean selected inner \(R^2\) exceeds the mean outer \(R^2\) by 0.004. Part of the difference is the optimism of selection (Section 20.3) and part is variation between test folds; with a grid of 20 similar candidates and 442 cases, both are small relative to the fold-to-fold standard deviation of 0.091. The outer estimate is the one to report.
21.5 Tuning the elastic net
For the elastic net, train() selects \(\lambda\) by cv.glmnet() within each training set (Section 19.1). The mixing parameter \(\alpha\) is tuned by grid search. For a binary outcome, the default selection metric, balanced accuracy, depends on the 0.5 threshold, which classifies few patients as dying in these data (Section 9.3). The Brier score, a proper scoring rule, is selected instead, with maximize = FALSE:
hf <- read_heart_failure()
mod_enet <- train(
hf,
hyperparameters = setup_GLMNET(
alpha = tune_over(0, 0.25, 0.5, 0.75, 1),
which_lambda_cv = "lambda.min",
ifw = FALSE
),
tuner_config = setup_GridSearch(
resampler_config = setup_KFold(n_resamples = 5L, seed = 2027L),
metric = "brier_score",
maximize = FALSE
),
outer_resampling_config = setup_KFold(n_resamples = 10L, seed = 2026L),
execution_config = setup_SerialExecution(seed = 2026L),
question = hf_question
)<Resampled Classification Model>
GLMNET (Elastic net (glmnet))
⚙ Tuned using exhaustive grid search.
⟳ Tested using 10 independent folds.
<Resampled Classification Training Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference Yes No
Yes 36 450
No 5 2119
Showing mean (sd) across resamples.
Sensitivity: 0.074 (0.031)
Specificity: 0.998 (2.5e-03)
Balanced Accuracy: 0.536 (0.015)
PPV: 0.888 (0.127)
NPV: 0.825 (4.6e-03)
F1: 0.135 (0.054)
Accuracy: 0.826 (0.006)
AUC: 0.772 (0.012)
Brier Score: 0.129 (2.2e-03)
<Resampled Classification Test Metrics>
Aggregate Confusion Matrix across resamples.
Predicted
Reference Yes No
Yes 3 51
No 2 234
Showing mean (sd) across resamples.
Sensitivity: 0.053 (0.086)
Specificity: 0.992 (0.018)
Balanced Accuracy: 0.523 (0.042)
PPV: NA (NA)
NPV: 0.821 (0.016)
F1: NA (NA)
Accuracy: 0.818 (0.024)
AUC: 0.705 (0.111)
Brier Score: 0.139 (0.012)
Fold_1 Fold_2 Fold_3 Fold_4 Fold_5 Fold_6 Fold_7 Fold_8 Fold_9 Fold_10
0.75 0.50 0.75 0.25 0.75 0.25 0.00 0.00 0.50 0.25 The selected \(\alpha\) ranges from 0 to 0.75 across folds. Within a fold, the inner Brier scores of the best and worst of the five candidates differ by 0.0015 on average, against a standard deviation of the outer test Brier score of 0.012 across folds: the candidates are nearly equivalent, and the choice among them varies with the training set. The tuned procedure has a mean test AUC of 0.705 and Brier score of 0.139, similar to the fixed ridge model of Section 19.4.
21.6 Random search
search_type = "randomized" evaluates a random proportion randomize_p of the grid:
tuner_random <- setup_GridSearch(
resampler_config = setup_KFold(n_resamples = 5L, seed = 2027L),
search_type = "randomized",
randomize_p = 0.25
)
mod_spls_random <- train(
db,
hyperparameters = hp_spls,
tuner_config = tuner_random,
outer_resampling_config = setup_KFold(n_resamples = 10L, seed = 2026L),
execution_config = setup_SerialExecution(seed = 2026L),
verbosity = 0L
)
c(
exhaustive = mod_spls@metrics_test@mean_metrics[["rsq"]],
randomized = mod_spls_random@metrics_test@mean_metrics[["rsq"]]
)exhaustive randomized
0.4917 0.4937 Each inner search evaluated 5 of the 20 combinations, at a quarter of the cost.
21.7 The final model
The final model is obtained by applying the tuning procedure to all the data, without outer resampling. Its hyperparameters are selected by the same inner 5-fold search:
mod_spls_final <- train(
db,
hyperparameters = hp_spls,
tuner_config = tuner,
execution_config = setup_SerialExecution(seed = 2026L),
question = diabetes_question,
verbosity = 0L
)
mod_spls_final@tuner<GridSearch>
An exhaustive grid search of 20 parameter combinations was performed using 5 independent folds.
mse was minimized with the following config:
k: <int> 3
eta: <nmr> 0.70
The nested estimate above, a mean test \(R^2\) of 0.492, is the estimate of the performance to expect from this final model on new patients. review() reports selections at the edge of the search range:
review(mod_spls)<SupervisedReview>
Sparse Partial Least Squares was used for regression. Hyperparameter tuning was performed using exhaustive grid search. Mean R-squared was 0.51 on the training set and 0.49 on the test set across 10 independent folds.
10 resamples: 442 out-of-sample predictions of 442 cases, at least 397 training cases per resample; 10 predictors (39.7 training cases per predictor).
Performance, mean (SD) over 10 resamples
Training Test Training - test
MAE 43.437 (0.508) 44.194 (4.725) -0.757
MSE 2883.673 (55.224) 2994.913 (547.165) -111.240
RMSE 53.698 (0.515) 54.522 (4.969) -0.825
R² 0.514 (0.009) 0.492 (0.091) 0.022
Baseline comparisons, mean over resamples
MSE 2994.913 vs 5933.328 (training mean); better in 10 of 10 resamples compared
MAE 44.194 vs 65.788 (training mean); better in 10 of 10 resamples compared
R² 0.492 vs -0.005 (training mean); better in 10 of 10 resamples compared
Tuning
k: searched 1 to 4 (4 values), 3 of 10 resamples at an extendable edge
eta: searched 0.1 to 0.9 (5 values), 1 of 10 resamples at an extendable edge
Findings
* FOLD_VARIATION (note): Test mean squared error ranged from 2378.794 to 3739.872 across 10 resamples (mean 2994.913, SD 547.165); 10 of 10 resamples in which the metric is defined outperformed their baseline. Resamples share training cases, so their results are not independent, and no confidence interval or test is computed from them.
* TUNING_GRID_EDGE (note): 3 of 10 resamples selected k at the edge of the values searched (1.000 to 4.000); the best value may lie beyond it.
Suggestion: Extend the search range of k past that edge.
* TUNING_GRID_EDGE (note): 1 of 10 resamples selected eta at the edge of the values searched (0.100 to 0.900); the best value may lie beyond it.
Suggestion: Extend the search range of eta past that edge.
Limitations
- Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
- Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
- Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.