5  Supervised Learning in rtemis

This chapter introduces the rtemis interface for supervised learning: the form of the input data, the configuration objects, train(), the fitted model object, and prediction on new cases. It then reproduces the pattern of Figure 4.2 on real data, with training and test error diverging as the model class grows.

5.1 Input data

train() takes a data.table or data.frame in which each row is a case, the outcome is the last column, and every other column is a feature. Features are numeric or factors. The type of the outcome selects the task: a numeric outcome defines regression, and a factor defines classification.

The diabetes data have a numeric outcome, progression, in the last column:

db <- read_diabetes()
str(db)

Classes 'data.table' and 'data.frame':  442 obs. of  11 variables:
 $ age        : int  59 48 72 24 50 23 36 66 60 29 ...
 $ sex        : Factor w/ 2 levels "1","2": 2 1 2 1 1 1 2 2 2 1 ...
 $ bmi        : num  32.1 21.6 30.5 25.3 23 22.6 22 26.2 32.1 30 ...
 $ bp         : num  101 87 93 84 101 89 90 114 83 85 ...
 $ s1         : int  157 183 156 198 192 139 160 255 179 180 ...
 $ s2         : num  93.2 103.2 93.6 131.4 125.4 ...
 $ s3         : num  38 70 41 40 52 61 50 56 42 43 ...
 $ s4         : num  4 3 4 5 4 2 3 4.55 4 4 ...
 $ s5         : num  4.86 3.89 4.67 4.89 4.29 ...
 $ s6         : int  87 69 85 89 80 68 82 92 94 88 ...
 $ progression: int  151 75 141 206 135 97 138 63 110 310 ...
 - attr(*, ".internal.selfref")=<pointer: 0x103974e10> 

When the outcome is in another position, set_outcome() moves the named column to the end:

head(set_outcome(db, "bmi"), 3)

     age    sex    bp    s1    s2    s3    s4    s5    s6 progression   bmi
   <int> <fctr> <num> <int> <num> <num> <num> <num> <int>       <int> <num>
1:    59      2   101   157  93.2    38     4 4.860    87         151  32.1
2:    48      1    87   183 103.2    70     3 3.892    69          75  21.6
3:    72      2    93   156  93.6    41     4 4.673    85         141  30.5

5.2 Configuration

Each algorithm is configured by a setup_*() function, whose arguments are the algorithm’s hyperparameters. available_supervised() lists the algorithms:

available_supervised()

                     BART: Bayesian Additive Regression Trees (stochtree)
                     CART: CART decision tree (rpart)
  ConditionalSuperLearner: Conditional SuperLearner: an oracle routes each case to one of a library of experts
                      GAM: GAM (generalized additive model)
                      GLM: GLM (generalized linear model)
                   GLMNET: Elastic net (glmnet)
                  GLMTree: Model-Based Recursive Partitioning: a tree with a GLM in each leaf
                      HAL: Highly Adaptive Lasso (hal9001)
                 Isotonic: Isotonic regression
             MonotonicHAL: Monotonic Highly Adaptive Lasso (hal9001)
                      KNN: k-Nearest Neighbors (kknn)
                LightCART: Single LightGBM tree (CART mode)
                 LightGBM: LightGBM gradient boosting
                  LightRF: LightGBM random forest
             LightRuleFit: LightRuleFit (LightGBM rules + GLMNET)
                    LINAD: Linear Additive Tree
              LINADForest: Bagged ensemble of Linear Additive Trees
                     MARS: Multivariate Adaptive Regression Splines (earth)
                      MLP: Multilayer perceptron (torch)
         ModalityStacking: ModalityStacking: stacked ensemble with one learner per feature group
                     NNLS: Non-negative least squares (nnls)
                   Ranger: Ranger random forest
             SuperLearner: SuperLearner: cross-validated stacked ensemble
                LinearSVM: SVM with linear kernel (e1071)
                RadialSVM: SVM with radial kernel (e1071)
                     SPLS: Sparse Partial Least Squares
                   TabNet: TabNet neural network

A configuration is an object that can be printed and inspected before training. setup_GLM() configures a generalized linear model, which is a linear regression for a numeric outcome and a logistic regression for a binary outcome (Chapter 14):

setup_GLM()

<GLMHyperparameters>
        hyperparameters: 
                         ifw: <lgc> FALSE
tunable_hyperparameters: <int> 1
  fixed_hyperparameters: <int> 0
                  tuned: <int> -1
              resampled: <int> 0
              n_workers: <int> 1

  No search values defined for tunable hyperparameters.

setup_SerialExecution() configures how computations run, here on a single process, and sets the random seed that makes the run reproducible.

5.3 Training and testing on a single split

The simplest held-out design divides the cases once into a training set and a test set. resample() with setup_StratSub() draws a random subsample of the cases for training, stratified on the outcome so that its distribution is similar in both sets (Chapter 10). With n_resamples = 1L and train_p = 0.75, it returns one set of training indices containing 75% of the cases:

split <- resample(
  db,
  setup_StratSub(n_resamples = 1L, train_p = 0.75, seed = 2026L)
)
db_training <- db[split[[1]], ]
db_test <- db[-split[[1]], ]
c(training = nrow(db_training), test = nrow(db_test))

training     test 
     331      111 

train() fits the model on the training set and, when dat_test is given, evaluates it on the test set. While it runs, train() reports each step with a timestamp and the name of the function that produced the message:

mod_glm <- train(
  db_training,
  dat_test = db_test,
  hyperparameters = setup_GLM(),
  execution_config = setup_SerialExecution(seed = 2026L),
  question = diabetes_question
)

2026-10-11 12:40:27 Checking data is ready for training... ✔ [check_supervised]
2026-10-11 12:40:27 ▶ [train]
2026-10-11 12:40:27 Training set: 331 cases x 10 features. [summarize_supervised]
2026-10-11 12:40:27     Test set: 111 cases x 10 features. [summarize_supervised]
2026-10-11 12:40:27 // CPU | serial | 1 worker: algorithm 1 thread, tuning 1, outer resampling 1 [msg_resources]
2026-10-11 12:40:27 Shared memory (auto): not sharing training data (26.2 kB) -- this run transfers nothing to workers. [report_shared_memory]
2026-10-11 12:40:27 Training GLM Regression... [train]
2026-10-11 12:40:27 Checking data is ready for training... ✔ [check_supervised]

<Regression>
GLM (GLM (generalized linear model))

  <Training Regression Metrics>
   MAE: 42.76
   MSE: 2778.85
  RMSE: 52.71
    R²: 0.53

  <Test Regression Metrics>
   MAE: 44.66
   MSE: 3138.31
  RMSE: 56.02
    R²: 0.48

2026-10-11 12:40:27 Done in 0.80 seconds. [train]

The printed summary reports regression metrics on the training and test sets: mean absolute error (MAE), mean squared error (MSE), its square root (RMSE), and \(R^2\), the proportion of the outcome’s variance explained (Chapter 8). Subsequent chapters hide the progress messages and show the summary.

5.4 The fitted model

train() returns an object of class Regression or Classification. Its properties are accessed with @:

class(mod_glm)

[1] "rtemis::Regression" "rtemis::Supervised" "S7_object"         

mod_glm@metrics_test

<Test Regression Metrics>
 MAE: 44.66
 MSE: 3138.31
RMSE: 56.02
  R²: 0.48

describe() gives a one-sentence description of the fitted model:

describe(mod_glm)

GLM (generalized linear model) was used for regression. R-squared was 0.53 on the training set and 0.48 on the test set. 

predict() applies the model to new cases. features() returns the feature columns of a dataset, dropping the outcome:

head(predict(mod_glm, features(db_test)))

    1     2     3     4     5     6 
129.1 104.6 168.2 149.8 123.1 123.6 

plot_true_pred() plots observed against predicted outcomes for the training and test sets:

plot_true_pred(mod_glm)

5.5 Classification

A factor outcome defines classification. For a binary outcome, the second factor level is the positive class. In the heart failure data, death has levels "No" and "Yes", so "Yes" is the positive class:

hf <- read_heart_failure()
levels(hf[["death"]])

[1] "No"  "Yes"

hf_split <- resample(
  hf,
  setup_StratSub(n_resamples = 1L, train_p = 0.75, seed = 2026L)
)
hf_training <- hf[hf_split[[1]], ]
hf_test <- hf[-hf_split[[1]], ]
mod_hf <- train(
  hf_training,
  dat_test = hf_test,
  hyperparameters = setup_GLM(),
  execution_config = setup_SerialExecution(seed = 2026L),
  question = hf_question
)

<Classification>
GLM (GLM (generalized linear model))

  <Training Classification Metrics>
                     Predicted
          Reference  Yes  No   
                Yes   10   30
                 No    3  174

                     Overall  
        Sensitivity  0.250  
        Specificity  0.983  
  Balanced Accuracy  0.617  
                PPV  0.769  
                NPV  0.853  
                 F1  0.377  
           Accuracy  0.848  
                AUC  0.788  
        Brier Score  0.122  

     Positive Class Yes

  <Test Classification Metrics>
                     Predicted
          Reference  Yes  No  
                Yes    2  12
                 No    1  58

                     Overall  
        Sensitivity  0.143  
        Specificity  0.983  
  Balanced Accuracy  0.563  
                PPV  0.667  
                NPV  0.829  
                 F1  0.235  
           Accuracy  0.822  
                AUC  0.653  
        Brier Score  0.142  

     Positive Class Yes

For a binary classification model, predict() returns the estimated probability of the positive class:

prob_test <- predict(mod_hf, features(hf_test))
head(prob_test)

         Yes
[1,] 0.01619
[2,] 0.18548
[3,] 0.41722
[4,] 0.19115
[5,] 0.08889
[6,] 0.21682

A class prediction follows from a threshold on this probability. The class predictions stored in the model use a threshold of 0.5:

table(
  observed = hf_test[["death"]],
  predicted = mod_hf@predicted_test
)

        predicted
observed No Yes
     No  58   1
     Yes 12   2

With a threshold of 0.5, the model predicts death for 3 of the 14 test patients who died within 60 days. The threshold is a decision, separate from the model, and the probabilities carry more information than any single threshold. Chapter 8 separates the evaluation of probabilities from the evaluation of class predictions, and Chapter 40 discusses the choice of threshold.

5.6 Overfitting with a growing model class

A linear model with the ten diabetes features has 11 coefficients. Adding all products of pairs of features, the two-way interactions, enlarges the model class to linear functions of 55 terms; adding all products of three features enlarges it to 175 terms. model.matrix() constructs these terms, and the same training and test cases are used for each model:

expand_features <- function(dat, order) {
  formula <- if (order == 1L) {
    progression ~ .
  } else {
    as.formula(paste0("progression ~ (.)^", order))
  }
  x <- model.matrix(formula, data = dat)[, -1]
  data.table(x, progression = dat[["progression"]])
}
db_order <- lapply(1:3, function(k) expand_features(db, k))
vapply(db_order, ncol, integer(1)) - 1L

[1]  10  55 175

mods_order <- lapply(db_order, function(dat) {
  train(
    dat[split[[1]], ],
    dat_test = dat[-split[[1]], ],
    hyperparameters = setup_GLM(),
    execution_config = setup_SerialExecution(seed = 2026L),
    verbosity = 0L
  )
})
rsq <- data.table(
  terms = c("Main effects", "Two-way interactions", "Three-way interactions"),
  n_features = vapply(db_order, ncol, integer(1)) - 1L,
  rsq_training = vapply(mods_order, function(m) m@metrics_training@metrics[["rsq"]], numeric(1)),
  rsq_test = vapply(mods_order, function(m) m@metrics_test@metrics[["rsq"]], numeric(1))
)
knitr::kable(rsq, digits = 3)

terms	n_features	rsq_training	rsq_test
Main effects	10	0.529	0.478
Two-way interactions	55	0.623	0.386
Three-way interactions	175	0.753	-0.393

Training \(R^2\) rises from 0.529 to 0.753 as terms are added. Test \(R^2\) falls from 0.478 to -0.393. \(R^2\) is computed as one minus the ratio of the model’s squared error to the squared error of a constant prediction equal to the mean of the test outcomes, so a negative value means that the model predicts the test cases less accurately than that constant. With 175 terms estimated from 331 training cases, the model fits noise that does not recur in new cases, the pattern of Figure 4.2. plot_true_pred() shows the discrepancy for the three-way interaction model:

plot_true_pred(mods_order[[3]])

A model class can be large and still generalize, if its fitting procedure controls complexity. Chapter 18 fits the same expanded models with penalties on the size of the coefficients.