9  Performance Metrics in rtemis

rtemis computes performance metrics for every fitted model, on the training set and on every test set. This chapter shows where the metrics are stored, computes them directly from predictions, examines the effect of the classification threshold, and reads each metric against its baseline (Chapter 8).

9.1 Metrics stored with a model

The logistic regression of Chapter 3 is fitted again with 10-fold cross-validation:

hf <- read_heart_failure()
mod_hf <- train(
  hf,
  hyperparameters = setup_GLM(),
  outer_resampling_config = setup_KFold(n_resamples = 10L, seed = 2026L),
  execution_config = setup_SerialExecution(seed = 2026L),
  question = hf_question
)

<Resampled Classification Model>
GLM (GLM (generalized linear model))
⟳ Tested using 10 independent folds.

  <Resampled Classification Training Metrics>
  Aggregate Confusion Matrix across resamples.
                     Predicted
          Reference  Yes  No    
                Yes  111   375
                 No   62  2062

  Showing mean (sd) across resamples.
        Sensitivity: 0.228 (0.025)
        Specificity: 0.971 (0.006)
  Balanced Accuracy: 0.600 (0.011)
                PPV: 0.644 (0.042)
                NPV: 0.846 (3.8e-03)
                 F1: 0.336 (0.027)
           Accuracy: 0.833 (4.7e-03)
                AUC: 0.780 (0.010)
        Brier Score: 0.123 (1.6e-03)

  <Resampled Classification Test Metrics>
  Aggregate Confusion Matrix across resamples.
                     Predicted
          Reference  Yes  No   
                Yes    9   45
                 No    9  227

  Showing mean (sd) across resamples.
        Sensitivity: 0.170 (0.147)
        Specificity: 0.962 (0.047)
  Balanced Accuracy: 0.566 (0.065)
                PPV: NA (NA)
                NPV: 0.836 (0.029)
                 F1: NA (NA)
           Accuracy: 0.814 (0.040)
                AUC: 0.709 (0.117)
        Brier Score: 0.140 (0.015)

The metrics_training and metrics_test properties of a resampled model hold the metrics of every resample, in res_metrics, and their means and standard deviations across resamples, in mean_metrics and sd_metrics:

mod_hf@metrics_test@mean_metrics

  sensitivity specificity balanced_accuracy ppv    npv  f1 accuracy    auc
1        0.17       0.962             0.566 NaN 0.8355 NaN   0.8142 0.7085
  brier_score
1      0.1395

The mean PPV and F1 are undefined: in 2 of the 10 test folds, the model predicted no deaths at the default threshold of 0.5, so \(\mathrm{TP} + \mathrm{FP} = 0\) and PPV has no value.

9.2 Metrics from predictions

Each case appears in exactly one test fold of a k-fold cross-validation, so the test predictions of all folds together form one out-of-fold prediction for every case. The shared setup file defines oof_predictions(), which collects them from a resampled model:

oof_hf <- oof_predictions(mod_hf)
head(oof_hf)

   resample observed predicted    prob
     <char>   <fctr>    <fctr>   <num>
1:   Fold_1      Yes        No 0.25314
2:   Fold_1      Yes       Yes 0.81553
3:   Fold_1      Yes        No 0.46501
4:   Fold_1      Yes        No 0.30682
5:   Fold_1      Yes        No 0.08717
6:   Fold_1      Yes        No 0.30741

classification_metrics() computes the metrics of class predictions and, when given probabilities, the AUC and Brier score:

classification_metrics(
  true_labels = oof_hf[["observed"]],
  predicted_labels = oof_hf[["predicted"]],
  predicted_prob = oof_hf[["prob"]]
)

<Classification Metrics>
                   Predicted
        Reference  Yes  No   
              Yes    9   45
               No    9  227

                   Overall  
      Sensitivity  0.167  
      Specificity  0.962  
Balanced Accuracy  0.564  
              PPV  0.500  
              NPV  0.835  
               F1  0.250  
         Accuracy  0.814  
              AUC  0.715  
      Brier Score  0.140  

     Positive Class Yes

Metrics computed on the pooled predictions differ from the mean of the per-fold metrics. The pooled AUC compares cases from different folds, whose predictions come from different models, while the per-fold AUC compares only cases within a fold. Both are in use; rtemis reports the mean of per-fold metrics in its summaries, and plot_roc() draws the pooled curve.

9.3 The threshold

At a threshold of 0.5, the model identifies few of the patients who died. The predicted probabilities show why:

draw_histogram(
  oof_hf[["prob"]],
  group = oof_hf[["observed"]],
  bin_edges = seq(0, 1, by = 0.05),
  xlab = "Out-of-fold predicted probability of death"
)

Since 18.6% of patients died, most predicted probabilities are low: the median is 0.13 and 90% are below 0.42. A threshold of 0.5 lies above most probabilities, including those of most patients who died. The function below recomputes the class predictions at a given threshold:

metrics_at <- function(threshold) {
  predicted <- factor(
    ifelse(oof_hf[["prob"]] >= threshold, "Yes", "No"),
    levels = c("No", "Yes")
  )
  m <- classification_metrics(oof_hf[["observed"]], predicted, oof_hf[["prob"]])
  data.table(
    threshold = threshold,
    m@metrics[["overall"]][c("sensitivity", "specificity", "ppv", "npv", "balanced_accuracy")]
  )
}
by_threshold <- rbindlist(lapply(c(0.1, 0.2, 0.3, 0.5), metrics_at))
knitr::kable(by_threshold, digits = 3)

threshold	sensitivity	specificity	ppv	npv	balanced_accuracy
0.1	0.815	0.441	0.250	0.912	0.628
0.2	0.574	0.750	0.344	0.885	0.662
0.3	0.389	0.860	0.389	0.860	0.625
0.5	0.167	0.962	0.500	0.835	0.564

At a threshold of 0.2, close to the prevalence, sensitivity is 0.574 and specificity 0.750; 34.4% of patients predicted to die did die. The probabilities and the AUC are the same at every threshold; only the conversion to classes changes. Which threshold is appropriate depends on the relative cost of missing a death and of a false alarm, which is a property of the decision the model informs (Chapter 40).

9.4 ROC curve and AUC

plot_roc() draws the ROC curves of a fitted model’s training and test predictions:

plot_roc(mod_hf)

draw_roc() from rtemis.draw draws the curve from any labels and probabilities:

draw_roc(oof_hf[["observed"]], oof_hf[["prob"]])

9.5 Probability metrics

The Brier score and log loss evaluate the probabilities directly (Equation 8.4). The baseline is a constant probability equal to the prevalence:

y01 <- as.integer(oof_hf[["observed"]] == "Yes")
p <- oof_hf[["prob"]]
prevalence <- mean(y01)
c(
  brier = mean((y01 - p)^2),
  brier_baseline = prevalence * (1 - prevalence),
  log_loss = -mean(y01 * log(p) + (1 - y01) * log(1 - p)),
  log_loss_baseline = -(prevalence * log(prevalence) + (1 - prevalence) * log(1 - prevalence))
)

            brier    brier_baseline          log_loss log_loss_baseline 
           0.1396            0.1515            0.4451            0.4807 

The model improves on the baseline in both scores. The constant baseline uses the prevalence of the full dataset, which in practice would be estimated from the training set; review(), below, uses the training proportion of each resample.

9.6 Regression metrics

For the diabetes data, the regression metrics of a cross-validated linear model are:

db <- read_diabetes()
mod_db <- train(
  db,
  hyperparameters = setup_GLM(),
  outer_resampling_config = setup_KFold(n_resamples = 10L, seed = 2026L),
  execution_config = setup_SerialExecution(seed = 2026L),
  question = diabetes_question
)

<Resampled Regression Model>
GLM (GLM (generalized linear model))
⟳ Tested using 10 independent folds.

  <Resampled Regression Training Metrics>
  Showing mean (sd) across resamples.
   MAE: 43.239 (0.482)
   MSE: 2853.158 (57.301)
  RMSE: 53.413 (0.537)
    R²: 0.519 (0.009)

  <Resampled Regression Test Metrics>
  Showing mean (sd) across resamples.
   MAE: 44.131 (4.284)
   MSE: 2984.432 (539.155)
  RMSE: 54.432 (4.901)
    R²: 0.493 (0.089)

regression_metrics() computes the same metrics from predictions. The baseline predicts the training mean of each fold for every test case:

oof_db <- oof_predictions(mod_db)
regression_metrics(oof_db[["observed"]], oof_db[["predicted"]])

<Regression Metrics>
 MAE: 44.14
 MSE: 2986.27
RMSE: 54.65
  R²: 0.50

fold_means <- vapply(
  seq_along(mod_db@outer_resampler@resamples),
  function(i) mean(db[["progression"]][mod_db@outer_resampler@resamples[[i]]]),
  numeric(1)
)
names(fold_means) <- names(mod_db@y_test)
baseline_pred <- fold_means[oof_db[["resample"]]]
regression_metrics(oof_db[["observed"]], baseline_pred)

<Regression Metrics>
 MAE: 65.80
 MSE: 5935.55
RMSE: 77.04
  R²: -9.6e-04

The model’s out-of-fold RMSE, 54.6, is 29.1% lower than the baseline’s, 77.0. The \(R^2\) of the baseline is slightly below 0, since each fold’s training mean differs from the mean of the outcomes in its test fold. plot_true_pred() plots observed against predicted values:

plot_true_pred(mod_db)

9.7 Multiclass metrics

The multiclass metrics can be illustrated by dividing the diabetes outcome into tertiles, low, medium and high progression, and assigning each out-of-fold prediction to the tertile into which it falls:

cuts <- quantile(db[["progression"]], c(0, 1 / 3, 2 / 3, 1))
to_tertile <- function(x) {
  cut(x, cuts, include.lowest = TRUE, labels = c("Low", "Medium", "High"))
}
predicted_tertile <- to_tertile(pmin(pmax(oof_db[["predicted"]], min(cuts)), max(cuts)))
classification_metrics(to_tertile(oof_db[["observed"]]), predicted_tertile)

<Classification Metrics>
                   Predicted
        Reference  Low  Medium  High  
              Low   66      75     7
           Medium   21      97    29
             High    2      56    89

                   Overall  
Balanced Accuracy  0.570  
               F1  0.576  
         Accuracy  0.570  
                   Low    Medium  High   
      Sensitivity  0.446  0.660   0.605
      Specificity  0.922  0.556   0.878
Balanced Accuracy  0.684  0.608   0.742
              PPV  0.742  0.425   0.712
              NPV  0.768  0.766   0.817
               F1  0.557  0.517   0.654

Per-class metrics treat each tertile in turn as the positive class. Discretizing a numeric outcome discards the distances between its values: an error between adjacent tertiles counts the same as an error between the lowest and the highest.

9.8 Baselines in review()

review() reports each metric against its baseline, computed within each resample:

review(mod_hf)

<SupervisedReview>
  GLM (generalized linear model) was used for classification. Mean balanced accuracy was 0.60 in the training set and 0.57 in the test set across 10 independent folds.
  10 resamples: 290 out-of-sample predictions of 290 cases, at least 260 training cases per resample; 11 predictors (4.4 minority-class training cases per predictor).

  Performance, mean (SD) over 10 resamples
                            Training           Test  Training - test
    Sensitivity        0.228 (0.025)  0.170 (0.147)            0.058
    Specificity        0.971 (0.006)  0.962 (0.047)            0.009
    Balanced Accuracy  0.600 (0.011)  0.566 (0.065)            0.034
    PPV                0.644 (0.042)                                
    NPV                0.846 (0.004)  0.836 (0.029)            0.011
    F1                 0.336 (0.027)                                
    Accuracy           0.833 (0.005)  0.814 (0.040)            0.018
    AUC                0.780 (0.010)  0.709 (0.117)            0.072
    Brier Score        0.123 (0.002)  0.140 (0.015)           -0.016

  Baseline comparisons, mean over resamples
    Accuracy 0.814 vs 0.814 (most common training class); better in 4 of 10 resamples compared
    Balanced Accuracy 0.566 vs 0.500 (chance level); better in 7 of 10 resamples compared
    AUC 0.709 vs 0.500 (chance level); better in 10 of 10 resamples compared
    Brier Score 0.140 vs 0.151 (training proportion as probability); better in 7 of 10 resamples compared

  Findings
    * FOLD_VARIATION (note): Test balanced accuracy ranged from 0.479 to 0.679 across 10 resamples (mean 0.566, SD 0.065); 7 of 10 resamples in which the metric is defined outperformed their baseline. Resamples share training cases, so their results are not independent, and no confidence interval or test is computed from them.
    * FEW_CASES_PER_PREDICTOR (note): There are 4.364 minority-class training cases per learner column (11 columns), fewer than 10. This is a rule of thumb from logistic regression (events per variable), not a validated sample-size requirement for every algorithm.
      Suggestion: Judge the model by held-out performance. Consider a regularized or sparse algorithm, or a decomposition step in train(), which is fitted on training cases only.
    * PRESELECTION_RISK (note): With few cases per predictor, predictors are often selected before modeling. Selection or filtering of predictors done on all the data before it was passed to train() biases evaluation; this review cannot determine whether that happened. Preprocessing, decomposition and tuning within train() use training cases only.
      Suggestion: Avoid selecting predictors on all the data before train(). To reduce the predictors, use a sparse algorithm or a decomposition step in train(); any selection done outside rtemis must use only the training cases of each split.

  Limitations
    - Whether the model is useful depends on the question it addresses, the costs of different errors, and how its predictions will be used; performance metrics cannot establish this.
    - Preprocessing, decomposition and tuning done by rtemis are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data was passed to rtemis -- selecting or filtering predictors, transforming or removing cases using all the data, or choosing among models by their test performance -- can bias evaluation; this review cannot determine whether that happened.
    - Performance estimates apply to cases from the population the data was drawn from; performance on a different population or setting may differ.

The review states, for each resample, whether the model outperformed the baseline: the most common training class for accuracy, chance level for balanced accuracy and AUC, and the training proportion as a constant probability for the Brier score. It also reports the variation of the metrics across folds and the number of minority-class cases per predictor (Section 6.7).