25  Nearest Neighbors in rtemis

setup_KNN() fits weighted k-nearest-neighbor models with the kknn package (Hechenbichler and Schliep 2004). This chapter examines the effect of \(k\) and of feature scaling, and tunes \(k\) by nested resampling.

setup_KNN()
<KNNHyperparameters>
        hyperparameters: 
                                k: <int> 7
                           kernel: <chr> optimal
                         distance: <nmr> 2.00
                            scale: <lgc> TRUE
                              ifw: <lgc> FALSE
tunable_hyperparameters: <int> 4
  fixed_hyperparameters: <int> 1
                  tuned: <int> -1
              resampled: <int> 0
              n_workers: <int> 1

  No search values defined for tunable hyperparameters.

k is the number of neighbors; kernel the function that weights neighbors by distance, with "optimal" the kernel derived by Hechenbichler and Schliep (2004) and "rectangular" the unweighted average; distance the parameter \(q\) of the Minkowski distance, 2 for Euclidean; and scale = TRUE standardizes the features before computing distances. Factors are one-hot encoded internally.

25.1 The effect of k

db <- read_diabetes()
hf <- read_heart_failure()
cv <- setup_KFold(n_resamples = 10L, seed = 2026L)
k_values <- c(1L, 5L, 15L, 30L, 60L)
knn_db <- rbindlist(lapply(k_values, function(k) {
  mod <- train(
    db,
    hyperparameters = setup_KNN(k = k),
    outer_resampling_config = cv,
    execution_config = setup_SerialExecution(seed = 2026L),
    verbosity = 0L
  )
  data.table(
    k = k,
    rsq_training = mod@metrics_training@mean_metrics[["rsq"]],
    rsq_test = mod@metrics_test@mean_metrics[["rsq"]]
  )
}))
knn_db
       k rsq_training rsq_test
   <int>        <num>    <num>
1:     1       1.0000 -0.06657
2:     5       0.8175  0.33760
3:    15       0.6444  0.43923
4:    30       0.5710  0.45846
5:    60       0.5127  0.45310
draw_line(
  k_values,
  list(Training = knn_db[["rsq_training"]], Test = knn_db[["rsq_test"]]),
  xlab = "k (number of neighbors)",
  ylab = "R-squared"
)

With \(k = 1\) the training \(R^2\) is 1, since each training case is its own nearest neighbor, and the test \(R^2\) is below 0. As \(k\) grows, training \(R^2\) falls and test \(R^2\) rises, then levels off: the bias–variance tradeoff of Section 24.3, with flexibility decreasing from left to right.

For the heart failure data, the AUC and Brier score of the predicted probabilities:

knn_hf <- rbindlist(lapply(k_values, function(k) {
  mod <- train(
    hf,
    hyperparameters = setup_KNN(k = k),
    outer_resampling_config = cv,
    execution_config = setup_SerialExecution(seed = 2026L),
    verbosity = 0L
  )
  data.table(
    k = k,
    auc = mod@metrics_test@mean_metrics[["auc"]],
    brier_score = mod@metrics_test@mean_metrics[["brier_score"]]
  )
}))
knn_hf
       k    auc brier_score
   <int>  <num>       <num>
1:     1 0.5269      0.2518
2:     5 0.6242      0.1777
3:    15 0.6667      0.1503
4:    30 0.7048      0.1445
5:    60 0.7178      0.1418

With 49 deaths in each training set, small neighborhoods contain few deaths, and the estimated probabilities are coarse; performance improves up to the largest \(k\) examined.

25.2 Feature scaling

The diabetes features have different scales: age in years, BMI in kg/m², the serum measurements in their own units. Without standardization:

scaling <- rbindlist(lapply(c(TRUE, FALSE), function(s) {
  mod <- train(
    db,
    hyperparameters = setup_KNN(k = 30L, scale = s),
    outer_resampling_config = cv,
    execution_config = setup_SerialExecution(seed = 2026L),
    verbosity = 0L
  )
  data.table(scale = s, rsq_test = mod@metrics_test@mean_metrics[["rsq"]])
}))
scaling
    scale rsq_test
   <lgcl>    <num>
1:   TRUE   0.4585
2:  FALSE   0.3114

Without scaling, the distance is dominated by the features with the largest standard deviations, s1 (34.6) and s2 (30.4), while s5, one of the most predictive features (Section 15.1), has a standard deviation of 0.52 and contributes almost nothing.

25.3 Tuning k

\(k\) is tuned by nested resampling (Chapter 21):

mod_knn <- train(
  db,
  hyperparameters = setup_KNN(k = tune_over(5L, 10L, 20L, 30L, 50L, 80L)),
  tuner_config = setup_GridSearch(
    resampler_config = setup_KFold(n_resamples = 5L, seed = 2027L)
  ),
  outer_resampling_config = cv,
  execution_config = setup_SerialExecution(seed = 2026L),
  question = diabetes_question
)
<Resampled Regression Model>
KNN (k-Nearest Neighbors (kknn))
⚙ Tuned using exhaustive grid search.
⟳ Tested using 10 independent folds.

  <Resampled Regression Training Metrics>
  Showing mean (sd) across resamples.
   MAE: 40.643 (1.000)
   MSE: 2567.891 (108.159)
  RMSE: 50.665 (1.052)
    R²: 0.567 (0.017)

  <Resampled Regression Test Metrics>
  Showing mean (sd) across resamples.
   MAE: 45.413 (4.753)
   MSE: 3219.920 (587.844)
  RMSE: 56.515 (5.368)
    R²: 0.456 (0.082)

vapply(mod_knn@models, function(m) m@hyperparameters@hyperparameters[["k"]], numeric(1))
 Fold_1  Fold_2  Fold_3  Fold_4  Fold_5  Fold_6  Fold_7  Fold_8  Fold_9 Fold_10 
     30      30      30      30      30      50      30      30      30      30

The tuned kNN model has a mean test \(R^2\) of 0.456, below the 0.493 of the linear model. The result is consistent with the additive models of Section 23.1, in which most features have linear or nearly linear effects: a global model with ten coefficients estimates such a dependence with less variance than averages over local neighborhoods in ten dimensions.