out <- file.path(tempdir(), "amlr-reporting")
dir.create(out, showWarnings = FALSE)51 Reproducibility and Reporting in rtemis
rtemis records each analysis in three forms. A configuration describes the requested analysis, every setting of every step, and can be written to and read from a JSON file. A run record describes what a fitted model did: its resolved settings, data fingerprints, fold-level results and software provenance. A write-up turns the record into Methods and Results text. This chapter produces each for the random forest of Section 31.4, and reruns the analysis from its configuration file.
The files are written to a temporary directory:
51.1 A configuration file
setup_SuperConfig() combines the path to a dataset, the outcome, and the configurations of every step. The 60-day heart failure data are written to a file, so that the configuration can refer to them:
hf <- read_heart_failure()
data_path <- file.path(out, "heart_failure_60d.csv")
fwrite(hf, data_path)
config <- setup_SuperConfig(
dat_training_path = data_path,
outcome = "death",
hyperparameters = setup_Ranger(min_node_size = 20L, seed = 2026L),
outer_resampling_config = setup_KFold(n_resamples = 10L, seed = 2026L),
execution_config = setup_SerialExecution(seed = 2026L),
question = hf_question,
outdir = NULL,
verbosity = 0L
)
config_path <- file.path(out, "config.json")
write_config(config, config_path)The file states the analysis in a form that does not depend on R:
{
"$schema": "https://schema.rtemis.org/supervised/r/v1/schema.json",
"outcome": "death",
"hyperparameters": {
"algorithm": "Ranger",
"num_trees": 500,
"importance": "impurity",
"write_forest": true,
"probability": false,
"min_node_size": 20,
"replace": true,
"sample_fraction": 1,
"num_random_splits": 1,
"alpha": 0.5,
"minprop": 0.1,
"poisson_tau": 1,
"scale_permutation_importance": false,
"local_importance": false,
"regularization_factor": 1,
"regularization_usedepth": false,
"keep_inbag": false,
"holdout": false,
"quantreg": false,
"oob_error": true,
"save_memory": false,read_config() reads the file and reconstructs the typed configuration; train() accepts it in place of data and setup arguments:
mod <- train(read_config(config_path))The model fitted from the configuration file reproduces the model fitted interactively in Section 31.4:
mod_interactive <- train(
hf,
hyperparameters = setup_Ranger(min_node_size = 20L, seed = 2026L),
outer_resampling_config = setup_KFold(n_resamples = 10L, seed = 2026L),
execution_config = setup_SerialExecution(seed = 2026L),
verbosity = 0L
)
all.equal(
mod@metrics_test@mean_metrics,
mod_interactive@metrics_test@mean_metrics
)[1] TRUE51.2 The run record
record() returns the run record as a list, and write_record() writes it as JSON. The record holds the resolved configuration, the metrics of every fold, and the provenance of the run:
rec <- record(mod)
names(rec) [1] "$schema" "outcome"
[3] "features" "weights"
[5] "positive_class" "question"
[7] "outdir" "verbosity"
[9] "dat_training_path" "dat_validation_path"
[11] "dat_test_path" "character2factor"
[13] "preprocessor_config" "decomposition_config"
[15] "hyperparameters" "tuner_config"
[17] "outer_resampling_config" "execution_config"
[19] "origin" "folds"
[21] "metrics" "metrics_sd"
[23] "provenance" "session" rec[["provenance"]][["implementation"]]$name
[1] "rtemis"
$version
[1] "1.4.2"
$language
[1] "r"
$language_version
[1] "R version 4.6.1 (2026-06-24)"rec[["provenance"]][["data_training"]][c("algorithm", "hash", "n_rows", "n_cols")]$algorithm
[1] "sha256"
$hash
[1] "7ebb99aaf53bdd6802ddd6f31f3075c3c96f9ee66087a77a4994bc0050d36d85"
$n_rows
[1] 290
$n_cols
[1] 12write_record(mod, file.path(out, "record.json"))The data fingerprint is a SHA-256 hash of the training data as R received it, which identifies the exact data a model was fitted to. The complete session information, the versions of R and of every attached package, is stored with the model:
mod@session_info[["R.version"]][["version.string"]][1] "R version 4.6.1 (2026-06-24)"51.3 Review and write-up
review() reports the model’s performance against baselines, the variation across folds, and findings about the analysis (Section 9.8). writeup() produces Methods and Results text from the recorded model, with tables of hyperparameters and performance:
report <- writeup(mod)
report<SupervisedWriteup>
Methods
Data
The data comprised 290 cases and 11 predictors. The outcome, `death`, had 2
classes: `No` and `Yes`.
Model
Ranger random forest was used for classification, fitted with the ranger
[1] package. Table 1 lists the primary hyperparameters of the algorithm,
every hyperparameter that was tuned or specified, and the values selected
during fitting. Supplementary Table S1 lists every hyperparameter that
applied to the fit, its value, how it was chosen and the values tuning
evaluated.
Evaluation
Performance was estimated by 10-fold cross-validation, stratified by
outcome class (random seed 2026). In each resample, model fitting used only
the training cases, and the model was evaluated on the cases not used for
training.
Performance measures and statistical analysis
Performance was measured by sensitivity, specificity, balanced accuracy,
PPV, NPV, F1, accuracy, AUC and Brier score, with `Yes` as the positive
class. Balanced accuracy is the mean over classes of the proportion of each
class predicted correctly. The Brier score is the mean squared difference
between the predicted probability of the positive class and the outcome.
Performance is reported as the mean and standard deviation over the
resamples. The resamples share training cases, so their estimates are not
independent, and no confidence intervals or tests were computed. In each
resample, the model was compared with a baseline fitted to the training
cases of that resample: always predicting the most common training class
(accuracy), the chance level (balanced accuracy and AUC) and a constant
forecast of the training proportion of the positive class (Brier score).
Software
Analyses were performed in R 4.6.1 [2] with the rtemis package 1.4.2 [3]
and ranger 0.18.0 [1].
Table 1. The primary hyperparameters of the algorithm, every hyperparameter
that was tuned or specified, and the values selected during fitting. Where
resamples differ, each value is listed with the number of resamples that
used it, and resamples in which a hyperparameter had no effect are counted
as not applicable.
Hyperparameter Value
num_trees 500
mtry 3
min_node_size 20
max_depth unset
sample_fraction 1
splitrule gini
seed 2026
max_depth: Unset or 0 means unlimited.
Results
Sample
Outer resampling completed 10 of 10 resamples; the smallest training set
had 260 cases. The data included 236 `No` and 54 `Yes` cases.
Performance
Performance on training and test cases is shown in Table 2. Averaged over
resamples, on the test cases, balanced accuracy was 0.546 (SD 0.068), AUC
was 0.745 (SD 0.118), sensitivity was 0.113 (SD 0.136), specificity was
0.979 (SD 0.036) and Brier score was 0.136 (SD 0.020). Averaged over
resamples, on the training cases, balanced accuracy was 0.687 (SD 0.022).
Comparison with a baseline
The model's accuracy improved on always predicting the most common training
class in 3 of the 10 resamples in which both were defined (mean 0.818
against 0.814). The model's balanced accuracy improved on the chance level
in 5 of the 10 resamples in which both were defined (mean 0.546 against
0.500). The model's AUC improved on the chance level in 10 of the 10
resamples in which both were defined (mean 0.745 against 0.500). The
model's Brier score improved on a constant forecast of the training
proportion of the positive class in 7 of the 10 resamples in which both
were defined (mean 0.136 against 0.151).
Table 2. Performance on training and test cases, mean (SD) over 10
resamples.
Metric Training, mean (SD) Test, mean (SD)
Sensitivity 0.374 (0.044) 0.113 (0.136)
Specificity 0.999 (0.002) 0.979 (0.036)
Balanced Accuracy 0.687 (0.022) 0.546 (0.068)
PPV 0.990 (0.020)
NPV 0.875 (0.007) 0.829 (0.028)
F1 0.542 (0.045)
Accuracy 0.883 (0.007) 0.818 (0.041)
AUC 0.978 (0.004) 0.745 (0.118)
Brier Score 0.076 (0.002) 0.136 (0.020)
References
[1] Wright MN, Ziegler A (2017). "ranger: A Fast Implementation of Random
Forests for High Dimensional Data in C++ and R." Journal of Statistical
Software, 77(1), 1-17. doi:10.18637/jss.v077.i01
https://doi.org/10.18637/jss.v077.i01.
[2] R Core Team (2026). R: A Language and Environment for Statistical
Computing. R Foundation for Statistical Computing, Vienna, Austria.
doi:10.32614/R.manuals https://doi.org/10.32614/R.manuals.
https://www.R-project.org/.
[3] Gennatas ED (2017). Towards Precision Psychiatry: Gray Matter
Development and Cognition in Adolescence. Ph.D. thesis, University of
Pennsylvania.
Supplementary material
Table S1. Every hyperparameter that applied to the fit, its value, how it
was chosen and the values tuning evaluated. Where resamples differ, each
value is listed with the number of resamples that used it, and resamples in
which a hyperparameter had no effect are counted as not applicable.
Hyperparameter Value Chosen by
num_trees 500 default
mtry 3 resolved during fitting
importance impurity default
write_forest true default
probability false default
min_node_size 20 specified
min_bucket 1 resolved during fitting
max_depth unset
replace true default
sample_fraction 1 default
class_weights unset
splitrule gini resolved during fitting
num_random_splits 1 default
alpha 0.5 default
minprop 0.1 default
poisson_tau 1 default
split_select_weights unset
always_split_variables unset
respect_unordered_factors ignore resolved during fitting
scale_permutation_importance false default
local_importance false default
regularization_factor 1 default
regularization_usedepth false default
keep_inbag false default
inbag unset
holdout false default
quantreg false default
time_interest unset
oob_error true default
save_memory false default
verbose true default
node_stats false default
seed 2026 specified
na_action na.learn default
ifw false default
max_depth: Unset or 0 means unlimited.
class_weights: Unset weights every class equally.
split_select_weights: Unset gives every feature the same probability.
always_split_variables: Unset adds none.
inbag: Unset draws each tree's sample by replace and sample_fraction, with
the training case weights as sampling weights where training supplies them.
time_interest: Unset uses the observed event times.
Not recorded by the model
- The source of the data, the study design, the setting and the eligibility
criteria for cases.
- How the outcome and the predictors were defined and measured.
- Missing data and any processing applied before the data was passed to
rtemis.
Every number in the text comes from the fitted object. The write-up contains what the model records; the study-specific elements of Table 50.1, such as the source and period of the data, the inclusion criteria, and the derivation of the outcome, come from the analyst. write_text() writes the review and the write-up to Markdown files, for inclusion in a report:
write_text(report, file.path(out, "methods-results.md"))
write_text(review(mod), file.path(out, "review.md"))
list.files(out)[1] "config.json" "heart_failure_60d.csv" "methods-results.md"
[4] "record.json" "record.session.parquet" "review.md" 51.4 What this book records
The analyses of this book fix the seeds of resampling and of execution, read their data through the functions of the shared setup file, whose steps are derived in Chapter 7, and pass the written question to train() for the principal model of each analysis. The book was rendered with:
c(
R = R.version.string,
rtemis = as.character(packageVersion("rtemis")),
rtemis.draw = as.character(packageVersion("rtemis.draw")),
data.table = as.character(packageVersion("data.table"))
) R rtemis
"R version 4.6.1 (2026-06-24)" "1.4.2"
rtemis.draw data.table
"0.5.4" "1.18.6.1"