50  Reproducibility and Reporting

A predictive model is the product of a sequence of decisions: the question, the data and their preparation, the outcome definition, the resampling design, preprocessing, the algorithms and their hyperparameters, the selection among models, and the assessment. A report enables readers to evaluate the model only if it describes these decisions, and enables others to reproduce it only if the data, code, configuration and software are available. This chapter describes what reproducibility requires, what a report of a prediction model contains, and how the decisions made during an analysis affect the validity of what is reported.

50.1 Reproducibility

An analysis is reproducible when the same data, code and computational environment produce the same results (Peng 2011). Its elements are (Sandve et al. 2013):

• Data, preserved in the form used, with a record of their source and of every transformation, and a fingerprint, such as a cryptographic hash, that identifies them exactly.
• Code under version control, including the code that prepares the data.
• Configuration: every setting of every step, including defaults, since a default can change between software versions.
• Randomness: the random seeds of every step that uses random numbers, such as resampling, model fitting and tuning.
• Software: the versions of the language, packages and system libraries.

A result that depends on an unrecorded setting or seed cannot be checked. Data and configurations that are findable, accessible, interoperable and reusable, the FAIR principles (Wilkinson et al. 2016), can be reused beyond the original analysis.

Replication is different: obtaining consistent results with new data. For prediction models, it corresponds to external validation, the assessment of the model in a new population or setting (Section 2.4).

50.2 Reporting a prediction model

The TRIPOD statement (Collins et al. 2015), updated as TRIPOD+AI for models developed with machine learning methods (Collins et al. 2024), lists the items that a report of a prediction model includes. They follow the order of the analysis:

Table 50.1: Elements of a prediction model report, after TRIPOD

Element	Content
Question	The outcome, population, prediction time and intended use (Section 1.3)
Data	Source, period, inclusion criteria, sample size, number of events
Outcome	Definition, time horizon, how and when it was ascertained (Section 6.3)
Predictors	Definitions, measurement, timing relative to the prediction (Section 6.4)
Missing data	Amount and handling (Section 12.5)
Analysis	Preprocessing, algorithms, hyperparameters and their selection, resampling design (Chapter 10, Chapter 20)
Performance	Discrimination and calibration, with their variability (Chapter 8, Chapter 42)
Model	The final model, or access to it, and how to apply it to new cases
Limitations	Sample size, generalizability, potential biases

50.3 Decisions that affect validity

A resampling estimate describes the performance of the procedure that was resampled. Decisions made outside that procedure, after seeing results, make the estimate optimistic:

• comparing many models and reporting the best one’s estimate, without a nested or independent assessment of the selection (Section 20.3);
• changing the outcome definition, the inclusion criteria or the features after seeing performance;
• selecting features on the full dataset (Section 13.3).

Each such decision should be part of the resampled procedure, or recorded and reported, with all models that were compared. A written analysis plan, prepared before the outcome data are examined, separates confirmatory from exploratory decisions.

50.4 Summary

• Reproducibility requires the data, code, configuration, seeds and software versions of an analysis.
• Replication with new data corresponds to external validation of a prediction model.
• TRIPOD and TRIPOD+AI list the items of a prediction model report, from the question to the limitations.
• Decisions made after seeing results and outside the resampled procedure bias the reported performance unless they are reported and assessed.

50.5 Further reading

• Peng (2011) and Sandve et al. (2013) on reproducible computational research; Wilkinson et al. (2016) on the FAIR principles.
• Collins et al. (2015) and Collins et al. (2024) on TRIPOD and TRIPOD+AI; Steyerberg (2019) on the development and reporting of clinical prediction models.
• Wikipedia: Reproducibility, FAIR data.

Collins, G. S., J. B. Reitsma, D. G. Altman, and K. G. M. Moons. 2015. “Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD): The TRIPOD Statement.” BMJ 350 (jan07 4): g7594–94. https://doi.org/10.1136/bmj.g7594.

Collins, Gary S, Karel G M Moons, Paula Dhiman, et al. 2024. “TRIPOD+AI Statement: Updated Guidance for Reporting Clinical Prediction Models That Use Regression or Machine Learning Methods.” BMJ 385 (April): e078378. https://doi.org/10.1136/bmj-2023-078378.

Peng, Roger D. 2011. “Reproducible Research in Computational Science.” Science 334 (6060): 1226–27. https://doi.org/10.1126/science.1213847.

Sandve, Geir Kjetil, Anton Nekrutenko, James Taylor, and Eivind Hovig. 2013. “Ten Simple Rules for Reproducible Computational Research.” PLoS Computational Biology 9 (10): e1003285. https://doi.org/10.1371/journal.pcbi.1003285.

Steyerberg, Ewout W. 2019. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. In Statistics for Biology and Health. Springer International Publishing. https://doi.org/10.1007/978-3-030-16399-0.

Wilkinson, Mark D., Michel Dumontier, IJsbrand Jan Aalbersberg, et al. 2016. “The FAIR Guiding Principles for Scientific Data Management and Stewardship.” Scientific Data 3 (1). https://doi.org/10.1038/sdata.2016.18.