(T-016) Bridging the Gap: A Comprehensive Checklist for Hybrid Pharmacometrics-Machine Learning Model Building to Support Oncology Clinical Development

Anna Fochesato, Ph.D. – Post Doctorate Scientist, Pharma Research and Early Development, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd; Jitao David Zhang, Ph.D. – Expert Scientist, Pharma Research and Early Development, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd; Logan Brooks, M.Sc. – Senior Scientist, Clinical Pharmacology, South San Francisco, USA, Genentech Inc.; Manuel Kohler, M.Sc. – Senior Scientist, Pharma Research and Early Development, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd; Candice Jamois, Pharm.D. – Senior Principal Scientist, Pharma Research and Early Development, Roche Innovation Center Basel, F. Hoffmann-La Roche Ltd

Objectives: Model development recommendations encompass a series of building steps and procedures that have been proven to be effective and thus desirable to be closely followed. Well-established pharmacometrics (PMx) development steps have been beneficial in formalizing modeling and simulation tasks toward informed decision-making and regulatory submission.
In recent years, machine learning (ML) methods have gained popularity in the PMx community because of their potential to decipher relationships between patient characteristics and to predict individual survival. While combining the strengths of PMx and ML into hybrid PMx-ML (hPMxML) models seems promising, careful considerations of model development steps are essential to ensure reliable and transparent conclusions.
Building on a systematic review of the currently published hPMxML models, we report current gaps and propose a checklist of developmental and reporting requirements to predict individual survival from baseline characteristics and early biomarker data in oncology clinical development.

Methods: From published guidelines (TRIPOD [1], TRIPOD-AI [2], MINIMAR [3], and REMARK [4]), we identified components of clinical predictive models to be considered as “adequately documented”, thus deriving a framework to objectively evaluate the quality of published hPMxML examples.

Results: While emphasized by the guidelines, reporting of estimands, success criteria, and potential interactions between treatment and anticipated predictors needs to be improved. Sensitivity to the selection of cut-offs (e.g. for dichotomization), landmarks, or horizons in time-to-event studies was usually absent. Adequacy between selected predictive performance metrics and the initial clinical question was only partially documented. Standardized benchmarking and evaluations of predictor stability and of uncertainty propagation were rare.
Based on these deficiencies, we recommend that building blocks of future hPMxML models for individual survival predictions are:
(1) estimand definition
(2) data processing
(3) model fitting/training on the development set
(3.1) model definition
(3.2) covariate selection in cross-validation
(3.3) parameter tuning in cross-validation
(3.4) convergence assessment
(3.5) model explainability
(3.6) model diagnostics
(3.7) uncertainty quantification
(4) model prediction on the external validation set including time-point/subgroup calibration
(5) sensitivity analysis
(6) applicability domain
(7) benchmark studies

Conclusions: To mitigate existing challenges in hPMxML applications, we have introduced a series of model development steps designed to align with best practices across domains. This structured checklist ensures that model outputs are reliable and can be confidently embraced by clinical stakeholders, thereby advancing the adoption of hPMxML models in drug development.

Citations: [1] Collins, G.S. et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. BMC Med 13, 1 (2015)
[2] Collins, G.S. et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378
[3] Hernandez-Boussard, T. et al. MINIMAR (MINimum Information for Medical AI Reporting): Developing reporting standards for artificial intelligence in health care, Journal of the American Medical Informatics Association, Volume 27, Issue 12, December 2020, Pages 2011–2015
[4] Altman, D.G. et al. Reporting Recommendations for Tumor Marker Prognostic Studies (REMARK): explanation and elaboration. PLoS Med. 2012;9(5):e1001216