(T-071) A Framework for Evaluating Predictive Models Using Synthetic Image Covariates and Longitudinal Data

Simon Deltadahl, MSc – PhD student, Department of Applied Mathematics and Theoretical Physics, University of Cambridge; Andreu Vall, PhD – Machine learning researcher, Pumas-AI; Vijay Ivaturi, PhD – CEO, Pumas-AI

Objectives

We present a novel framework to synthesize data sets with complex covariates, such as medical images, linked to simulated longitudinal patient outcomes. This allows us to evaluate and refine predictive models by providing training data where true relationships are known. Our implementation focuses on generating optical coherence tomography (OCT) scans, but the framework is adaptable to other covariates. We also show that, once we have such data, we can model it and appropriately leverage images to predict longitudinal outcomes.

Methods

We used 109,309 2D OCT image slices to train a model (combining a VAE [1] with Stable Diffusion [2]) to generate images from a 128-dimensional normally distributed latent space. We also defined an NLME model to generate longitudinal observations based on a latent space of three random effects. By introducing covariance in the sampling of the two models' latent space, we generated pairs of images and longitudinal measurements where a tunable proportion of the between-subject variability is predictable by features in the image.

We generated a dataset of 1.1 million OCT scan slices, each paired with five sets of longitudinal measurements where the image has different degrees of predictiveness.

We modeled this data to verify that we could predict outcomes from the images by:

Modeling the longitudinal patient measurements with an NLME model - obtaining empirical Bayes estimates for each patient.
Training a ResNet model [3] to predict a patient's empirical Bayes estimates from their OCT image.
Combining the trained ResNet and NLME models to individualize longitudinal predictions based on the images.

We repeated this for cases where the image could theoretically predict 100%, 50%, 10%, 5.26%, and 2% of the between-subject variability of the longitudinal data.

Results

The fitted models could dramatically improve longitudinal patient predictions by using image covariates. The predictivity of the image appropriately declined as we reduced the correlation between the synthetic images and longitudinal data. Importantly, in all but the 2% predictivity case, we got to within 50% of the theoretically best possible prediction on withheld data—indicating that we can find a signal even if it is weak.

The success of fitting the data also validates our synthesis methodology and mechanism to control the predictive signal we imbue the generated images.

The generated dataset is publicly available [4].

Conclusions

Here, we synthesize and model realistic data with complex covariates and longitudinal patient outcomes. While easily adaptable to other rich data types, we successfully apply the methodology to OCT images. In that process, we also showed that we can utilize images for rich longitudinal predictions.

This methodology enhances our ability to understand and model the impact of complex covariates on patient outcomes, ultimately leading to more accurate individualized patient predictions.

Citations: [1] Kingma, Diederik P., and Max Welling. “Auto-Encoding Variational Bayes.” arXiv Preprint arXiv:1312.6114, 2013, https://doi.org/10.48550/arXiv.1312.6114

[2] Rombach, Robin, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. “High-Resolution Image Synthesis With Latent Diffusion Models,” In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 10684-10695, https://doi.org/10.48550/arXiv.2112.10752

[3] He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. “Deep Residual Learning for Image Recognition.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–78, https://doi.org/10.1109/CVPR.2016.90.

[4] “Deltadahl/OCT-Longitudinal · Datasets at Hugging Face.” Accessed May 15, 2024. https://huggingface.co/datasets/Deltadahl/OCT-Longitudinal, https://doi.org/10.57967/hf/2089