• Skip to main content
  • Skip to header right navigation
  • Skip to site footer
CCAIM

CCAIM

Novel AI to transform healthcare

  • Home
  • About
    • Our Aims
    • Leadership
    • Joint Steering Committee
    • Independent Scientific Advisory Board
    • Our students
    • Visitors
    • Vacancies
  • Research
    • Papers
    • Breakthroughs
    • Demonstrators
    • Research Update: COVID-19
    • Blog
  • News
    • Latest News
    • COVID-19 News
  • Events
    • Seminar Series
    • WeCREATE
    • Inaugural Event
    • AI Clinic
  • Summer School
    • Program
    • Topics
    • Speakers
    • Industry
    • Participate
    • Summer School in Numbers
    • Exhibition
    • FAQ
  • Get involved
    • Collaborate
    • PhD Programmes
    • Partners
  • Connect

External Validity of Machine Learning-based Prognostic Scores for Cystic Fibrosis – new publication

21 December 2022 by Andreas Bedorf

A paper recently accepted for publication by PLOS Digital Health explores machine-learning based prognostic scores and their validity when applied to demographically different patient cohorts: the ‘external validity’. The publication is the latest accomplishment in CCAIM’s ongoing efforts to push forward in cystic fibrosis research with the Floto lab and van der Schaar lab on the frontlines. Follow the links for their respective research focuses on cystic fibrosis.

The paper is co-authored by Yuchao Qin, PhD student at the van der Schaar lab, Prof Ahmed Alaa, a van der Schaar lab alumnus and now professor at UC Berkeley, and the CCAIM directors Prof Andres Floto and Prof Mihaela van der Schaar. Its creation was supported by the US Cystic Fibrosis Foundation and the UK Cystic Fibrosis Trust, with the help from Dr Janet Allen (Floto lab).

Cystic fibrosis is a genetic disease that affects multiple organs of a patient, potentially including advanced lung damage that necessitates lung transplantation. Due to the scarcity of donor lungs, precise and timely selection of high-risk patients for lung transplant referral is of paramount importance. Machine learning models have been proven to be a viable tool for improving prognostic accuracy. However, the development of trustworthy machine learning models relies on a large volume of data that is difficult to provide given that cystic fibrosis only affects a small sub-population around the world.

Therefore, it would be desirable to be able to use developed machine learning models from a large population for a demographically different cohort. In this paper, our researchers evaluate the external validity, or applicability of a machine learning model with registry data of cystic fibrosis patients from the UK and Canada.

To achieve this, the authors used a unique approach employing the new state-of-the-art AutoML framework AutoPrognosis, developed by the van der Schaar lab and made available as open source earlier this year.

Fig 1. An overview of the AutoPrognosis framework.

They identified several risk factors and patient subgroups affected by variation across the two countries. FEV1 was discovered to be the most significant risk factor for adverse outcomes in cystic fibrosis patients and its impact on LTx referral was shown as being hugely affected by the cross-population variations in the external validation set from Canada. The appropriate consideration of these variation-associated subgroups helped with the adaptation of machine learning models for a different population.

Fig 2. Diagnostic accuracy of individual variables in the UK and Canadian cystic fibrosis cohorts.

These valuable insights highlight the importance of external validation of machine learning models for cystic fibrosis outcome diagnostic. For the first time, the authors provide useable guidance for the adaptation of high-precision machine learning models to different populations, and inspire new research on applying modern transfer learning methods for fine-tuning models in highly variable environments.

External Validity of Machine Learning-based Prognostic Scores for Cystic Fibrosis: A Retrospective Study using the UK and Canadian Registries

Yuchao Qin, Ahmed Alaa, Andres Floto, Mihaela van der Schaar

Abstract

Precise and timely referral for lung transplantation is critical for the survival of cystic fibrosis patients with terminal illness. While machine learning (ML) models have been shown to achieve significant improvement in prognostic accuracy over current referral guidelines, the external validity of these models and their resulting referral policies has not been fully investigated.

Here, we studied the external validity of machine learning-based prognostic models using annual follow-up data from the UK and Canadian Cystic Fibrosis Registries. Using a state-of-the-art automated ML framework, we derived a model for predicting poor clinical outcomes in patients enrolled in the UK registry, and conducted external validation of the derived model using the Canadian Cystic Fibrosis Registry. In particular, we studied the effect of (1) natural variations in patient characteristics across populations and (2) differences in clinical practice on the external validity of ML-based prognostic scores.

Overall, decrease in prognostic accuracy on the external validation set (AUCROC: 0.88, 95% CI 0.88-0.88) was observed compared to the internal validation accuracy (AUCROC: 0.91, 95% CI 0.90-0.92). Based on our ML model, analysis on feature contributions and risk strata revealed that, while external validation of ML models exhibited high precision on average, both factors (1) and (2) can undermine the external validity of ML models in patient subgroups with moderate risk for poor outcomes. A significant boost in prognostic power (F1 score) from 0.33 (95% CI 0.31-0.35) to 0.45 (95% CI 0.45-0.45) was observed in external validation when variations in these subgroups were accounted in our model. Our study highlighted the significance of external validation of ML models for cystic fibrosis prognostication.

The uncovered insights on key risk factors and patient subgroups can be used to guide the cross-population adaptation of ML-based models and inspire new research on applying transfer learning methods for fine-tuning ML models to cope with regional variations in clinical care.

For a full list of the Centre’s publications, click here.

In September 2022, the van der Schaar lab dedicated a Revolutionizing Healthcare engagement session to cystic fibrosis. You can find the recorded session in full here.

Learn more about AutoPrognosis and its potential for clinical research here.

Category: Cystic fibrosis, News, Organ transplantation, Research News
Previous Post: « Alexander Gimson – ML in Transplantation Medicine

Navigation

Home

News

About

University of Cambridge

  • University A-Z
  • Contact the University
  • Accessibility
  • Data Protection
  • Terms and conditions

Newsletter

Sign-up for updates on our research.

Follow us

  • Twitter
  • LinkedIn
  • YouTube

Copyright © 2023 CCAIM

Return to top

We are using cookies to give you the best experience on our website.

You can find out more about which cookies we are using or switch them off in settings.

CCAIM
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.

Strictly Necessary Cookies

Strictly Necessary Cookie should be enabled at all times so that we can save your preferences for cookie settings.

If you disable this cookie, we will not be able to save your preferences. This means that every time you visit this website you will need to enable or disable cookies again.

3rd Party Cookies

This website uses Google Analytics to collect anonymous information such as the number of visitors to the site, and the most popular pages.

Keeping this cookie enabled helps us to improve our website.

Please enable Strictly Necessary Cookies first so that we can save your preferences!