Case study

COVID precise: Living systematic review of diagnostic and prognostic prediction and machine learning models for COVID-19

In March 2020, the COVID-19 pandemic plunged the world into turmoil. Doctors everywhere found themselves confronted with questions. Which patients were at risk of severe illness? And what were an individual’s chances of survival in the intensive care unit? Laure Wynants immediately realised that she, as a statistician, could help them find the answers by identifying accurate prediction models. Her initiative was eagerly embraced by experts in the Netherlands and abroad. Together, they developed a plan to systematically review models and papers being published on the topic. Through high-quality methodological research, Wynants and her team were able to provide doctors and policymakers on the front lines with a useful tool. The living systematic review was even adopted as the standard by the WHO. It received praise for its speed, relevance, and methodological quality. And that wasn’t all. More generally, Wynants’s approach has served as a wake-up call for the quality of prediction models.

Precise Risk Estimation to optimise COVID-19 Care for Infected or Suspected patients in diverse sEttings

Start project:
Project website: www.covprecise.org

Important decisions

With the pandemic threatening to overwhelm all national health services, it was essential to make early predictions of disease progression to guide preventive, diagnostic and therapeutic decision-making. Studies had already shown that prediction models could be used for various diseases. The same must be true for COVID, thought Wynants. Predictions could then help inform all kinds of important decisions. For example, when testing capacity was still limited: does this person need to be tested? Or, at the GP surgery: can this patient recover at home or do they require hospitalisation? In the hospital: can this patient be admitted to the COVID ward or do they need intensive care? And in the ICU: what is the prognosis for this patient? How long will we continue to provide care?

The gold standard

From the moment Wynants began her search for useful prediction models, the initiative took off. Some 50 researchers and 40 institutes in the Netherlands and abroad were keen to work with her. The BMJ, a renowned journal, invited Wynants and her colleagues to conduct its first-ever living review. In the absence of tools available for use in clinical practice, the WHO adopted their recommendations. The team continuously updated the review, identifying accurate tools and offering suggestions. They reviewed tools ranging from traditional prediction models based on regression models and machine learning to automatic artificial intelligence-based CT image analysis. At each stage of the pandemic, they selected the best state-of-the-art tools. Wynants’s review became the gold standard for practitioners, researchers and policymakers alike. By bridging the gaps between these stakeholders, it had a significant positive impact on society.

Contact

Laure Wynants, PhD
laure.wynants@maastrichtuniversity.nl

Quality

But the review was not just a powerful weapon in the fight against coronavirus. It has also highlighted the importance of conducting sound methodological research. Most models were found to be of poor quality, especially those published in the initial phase of the pandemic. Several models turned out to be even worse at predicting disease progression than using age as the only predictor. The number of high-quality, representative and large datasets did grow over time, albeit slowly. But the vast majority of studies continued to be at high risk of bias due to poor methodologies and statistics – a problem that could easily be prevented or solved by focusing on adequate methods in teaching, research funding and publications.

Wynants’s international team ended up screening 126,978 titles and including 412 studies describing 731 new prediction models or validations. They were able to identify just a handful of reliable tools. The Qcovid models, the PRIEST score, Carr’s model, the ISARIC4C Deterioration model, and the Xie model were the only models to show adequate performance in predicting COVID-19 disease progression in studies at low risk of bias.

What’s next? Three main issues

The pandemic is over, and the living review has been closed. But Wynants’s work is far from done. She is turning her attention to three main issues. Firstly, she would like to draw up recommendations for model developers – judging from the sheer number of citations of the living review, there is a knowledge gap to be filled. Secondly, she plans to use the data for international validations to see if specific models perform equally well in other circumstances. One such validation study has already been published in the BMJ. Most of the models reviewed were developed using data and variables from wealthy countries; in collaboration with the WHO, a validation study has now been launched in low- and middle-income countries. There are lessons to be drawn from this. It is necessary to develop versions of models for low- and middle-income countries using only patient characteristics and widely available clinical tests as input variables; in addition, it is important to support data collection in those settings and to develop tools to combat the continued existence and growth of inequality. Finally, a model has been developed for general practitioners. GPs were largely left out in the cold during the pandemic, when it was they who often had to make important decisions – such as which patients needed to go to hospital or receive antiviral treatment.

Text: Eline Dekker
Translation: Emdash

Research team

The COVID PRECISE consortium consists of clinical and prediction model experts across the globe. It now includes over 50 investigators who actively collaborate with the (meta-)data of hospitals and practices of numerous countries, among which, Belgium, Sweden, England, Italy, Spain, Denmark, Ireland, Turkey, Switzerland, Germany, Brazil, United States and Mexico.

The steering group of COVID PRECISE is:

Laure Wynants, PhD, Care and Public Health Research Institute Maastricht, Netherlands
Maarten van Smeden, PhD, Julius Center for Health Sciences and Primary Care Utrecht, Netherlands
Prof.dr. Carl Moons, Julius Center for Health Sciences and Primary Care Utrecht, Netherlands
Ben Van Calster, PhD, KU Leuven / LUMC Leiden, Belgium / The Netherlands

Collaborating partners

Involved research line

Optimising Patient Care

Our most important publications in scientific journals

Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, Bonten MMJ, Dahly DL, Damen JAA, Debray TPA, de Jong VMT, De Vos M, Dhiman P, Haller MC, Harhay MO, Henckaerts L, Heus P, Kammer M, Kreuzberger N, Lohmann A, Luijken K, Ma J, Martin GP, McLernon DJ, Andaur Navarro CL, Reitsma JB, Sergeant JC, Shi C, Skoetz N, Smits LJM, Snell KIE, Sperrin M, Spijker R, Steyerberg EW, Takada T, Tzoulaki I, van Kuijk SMJ, van Bussel B, van der Horst ICC, van Royen FS, Verbakel JY, Wallisch C, Wilkinson J, Wolff R, Hooft L, Moons KGM, van Smeden M. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020 Apr 7;369:m1328. doi: 10.1136/bmj.m1328.
Valentijn M T de Jong, Rebecca Z Rousset, Neftalí Eduardo Antonio-Villa, Arnoldus G Buenen, Ben Van Calster, Omar Yaxmehen Bello-Chavolla, Nigel J Brunskill, Vasa Curcin, Johanna A A Damen, Carlos A Fermín-Martínez, Luisa Fernández-Chirino, Davide Ferrari, Robert C Free, Rishi K Gupta, Pranabashis Haldar, Pontus Hedberg, Steven Kwasi Korang, Steef Kurstjens, Ron Kusters, Rupert W Major, Lauren Maxwell, Rajeshwari Nair, Pontus Naucler, Tri-Long Nguyen, Mahdad Noursadeghi, Rossana Rosa, Felipe Soares, Toshihiko Takada, Florien S van Royen, Maarten van Smeden, Laure Wynants, Martin Modrák; CovidRetro collaboration; Folkert W Asselbergs, Marijke Linschoten; CAPACITY-COVID consortium; Karel G M Moons, Thomas P A Debray 4. Clinical prediction models for mortality in patients with covid-19: external validation and individual participant data meta-analysis. BMJ. 2022 Jul 12;378:e069881. doi: 10.1136/bmj-2021-069881.

Invited talks
Several invited talks (>20) at national and international conferences and meetings