(C) PLOS One
This story was originally published by PLOS One and is unaltered.
 . . . . . . . . . .



A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality [1]

['Brian Critelli', 'Department Of Gastroenterology', 'Hepatology', 'Weill Cornell Medical College', 'New York', 'United States Of America', 'Amier Hassan', 'Ila Lahooti', 'Ohio State University Wexner Medical Center', 'Columbus']

Date: 2025-03 

Abstract Background An accurate prognostic tool is essential to aid clinical decision-making (e.g., patient triage) and to advance personalized medicine. However, such a prognostic tool is lacking for acute pancreatitis (AP). Increasingly machine learning (ML) techniques are being used to develop high-performing prognostic models in AP. However, methodologic and reporting quality has received little attention. High-quality reporting and study methodology are critical for model validity, reproducibility, and clinical implementation. In collaboration with content experts in ML methodology, we performed a systematic review critically appraising the quality of methodology and reporting of recently published ML AP prognostic models. Methods/findings Using a validated search strategy, we identified ML AP studies from the databases MEDLINE and EMBASE published between January 2021 and December 2023. We also searched pre-print servers medRxiv, bioRxiv, and arXiv for pre-prints registered between January 2021 and December 2023. Eligibility criteria included all retrospective or prospective studies that developed or validated new or existing ML models in patients with AP that predicted an outcome following an episode of AP. Meta-analysis was considered if there was homogeneity in the study design and in the type of outcome predicted. For risk of bias (ROB) assessment, we used the Prediction Model Risk of Bias Assessment Tool. Quality of reporting was assessed using the Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis—Artificial Intelligence (TRIPOD+AI) statement that defines standards for 27 items that should be reported in publications using ML prognostic models. The search strategy identified 6,480 publications of which 30 met the eligibility criteria. Studies originated from China (22), the United States (4), and other (4). All 30 studies developed a new ML model and none sought to validate an existing ML model, producing a total of 39 new ML models. AP severity (23/39) or mortality (6/39) were the most common outcomes predicted. The mean area under the curve for all models and endpoints was 0.91 (SD 0.08). The ROB was high for at least one domain in all 39 models, particularly for the analysis domain (37/39 models). Steps were not taken to minimize over-optimistic model performance in 27/39 models. Due to heterogeneity in the study design and in how the outcomes were defined and determined, meta-analysis was not performed. Studies reported on only 15/27 items from TRIPOD+AI standards, with only 7/30 justifying sample size and 13/30 assessing data quality. Other reporting deficiencies included omissions regarding human–AI interaction (28/30), handling low-quality or incomplete data in practice (27/30), sharing analytical codes (25/30), study protocols (25/30), and reporting source data (19/30). Conclusions There are significant deficiencies in the methodology and reporting of recently published ML based prognostic models in AP patients. These undermine the validity, reproducibility, and implementation of these prognostic models despite their promise of superior predictive accuracy. Registration Research Registry (reviewregistry1727)

Author summary Why was this study done? Acute pancreatitis is a common gastrointestinal disorder that affects a million patients worldwide annually and has an extremely variable disease course.

Using artificial intelligence (AI), many researchers have developed prediction tools to help forecast acute pancreatitis (AP) patients’ course (e.g., tool to predict who will develop a more severe form of disease).

Before using AI-based prediction tools to care for patients, it is extremely important to appraise the study design and conduct of the studies that produced the tools to ensure that the tools will not lead to misleading results and worse yet, widen health disparities. What did the researchers do and find? We conducted a comprehensive assessment of all the published and unpublished AI models for AP patients in the past 3 years by searching all the major medical databases.

We focused on appraising the quality of the methods and whether the model-building project was transparently reported from start to finish, using established quality standards.

Unfortunately, almost all studies omitted taking steps to ensure that AI model will not discriminate against certain populations.

Most studies did not take steps to avoid misleadingly optimistic results.

Many studies did not provide enough information about how the prediction tool was built, their research methods. For example, only 15 out of 27 key reporting items were included in the paper. What do these findings mean? The AI prediction tools used to predict outcomes for AP patients have significant flaws in how they are designed and reported so they are not ready to be implemented into clinical care.

Concerted efforts are needed to improve the quality of the study design, conduct of the study, and reporting of the studies so clinically useful and accurate AI prediction tools can be built and used.

Patient and stakeholder engagement to inform study design and interdisciplinary collaboration between data scientists, clinicians, and content experts are top priorities for future research in this area.

Citation: Critelli B, Hassan A, Lahooti I, Noh L, Park JS, Tong K, et al. (2025) A systematic review of machine learning-based prognostic models for acute pancreatitis: Towards improving methods and reporting quality. PLoS Med 22(2): e1004432. https://doi.org/10.1371/journal.pmed.1004432 Academic Editor: Amit Singal, University of Texas Southwestern Medical Center, UNITED STATES OF AMERICA Received: June 3, 2024; Accepted: January 7, 2025; Published: February 24, 2025 Copyright: © 2025 Critelli et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: The results data are publicly available at https://doi.org/10.6084/m9.figshare.26078743.v1. Funding: The author(s) received no specific funding for this work. Competing interests: The authors have declared that no competing interests exist. Abbreviations: AI, artificial intelligence; AP, acute pancreatitis; AUC, area under the curve; CHARMS, Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies; ML, machine learning; PRISMA, Preferred Reporting Items for Systematic reviews and Meta-Analyses; PROBAST, Prediction Model Risk of Bias Assessment Tool; ROB, risk of bias; TRIPOD+AI, Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis—Artificial Intelligence

Introduction Defined as acute inflammation of the pancreas, acute pancreatitis (AP) remains a common and costly cause of gastrointestinal-related hospitalization, with 1 million new cases each year globally and increasing incidence [1,2]. The etiology of the disease varies across patient demographics, with gallstones and alcohol comprising the majority of adult cases and diverse factors such as hypertriglyceridemia, drugs, infections, or trauma leading to a minority of cases [3]. The severity of AP can be further categorized as mild, moderately severe, or severe, with severe AP being defined by the presence of persistent organ failure [4]. The combination of persistent organ failure and infected pancreatic necrosis defines a ‘critical’ category of AP severity with the highest morbidity and mortality risk [5,6]. Survivors of AP can suffer from long-term sequelae including diabetes mellitus, recurrent or chronic pancreatitis, and pancreatic exocrine insufficiency [3,7–10]. Given the significant short- and long-term morbidity and mortality associated with AP, since 2018, and as recently as July 2024, the National Institute of Health has called for an accurate prognostic model in AP for use in research and the clinical setting [11–13]. Benefits of an accurate prognostic model are many, including enablement of cost-efficient clinical trials through cohort enrichment [14,15], identification of subphenotypes within a cohort that require different treatment strategies [16,17], and prompt triaging of patients in the emergency room [18]. Current prognostic models for AP were developed using regression-based techniques (e.g., Glasgow Criteria, Bedside Index for Severity in Acute Pancreatitis, etc.) which demonstrate suboptimal performance and limited clinical usefulness [19]. For example, in a prospective external evaluation of regression-based models predicting mortality, none of the models tested produced a post-test probability higher than 14% when “positive” [20]. There has been a call for new approaches to improve prediction accuracy [19,21]. Advances in the subset of artificial intelligence (AI) known as machine learning (ML) have facilitated the development of non-regression prediction models, which offer advantages over regression-based models by performing better in diseases with non-linear predictor–outcome relationships such as AP [22]. There has been an increasing number of published ML-based prognostic models that appear to outperform regression-based models [23–25]. However, ML experts have cited concerns regarding methodologic quality, model building practices, and lack of transparent reporting [26–28]. While there continues to be efforts to examine accuracies and clinical utility of published prognostic models in AP [19], collaborations with ML methodologists and focus on methodological and reporting qualities have been lacking. In a collaborative effort between content experts, clinicians, and methodologists, we therefore undertook a systematic review and critical appraisal of recent published studies proposing new non-regression ML-based prognostic models to detail any methodological shortcomings and/or gaps in reporting.

Methods Detailed methodology of this review has been published elsewhere [29]. We conducted a systematic review of all studies published between January 2021 and December 2023 in which a non-regression, ML-based prognostic model in AP was developed and/or validated (either internally or externally), with or without model updating. This review included studies of prospective or retrospective design including post-hoc analysis of clinical trials that: (a) enrolled only adult patients (i.e., 18-years old or older), (b) contained a prognostic model of AP developed with non-regression ML technique(s), (c) predicted any outcome(s) of AP, and (d) published in English. Studies involving participants with chronic pancreatitis, pancreatic cancer, or post-surgical pancreatitis were excluded, as were studies with animals, regression-based models, or models that predict the development of AP instead of disease outcomes. Studies published in abstract form only and review articles were also excluded. We searched the databases MEDLINE (OvidSP) and EMBASE (OvidSP) from January 1, 2021 to December 31, 2023 (Date of search for all data sources, January 31). We also searched pre-print servers medRxiv, bioRxiv, and arXiv for pre-prints registered between January 2021 and December 2023. Our search was limited to the most recent 3 years for the following reasons. (1) Significant advancements in AP management paradigm has led to a significant change in the natural history/prognosis of the disease over the last decade [30–37]. It was important to identify models trained/evaluated on datasets generated from the most recent cohort of AP. (2) New algorithms rapidly emerge, replacing older algorithms, and temporal quality degradation is an established phenomenon in AI models [38]. Validated search strategies [39,40] were used and are listed in S1 and S2 Tables, respectively. For medRxiv, bioRxiv, and arXiv, the search term “acute pancreatitis” to maximize the sensitivity of our search. Covidence software (Melbourne, Australia) was used to screened title-abstract and full text in sequential steps. Each stage required concordance between two independent reviewers (LN, IL, KT, JP, AH, BC, NM, or AL). Disagreements were resolved by a third independent reviewer (PJL or LAC). Included studies were then appraised in terms of risk of bias (ROB) in study design, completeness of reporting, and for summarization of model predictive performances. Necessary data for Prediction Model Risk of Bias Assessment Tool (PROBAST) and Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis—Artificial Intelligence (TRIPOD+AI) evaluation were extracted in accordance with the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modelling Studies (CHARMS) checklist [41]. Methodologic quality assessment The PROBAST was used to assess both ROB in study design of prospective models across four main domains: participants, predictors, outcomes, and analysis [42]. The assessment of Applicability section of PROBAST was planned if metadata were appropriate and feasible for meta-analysis. To optimize the validity of the PROBAST assessment, all evaluators underwent PROBAST rater training, which entailed weekly meetings with an AP content expert trained by PROBAST developers (PJL) to review all 20 signaling questions. Data scientists (JNA, LL, JQ, or DR) and ML content experts (LAC) were engaged to accurately complete CHARMS and PROBAST. Each model was assessed via the PROBAST framework by two independent reviewers (LN, IL, KT, JP, AH, BC, NM, AL, JNA, LL, JQ, or DR), and disagreements were resolved by an independent third reviewer (PJL or LAC). The pair of reviewers comprised a clinician and a data scientist. The ROB in each domain and overall ROB were reported for all studies. Reporting quality assessment To assess the quality of the reporting, we decided to use TRIPOD+AI statement, which contains a comprehensive list of items that need to be reported for papers reporting development and/or validation of prognostic AI model [43]. List of sections and items on this list covers every key part of a manuscript including title, abstract, introduction, methods, results, and discussion. Additionally, it contains items related to open science and patient and public involvement. Summary statistics of quality of reporting according to the standards of TRIPOD+AI [43] were calculated for each study. This review has been registered at Research Registry (reviewregistry1727). All data reporting in this systematic review adhered to the guidelines of Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) and the checklist can be found in a separate supplementary file (S1 PRISMA Checklist).

Discussion In this systematic review, we assessed the quality of the methodology and reporting of studies that develop and/or validated non-regression ML-based models in AP literature. While the performance of the published models was high (mean AUC 0.91), we identified several key limitations in the recently published models. Unfortunately, these shortcomings are like those identified in other fields such as oncology [28] and anesthesiology [73]. First, the concern relates to the high ROB most notably in the statistical analysis section, which can undermine the validity of the models. Second, due to the lack of external validation studies, generalizability of the ML models may be limited. Third relates to open science practice, where in over 90% of the studies, the code was not shared and no information was provided on how the model was built. Additionally, there was a lack of reporting on how the ML model can be implemented in clinical practice. Lastly, none of the studies described potential health inequities among different sociodemographic groups, which risks widening disparities in healthcare, if implemented in real clinical practice. The quality of the statistical analyses is one of the most important facets of model development. The PROBAST ROB tool dedicates nine signaling questions to this domain [42]. Two particularly deficient areas were sample size justification and guarding against overfitting. A robust sample size (especially for a ML model) and guarding against overfitting are critically important. When these steps are omitted, a model may perform well in the development dataset, but the predictive performance may not be reproducible [74]. We found that most published studies developed a model with a sample size of less than 1,000 participants and median events per variable was 9.5. Even for regression-based models, the minimum recommended events per variable is 20 [42]. While events per variable is not a singular reflection of sufficient sample size, it is generally accepted that ML models require much larger sample size (than regression-based models) due to the risk of model instability [75]. Potentially limited generalizability of the published models needs to be highlighted. Only one study conducted external validation but with limitations [60], and all but five studies were single-center design. While AP is a common gastrointestinal disease, with an annual worldwide 1 million new cases a year [76], international or large multi-center consortiums with efforts to build a generalizable model have been lacking. Lack of such collaboration results in siloed attempts at building models that may not be clinically utilized due to poor reproducibility and generalizability. As with the case with the regression-based models [21], we are seeing a similar trend in ML-based models in AP. Ultimately, prognostic models are built to aid clinical decision-making or enhance cohort enrichment in a research study. Therefore, steps need to be taken to thoughtfully consider real-life issues we will face when trying to deploy these models (e.g., ways to deal with missing values in real clinical practice when patients won’t have the data elements necessary for the ML model). We also found key missing items relevant to open science, that limit external validation studies by other investigators and clinical implementation by the hospitals. For example, only five studies shared the code to permit third-party evaluation and implementation [52,56,61,62,65], only three studies gave guidance on how to handle missing data [52,66,71], and one study detailed the specifics of what constitutes human–AI interaction [60]. As important, aspects of model building relevant healthcare equity (e.g., comparison of performance estimates among different sociodemographic subgroups) were not evaluated. Such deficiency leads to a potential to produce a model that widens the socioeconomic disparities [77]. Our study has several strengths. For transparency and rigor of our methodology, we have published our methods and adhered strictly to the standards of TRIPOD-SR/MA. Our work was conducted in collaboration between data scientists, ML methodologist, and content experts in AP, which we believe enhances the reliability of our findings. There are multiple aspects to PROBAST and TRIPOD+AI assessment that require both AP content and ML methodology expertise. Moreover, rigorous internal training for PROBAST assessment preceded the project, enhancing the validity of our ROS assessment. Several limitations deserve mention. Our search strategy extended to only the last 3 years so it is possible that our findings may not be fully representative of all the ML models published for AP thus far. We also recognize that TRIPOD+AI was published in 2024. While the pre-print version of TRIPOD+AI has been available since 2021, it is important to recognize that many investigators will not have been aware of the document. However, this does not diminish the importance of the message of our study and further highlight why TRIPOD+AI needed to be published. Additionally, TRIPOD+AI encapsulates many sound reporting practices that were already well established with its previous version, the TRIPOD statement published in 2015 [78]. Second, while PROBAST was developed by expert methodologists, it is possible that models deemed high ROB by PROBAST may still be valid, reproducible, and generalizable in AP. However, there is emerging data from other diseases that suggest models deemed high ROB by PROBAST perform poorly external validation studies [79]. Furthermore, due to significant study design heterogeneity, we did not perform quantitative analyses so quantitative conclusions are lacking. It is worth noting that the outcomes predicted were overwhelmingly severity-related in the included studies. While predicting severity is important for early inpatient management, additional clinically meaningful outcomes could be considered in the future. These include patient reported outcomes such as quality of life, disability or unemployment after discharge, and outcomes relevant to invasive intervention planning (e.g., a model to predict complication-free resolution of pancreatic necrosis). By focusing on these intermediate and long-term outcomes, the strengths and advantages of ML-based technique (i.e., imaging pattern recognition and analyses of complex multi-dimensional data) can be leveraged to forecast an accurate prognosis which would otherwise be challenging with regression-based models. For example, candidate covariates in such a model could include the extent of pancreatic and extra-pancreatic injury as determined by imaging studies, sets of laboratory values, social determinants of health, and comorbidities. These then could be included in a ML-based model to predict the suggested outcomes mentioned above, or be used to create subphenotypes of patients who survive AP In conclusion, the potential benefit of ML-based prognostic models is evident with an overall high AUC (mean 0.91 ± 0.8SD). However, this study indicates that there should be great caution in implementing the reported models because of the major concerns with the quality of the methodology and reporting. These raise questions about the validity, reproducibility, and generalizability of the prognostic models. It is recommended that AP-specific, standardized methodology that covers all four PROBAST domains and all items within TRIPOD+AI be used in developing and validating ML-based prognostic models. Only then implementation should be considered. Our study findings provide valuable baseline assessment of the quality of methods and reporting of ML-based models in AP. It is also timely given the recent publication of TRIPOD+AI [43], which was published in January of 2024 as an expansion of the original TRIPOD checklist released in 2015 [78]. TRIPOD provided reporting recommendations for prediction model studies and was subsequently adjusted for subsets of prediction model studies (i.e., TRIPOD for abstracts, TRIPOD-Cluster for models with clustered data, TRIPOD-SRMA systematic reviews and meta-analyses, TRIPOD-P for study protocols, and TRIPOD+AI for models with ML methods) [43]. Thus, our study provides an assessment of the landscape of quality of reporting in the AP literature, and hopefully will draw attention to these important facets of conducting and reporting prognostic model studies. It is beyond the scope of this study to provide an exhaustive set of recommendations on how to improve the methods and reporting of AP prognostic model studies. Nevertheless, informed by our study, we have listed high-priority areas of improvement and our suggestions for investigators, journal editors and reviewers in Table 2. PPT PowerPoint slide

PNG larger image

TIFF original image Download: Table 2. High-priority areas in methodology and reporting that could be improved. https://doi.org/10.1371/journal.pmed.1004432.t002

[END]
---
[1] Url: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1004432

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/