(C) PLOS One
This story was originally published by PLOS One and is unaltered.
 . . . . . . . . . .



Assessing eligibility for lung cancer screening using parsimonious ensemble machine learning models: A development and validation study [1]

['Thomas Callender', 'Department Of Respiratory Medicine', 'University College London', 'London', 'United Kingdom', 'Fergus Imrie', 'Department Of Electrical', 'Computer Engineering', 'University Of California', 'Los Angeles']

Date: 2023-10 

Abstract Background Risk-based screening for lung cancer is currently being considered in several countries; however, the optimal approach to determine eligibility remains unclear. Ensemble machine learning could support the development of highly parsimonious prediction models that maintain the performance of more complex models while maximising simplicity and generalisability, supporting the widespread adoption of personalised screening. In this work, we aimed to develop and validate ensemble machine learning models to determine eligibility for risk-based lung cancer screening. Methods and findings For model development, we used data from 216,714 ever-smokers recruited between 2006 and 2010 to the UK Biobank prospective cohort and 26,616 high-risk ever-smokers recruited between 2002 and 2004 to the control arm of the US National Lung Screening (NLST) randomised controlled trial. The NLST trial randomised high-risk smokers from 33 US centres with at least a 30 pack-year smoking history and fewer than 15 quit-years to annual CT or chest radiography screening for lung cancer. We externally validated our models among 49,593 participants in the chest radiography arm and all 80,659 ever-smoking participants in the US Prostate, Lung, Colorectal and Ovarian (PLCO) Screening Trial. The PLCO trial, recruiting from 1993 to 2001, analysed the impact of chest radiography or no chest radiography for lung cancer screening. We primarily validated in the PLCO chest radiography arm such that we could benchmark against comparator models developed within the PLCO control arm. Models were developed to predict the risk of 2 outcomes within 5 years from baseline: diagnosis of lung cancer and death from lung cancer. We assessed model discrimination (area under the receiver operating curve, AUC), calibration (calibration curves and expected/observed ratio), overall performance (Brier scores), and net benefit with decision curve analysis. Models predicting lung cancer death (UCL-D) and incidence (UCL-I) using 3 variables—age, smoking duration, and pack-years—achieved or exceeded parity in discrimination, overall performance, and net benefit with comparators currently in use, despite requiring only one-quarter of the predictors. In external validation in the PLCO trial, UCL-D had an AUC of 0.803 (95% CI: 0.783, 0.824) and was well calibrated with an expected/observed (E/O) ratio of 1.05 (95% CI: 0.95, 1.19). UCL-I had an AUC of 0.787 (95% CI: 0.771, 0.802), an E/O ratio of 1.0 (95% CI: 0.92, 1.07). The sensitivity of UCL-D was 85.5% and UCL-I was 83.9%, at 5-year risk thresholds of 0.68% and 1.17%, respectively, 7.9% and 6.2% higher than the USPSTF-2021 criteria at the same specificity. The main limitation of this study is that the models have not been validated outside of UK and US cohorts. Conclusions We present parsimonious ensemble machine learning models to predict the risk of lung cancer in ever-smokers, demonstrating a novel approach that could simplify the implementation of risk-based lung cancer screening in multiple settings.

Author summary Why was this study done? Screening and disease prevention programmes are increasingly bespoke; however, their simultaneous delivery at a population-scale presents considerable challenges.

Lung cancer is the most common cause of cancer death worldwide, with poor survival in the absence of early detection.

Screening for lung cancer among those at high-risk could reduce lung cancer-specific mortality by 20% to 24% among those screened, but the ideal way to determine if someone is high-risk remains uncertain and existing approaches are resource intensive. What did the researchers do and find? We used data from the UK Biobank and US National Lung Screening Trial to develop novel, parsimonious models, to simplify the prediction of lung cancer risk and selection to lung cancer screening programmes.

Using ensemble machine learning and 3 predictors—age, smoking duration, and pack-years—we found our models achieved or exceeded parity in performance with leading comparators despite requiring one-third of the variables.

Our models were externally validated in the US Prostate, Lung, Colorectal, and Ovarian (PLCO) Cancer Screening Trial and benchmarked against models that are either in use or have performed strongly in previous analyses. What do these findings mean? Risk assessment for lung cancer screening can be simplified without reducing performance, potentially improving the uptake and effectiveness of national lung cancer screening programmes, and therefore contributing to reducing deaths from lung cancer.

Future research should focus on the application of this modelling approach to other conditions such as cardiovascular disease, diabetes, and chronic kidney disease to support the implementation at scale of multiple concurrent risk-stratified prevention and early detection programmes for major causes of morbidity and mortality.

This study has key limitations as it is based on past data from the US and UK, so prospective evaluation in different countries and regions should be considered.

Citation: Callender T, Imrie F, Cebere B, Pashayan N, Navani N, van der Schaar M, et al. (2023) Assessing eligibility for lung cancer screening using parsimonious ensemble machine learning models: A development and validation study. PLoS Med 20(10): e1004287. https://doi.org/10.1371/journal.pmed.1004287 Received: March 15, 2023; Accepted: August 29, 2023; Published: October 3, 2023 Copyright: © 2023 Callender et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. Data Availability: To facilitate use of the UCL models, we have developed a website and have made the models themselves available: https://github.com/callta/lung-cancer-models. The underlying code for AutoPrognosis is available from https://github.com/vanderschaarlab/AutoPrognosis. UK Biobank, NLST, and PLCO data were used on license (references 68073, NLST-806 and PLCO-801, respectively). These data are subject to material transfer agreements and cannot be shared directly. However, researchers can apply for these data from the UK Biobank (https://www.ukbiobank.ac.uk/) and the US National Institutes of Health (PLCO: https://cdas.cancer.gov/plco/; NLST: https://cdas.cancer.gov/nlst/). Funding: This work was supported by the Wellcome Trust (222890/Z/21/Z to TC), the National Science Foundation (1722516 to FI and MvdS), the Medical Research Council (MR/T02481X/1 to NN and MR/W025051/1 to SMJ) and Cancer Research UK (EDDCPGM\100002 to SMJ). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. Competing interests: I have read the journal’s policy and the authors of this manuscript have the following competing interests: NN reports honoraria for non-promotional educational talks, conference support or advisory boards from Amgen, Astra Zeneca, Boehringer Ingelheim, Bristol Myers Squibb, Guardant Health, Janssen, Lilly, Merck Sharp & Dohme, Olympus, OncLive, PeerVoice, Pfizer, and Takeda. SMJ has received fees for advisory board membership in the last three years from Astra-Zeneca, Bard1 Lifescience, and Johnson and Johnson. He has received grant income from Owlstone and GRAIL Inc. He has received assistance with travel to an academic meeting from Cheisi. TC and SMJ are founders of, and own stock in, Mortimer Health. Abbreviations: AUC, area under the receiver operating curve; CT, computed tomography; E/O, expected/observed; LCRAT, Lung Cancer Risk Assessment Tool; LCDRAT, Lung Cancer Death Risk Assessment Tool; LDCT, low-dose computed tomography; LLP, Liverpool Lung Project; MICE, multiple imputation by chained equation; NLST, National Lung Screening Trial; PLCO, Prostate, Lung, Colorectal and Ovarian Cancer Screening Trial; USPSTF, US Preventive Services Taskforce

Introduction Screening, early detection, and disease prevention programmes are increasingly bespoke, with risk prediction algorithms determining an individual’s eligibility and management [1–3]. Such personalisation promises to improve the benefit-to-harm profile of such interventions and ultimately health outcomes [4–6]. However, the delivery of these programmes at a population scale requires 2 conditions of risk prediction models: that they generalise well to contexts where there are insufficient data for model development, retraining, or validation; and, that the trade-off between model complexity and implementation feasibility is considered. Screening for lung cancer—the foremost cause of death from cancer worldwide [7]—with low-dose computed tomography (LDCT) has been associated with a 20% to 24% reduction in lung cancer-specific mortality among those at high risk and are screened [8,9]. However, the ideal method to identify those at high risk remains unresolved. The US Preventive Services Taskforce (USPSTF) recommends the use of dichotomous criteria—age 50 to 80, ≥20 pack-years smoked, and <15 quit-years for former smokers—to select screening participants [10]. Nevertheless, identifying individuals for lung cancer screening based on risk prediction models has been shown to have both better benefit-to-harm profiles and cost-effectiveness than using dichotomous risk factor-based criteria alone [11–14], leading to risk-model-based selection criteria in European lung cancer screening pilots [15]. To date, most externally validated prediction models for lung cancer have been developed in United States datasets [12,16–21], reflecting the relatively limited availability of suitable cohorts with long-term follow-up for prediction modelling. This implies that most global healthcare systems that implement risk-based lung cancer screening will use prediction models developed in a US population, often using variables such as ethnicity, whose categorisation varies between countries and individual datasets, and academic qualifications that differ both over time and between jurisdictions. In the United Kingdom, existing models have been shown to underperform in specific groups, such as the more socioeconomically deprived, where underestimation of risk could lead to a screening programme systematically widening health inequalities [22]. Furthermore, the risk models currently in use are a challenge to implement. In the UK, eligibility for lung cancer screening pilots is based on the PLCOm2012 and Liverpool Lung Project risk models, requiring 17 unique variables, few of which are routinely available [23]. Collecting these variables from an individual who is potentially eligible and explaining the results currently averages between 5 and 10 min. To determine the screening eligibility of 1 million people would therefore require between 48 and 95 full-time staff a year. In the UK, there are an estimated 6.8 million ever-smokers aged 55 to 74 who are potentially eligible for lung cancer screening, with another 500,000 turning 55 on average each year [24,25]. As lung cancer screening is just one of several risk-based programmes that are either in development or in use, in their current form, these assessments present a formidable obstacle to the effective implementation of national screening programmes. In this study, we hypothesised that using ensemble machine learning with training data spanning different geographic regions, populations, and average risk levels, we could develop predictive models for lung cancer screening with a minimum number of features that has broad applicability. In so doing, we aimed to combine the simplicity of risk-factor-based criteria with the improved predictive performance of risk models, while maintaining generalisability to new settings.

Discussion We have developed parsimonious models for lung cancer screening that combine the simplicity of existing risk factor-based criteria with the predictive performance of complex risk prediction models. Furthermore, we show in benchmarking comparisons that ensemble machine learning models with 3 predictors—age, smoking duration, and smoking pack-years—have equivalent predictive performance and clinical usefulness to existing models requiring 11 predictors. In this analysis, we used ensemble machine learning to leverage the predictions of several optimised model pipelines. Ensemble modelling is based on the concept that different models make different types of mistake, and their errors begin to cancel each other out, such that combining these statistical models could be expected to improve the performance that any one might achieve [52]. By iteratively trialling and optimising a wide range of modelling approaches before subsequently creating ensembles of these approaches, AutoPrognosis ensures that the strongest performing model for that dataset will be derived and allows reproducibility by transparently showing how models were selected. This avoids the need to develop multiple independent models. In the UK, eligibility for National Health Service screening pilots is based on meeting either a five-year absolute risk of lung cancer of ≥2.5% with the LLP risk score or a six-year absolute risk of ≥1.51% with the PLCOm2012 [23]. The use of 2 risk scores where eligibility differs by more than a percentage point in predicted absolute risk, and where a higher risk is tolerated over a five-year period than a six-year period, highlights the policy challenge in adopting the optimal risk-based approach for a particular setting. This approach requires the collection of 17 different unique predictors, as well as the mapping of US educational levels and US ethnicity categorisations to the UK. With an estimated 7 million current smokers in the UK [25]—even ignoring former smokers—the time and resource requirements to determine screening eligibility at a population scale will be challenging. Using 3 unambiguous variables but with equivalent or improved performance, the UCL models could be completed more easily online or in primary healthcare, simplifying the implementation of lung cancer screening. A potential risk with using only few predictors is that the models will underperform in different subgroups. Across all major subgroups the performance of the UCL models was equivalent to existing models and importantly this included all 4 ethnic groups available in this analysis. The UCL models were also well calibrated in different subgroups, with no sex, smoking status, or age differences. However, there was some undercalibration in 2 groups used as predictors in comparator models: a history of COPD and family history of lung cancer. Furthermore, all models were undercalibrated in at least 1 ethnic group. As discrimination was good for most models, this suggests that lower thresholds could be considered, particularly in black populations, analogous to the relaxation in the age and smoking intensity criteria made between the USPSTF-2013 and USPSTF-2021 screening recommendations [53]. More work is required to improve calibration among different ethnicities, although it is notable that simply including this predictor in models had limited impact. In keeping with Katki and colleagues [19], we found that UCL-D, predicting the risk of death from lung cancer, had greater discrimination than models predicting lung cancer occurrence. In these analyses, there was >96% overlap between UCL-D and UCL-I in terms of those selected for screening, with those selected by UCL-D but not UCL-I showing a trend towards a greater risk of death from lung cancer with longer follow-up (Fig J in S1 Appendix). In microsimulation modelling, overall outcomes differed little between a model predicting death from lung cancer compared with models predicting developing lung cancer [13]. Given this, UCL-D would be the more appropriate model to consider for implementation. This study has strengths. We used large prospective cohorts for model development and validation. Our external validation cohort is both temporally and geographically distinct. We used robust methods for model development and internal validation while externally validating our models extensively using multiple approaches and in a wide range of subgroups. Further, we benchmarked our models against leading comparators. Moreover, by using few, unambiguous, variables, our models could be widely applied after further validation and, where necessary, recalibration. Finally, we have made our models openly available for independent assessment. This study has several limitations. We have used retrospective data, such that findings may differ if used to prospectively determine screening eligibility. However, both the PLCOm2012 and the LLP models have been studied in prospective settings, establishing the benefits of risk-model against risk-factor-based screening. By benchmarking against these models, we can be confident in the performance of our models in a screening programme. To confirm the generalisability of our models, validation in datasets from beyond the US and UK will be the subject of further work. Our analyses have been performed in research cohorts rather than routinely collected electronic health records that may better reflect the broader population. In keeping with the models used as benchmark comparators in this work, the UCL models may not perform to the same extent in routinely collected electronic health records as smoking data are not usually present in the same depth [54]. Nevertheless, screening programmes are unlikely to rely on existing electronic health records given known challenges of missing and inaccurately coded predictors [55]. Finally, our risk models exclude never-smokers. To date, no risk model has been able to discriminate those never smokers with sufficient risk to meet existing criteria for lung cancer screening. In conclusion, we used ensemble machine learning to explicitly maximise model parsimony, an approach that holds promise in multiple disease areas. Our prediction models to determine lung cancer screening eligibility require only 3 variables—age, smoking duration, and pack-years—and perform at or above parity with existing risk models in use. Further validation in alternative datasets as well as prospective implementation should be considered.

Acknowledgments This research has been conducted using the UK Biobank Resource under application number 68073 and we wish to thank all participants in the included studies, as well as the National Cancer Institute for access to NCI’s data collected by the National Lung Screening Trial (NLST) and Prostate, Lung, Colorectal and Ovarian Cancer Screening Trial (PLCO). We also wish to thank Arjun Nair and Sujin Kang for their feedback on earlier versions of this project, as well as Stephen Duffy for his comments on this work. The statements contained herein are solely those of the authors and do not represent or imply concurrence or endorsement by NCI.

[END]
---
[1] Url: https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1004287

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/