(C) PLOS One
This story was originally published by PLOS One and is unaltered.
 . . . . . . . . . .



What makes clinical machine learning fair? A practical ethics framework [1]

['Marine Hoche', 'Department Of Computer Science. Biomedical Informatics Group', 'Eth Zurich', 'Zurich', 'Switzerland', 'Olga Mineeva', 'Gunnar Rätsch', 'Ai Center', 'Effy Vayena', 'Department Of Health Sciences']

Date: 2025-05 

Machine learning (ML) can offer a tremendous contribution to medicine by streamlining decision-making, reducing mistakes, improving clinical accuracy and ensuring better patient outcomes. The prospects of a widespread and rapid integration of machine learning in clinical workflow have attracted considerable attention including due to complex ethical implications–algorithmic bias being among the most frequently discussed ML models. Here we introduce and discuss a practical ethics framework inductively-generated via normative analysis of the practical challenges in developing an actual clinical ML model (see case study). The framework is usable to identify, measure and address bias in clinical machine learning models, thus improving fairness as to both model performance and health outcomes. We detail a proportionate approach to ML bias by defining the demands of fair ML in light of what is ethically justifiable and, at the same time, technically feasible in light of inevitable trade-offs. Our framework enables ethically robust and transparent decision-making both in the design and the context-dependent aspects of ML bias mitigation, thus improving accountability for both developers and clinical users.

Artificial intelligence (AI) can be fruitfully integrated into clinical workflow, improving efficiency and accuracy in clinical decision-making. However, AI systems–such as machine learning algorithms–come with very well know limitations. One of the most discussed limitations of machine learning, in medicine and beyond, is the documented tendency of such systems to exhibit bias in their performance, as well as in their practical outcomes. Unfair medical algorithms can reproduce or exacerbate existing health inequalities by causing AI to perform differently for different patient groups, or resulting in unfair health outcomes. This paper offers a framework to ethically assess and minimize AI bias in clinical practice. Our framework integrates ethical and technical requirements, thus representing a practical contribution to ensure fairness in clinical AI.

Funding: This work was supported by the Swiss National Science Foundation (NRP77–SNSF 407740_187356 to AB and EV), by the Strategic Focus Area “Personalized Health and Related Technologies (PHRT)” of the ETH Domain (project grant #2022-278 to GR), the Max Planck/ETH Center for Learning Systems (to GR), and by ETH Core funding (to GR).

Data Availability: The HIRID dataset (Faltys et al., 2021), is publicly available for download on PhysioNet (Goldberger et al., 2000). The FAMEWS open-source tool is available at: https://github.com/ratschlab/famews

Copyright: © 2025 Hoche et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Table 1. Different mathematical formalization of model fairness. TPR: True Positive Rate; FPR: False Positive Rate; FNR: False Negative Rate; TNR: True Negative Rate; PPV: Positive Predictive Value; NPV: Negative Predictive Value; AUROC: area under the receiver operating characteristic (ROC) curve (as a function of TPR and FPR); AUPRC: area under the precision-recall curve (as a function of PPV and TPR).

Model fairness (often also called group fairness) refers to ensuring that ML models have similar performance for different patient groups. Outcome fairness, instead, means controlling for disparities in health outcomes in different patient groups for which a model is used. Different formalizations of fairness are defined in Table 1 . While interconnected, model and outcome fairness are two separate problems [ 6 ]. Mitigating model performance bias is likely to result in more equitable health outcomes. However, additional factors may have to be considered. Moreover, fair outcomes do not necessarily signal that model performance is indeed also fair, as they may be due to implementation features that compensate for model unfairness.

However, not all differences in model performance have equal ethical weight [ 3 , 4 ] especially when mitigation measures would result in deteriorated performance, thus giving rise to trade-offs. Moreover, some differences among different subgroups may be due to underlying clinical or biological characteristics, and may not be avoidable. For instance, predicting clinical outcomes may be harder for certain classes of patients than others. Those kinds of biases are due to the nature of the underlying phenomenon and while not unfair [ 5 ], they should still be documented for optimized clinical use–for instance by restricting model implementation.

In the context of clinical machine learning (ML), fairness means ensuring that a model performs similarly for different types of patients and that characteristics such as gender, sexual orientation, age, religion, ethnic origin and the like do not skew a model’s output, nor the outcomes of its use. Differences in model performance or in model outcomes across any subgroup of patients should be considered from an ethical point of view [ 2 ]–even if they do not affect socially disadvantaged groups, and even if–contrary to what others have argued (3,4)–they go to the advantage of patients belonging to unprivileged groups.

Fairness is often discussed as a specific requirement of justice in matters of resource distribution or allocation, as well as with respect to the differences in resources or goods between individual and population groups. Justice refers in general to giving people what is due to them, such as goods they are entitled to, or punishments they deserve [ 1 ]. Fairness specifies that certain individual or group characteristics (such as gender, sexual orientation, age, religion, ethnic origin and the like) should not influence how one is treated with respect to what is due to them or what they are entitled to–that is, that people should not have or attain less or more in virtue of those characteristics.

Our training dataset does not contain ground-truth labels for circulatory failure. We thus used a clinician-approved proxy, namely arterial lactate level above 2 mmol/L (irregular sampling upon doctor’s request) and mean arterial pressure below 65 mmHg (sampling rate: 2 minutes). The model outputs a prediction every 5 minutes. If the prediction is above the threshold the model triggers a warning; if the prediction is below the threshold, the model does not provide any feedback.

A ML model provides early warning of imminent (12 hours) circulatory failure in patients admitted to an intensive care unit (ICU) for any clinical reasons [ 14 ]. We obtained anonymized high-resolution times-series data (HiRID) [ 15 ] with monitoring, lab measurements, medications, observations as well as static clinical attributes (admission reason, admission type - emergency or elective - surgical status - i.e., whether the patient had underwent surgery) and demographics (age, gender) from Inselspital in Bern (Switzerland) for model training and validation. Data were collected between January 2008 and June 2016. This dataset is publicly available for research use.

In what follows we illustrate how potential sources of bias can be identified (Section 1), measured and mitigated when bias depends on model features (Sections 2 and 3), as well as when it depends on the conditions of its implementation and use (Section 4). We identify action points that specify the requirements of fairness for each step of our framework, and we show how to use it through a case study based on a ML model to predict circulatory failure in patients in intensive care units (see Box 1 ).

In the paper, we employ the use case merely to illustrate the framework with concrete examples of how bias can be measured across different mathematical formalization and different patient characteristics. Full methodological details about the case study are provided in the boxes.

This paper provides a conceptual framework that can be generally employed in clinical practice irrespective of medical subfield. A conceptual framework does not consist of a detailed mathematical pipeline, nor it includes specific debiasing techniques. Rather, a conceptual framework is used to decide between different possible mathematical formalization of fairness and debiasing techniques. Our framework was built inductively based on a normative analysis of the practical challenges posed by devising and calibrating the model described in the case study (see Boxes). Our normative analysis is based on philosophical accounts of structural injustice [ 7 ] and justice in healthcare [ 8 , 9 ]. We have followed a deontological normative approach [ 10 , 11 ], based on conceptual analysis techniques developed in the field of principle-based bioethics [ 1 , 12 , 13 ]. More specifically, we did not deduce our framework from our use case. Rather, based on theoretical assumptions about structural injustice, we inductively extracted salient normative features from the use case and abstracted them to formulate normative principles to be applied beyond the use case itself. As a result, none of the action-guides we propose is specific to the field of the use case (i.e., intensive care medicine).

Results

1. Identifying sources of bias Identifying possible sources of bias in training datasets is a first step for the construction of a fair clinical ML model [16,17]. => [A1] Developers should identify potential sources of bias to define for which classes of patients the model may under-perform. Most ethical discussions about ML bias focus on insufficient socio-demographic representativeness in training data as the principal cause of ML bias. This problem affects in particular minorities and socially disadvantaged groups that, typically, are underrepresented in available datasets (minority bias, Table 2). Training ML models on more balanced datasets is more likely to yield models that perform evenly across different socio-demographic groups. More inclusive training datasets, however, tend to also include protected attributes (such as age, gender, sex and sexual orientation, ethnicity, race, religion, etc.). Some argue that protected characteristics should not be taken into account in the training process, so as to avoid models that unfairly associate those features to their output [16]. However, obfuscating protected attributes such as age or gender in medical ML may deteriorate model performance in ways that are also ethically problematic. And, while it is possible that more inclusive datasets could make ML models fairer, sometimes available datasets are simply not equally representative of all demographics. In such cases, under-represented groups should be identified, so as to enable further testing of model performance and outcomes across different patient cohorts. PPT PowerPoint slide

PNG larger image

TIFF original image Download: Table 2. Potential sources of bias affecting model fairness (section 3) and outcome fairness (section 4). https://doi.org/10.1371/journal.pdig.0000728.t002 It is also possible that training and validation datasets do not contain sufficient socio-demographic information, to enable developers to control for minority bias. In such cases the effects of minority bias should be monitored in terms of health outcomes when the model is in use (section 4). Beyond minority bias, other well-documented sources of data-related bias include protected attribute bias, missing data bias and label bias [18] (Table 2 for definitions). Developers should carefully consider whether such forms of bias may affect their training datasets so as to determine for which patient classes their model may underperform (Box 2). The identification of a particular source of bias in training and validation data is not a sufficient reason to foreclose the use of such data for model development. The fundamentally opaque nature of ML models makes it particularly hard to predict, by simply looking at training data composition, if and how ML bias will manifest in a model. Moreover, some variation in model performance may reflect natural characteristics of the observed phenomenon rather than under- or over-sampling. Identifying possible sources of bias in training and validation data serves the purpose of defining which classes of patients or which patient attributes should be used to measure biased model performance (section 2). It should be noted that specific patterns of disadvantage often result from the intersection of different characteristics. According to intersectional accounts of justice certain individuals are particularly vulnerable to patterns of systematic disadvantage due to the multiple, overlapping forms of inequality and discrimination that they face. Therefore, it is important to identify (and tests) not only individual attributes (e.g., age, gender, ethnicity), but also how their combination further amplifies disadvantage. Once potential sources of bias have been identified, the second step to minimize unfairness is to measure how bias manifests itself in terms of different performance for different classes of patients (section 3). Potential sources of bias linked to model use such as, training-serving skew, automation bias and dismissal bias (Table 2) can instead best be addressed upon model implementation [18] (section 4). Box 2. Bias identification Variables such as ethnicity and socio-economic status were not reported in our training dataset and could thus not be controlled for. This model may thus suffer from minority bias, which needs to be taken into account upon implementation. The sex distribution in our data is 65% male and 35% female patients. We thus need to control for protected feature bias between male and female patients The age of patients in our data is quite diverse. We therefore have to control for protected feature bias among different age groups. Since we use a clinical proxy, we checked for potential label bias in our proxy definition and found out that for neurological patients–who have increased arterial pressure due to medications used to keep their circulatory pressure higher for clinical reasons–the model may produce inaccurate outcomes. We thus need to control for label bias across different clinical indications (reasons for admission).

2. Model fairness: Measuring model fairness To measure if a model is actually biased against any patient class, it is necessary to adopt a metric to compare model performance across relevant patient classes. Classes such as gender or ethnicity should routinely be tested whenever possible as they are known to account for substantial bias in medicine. Available formalizations of algorithmic fairness include: equalized odds, equal opportunity, precision-recall parity, predictive equality, predictive parity, equal conditional use accuracy, and equal selectivity (Table 1 for definitions) [19]. From a mathematical point of view, it is impossible to satisfy all the above-mentioned formalizations at the same time [20–23]. As a consequence, some degree of disparity in the way a model performs for different groups under different formalizations of fairness is unavoidable. Therefore, => [A2.1] Developers should define which mathematical formalization of fairness is best suited to the clinical task at hand. This entails that some ethical trade-off will most likely have to be accepted. General criteria to determine the best possible fairness metric are not available. Clinical and ethical considerations can guide the choice of the most appropriate measure of model performance in specific cases [24]. For instance, for a model that detects a serious condition requiring immediate intervention, a heightened fall-out (probability of false positive or false alarm) measured as predictive equality is more acceptable than a higher miss-out (probability of false negative). Furthermore, one needs to consider whether the condition the model is being used for is more prevalent in certain patient classes over others. If so, it is preferable to choose formalizations of fairness that are not affected by the difference in prevalence (e.g., equal opportunity) or to normalize the chosen metric for prevalence. =>[A2.2] Developers should comparatively assess model performance across identified patients classes based on the fairness metric(s) they have selected. For each defined patient class, it is possible to compute the metric values for class members versus the rest of the cohorts. Performance disparities can be quantified by Where p is a patient or data point, is the n -th bootstrapped sample, N is the total number of bootstrapped samples, G is the studied patient class, is the rest of the population. This formula quantifies disparities by comparing median metric values between one class and the rest of the population; an alternative is comparing to the entire population. While we focus on performance deltas that are directly interpretable and allow calculation of the number of patients affected, we note that standardized effect sizes being unitless, it enables comparisons across different metrics and studies. Therefore, we consider both approaches complementary and suggest that standardized effect sizes can be reported alongside performance deltas for a more comprehensive understanding. Note that multiple metrics can be measured to observe how the model behaves in general, to check if bias manifests for any given group across different formalizations of fairness. Such a comparative approach may help in the selection of the most effective debiasing technique (see below, Section 3). To support the practical implementation of our framework, tools such as FAMEWS [25] can facilitate quantitative analyses of fairness-performance trade-offs in clinical machine learning models.

4. Outcome fairness: Measuring outcome fairness During model implementation, specific sources of bias can lead to uneven patient outcomes. When a model is deployed in any given clinical setting, actual patients may have specific socio-demographic characteristics that do not necessarily match training and validation data (training-serving skew, Table 2), leading to worse health outcomes for patient groups that were not represented in the training data. This type of assessment is not always possible, depending on available information and metadata. In the absence of socio-demographic information in training and validation datasets, ad hoc monitoring should be in place at the site where the model is in use to compare health outcomes across different socio-demographic patient groups. ML training and validation datasets are representative if estimates obtained on that sample are generalizable to the target population [33], therefore, prior to model implementation => [A4.1] Discrepancies between the clinical and socio-demographic composition of the actual patient population and the composition of the training and validation data should be documented. In other words, developers should check for the possibility that training-serving skew (see Table 2) may lead to worse health outcomes for patient groups that were not represented in the training data. If relevant discrepancies in clinical features or protected attributes are detected => [A4.2] Regular monitoring should be set up to promptly detect signs of inferior outcomes for patient groups not sufficiently represented in the original data. Outcome-related disparities can depend on the model itself or on its users. Variations in health outcomes due to model predictions and assessments are model-dependent. While model dependent biases such as the time difference between alarm and event can be measured during model development (e.g., on a separate validation dataset), how such characteristic affects patient outcomes (model-dependent biases) can be measured ex post. On the other hand, outcome fairness depends also on how a model works and is operated in real-world clinical conditions, for instance how users such as healthcare professionals (HCPs) respond to its output, or translate model output into clinical decisions. Differential outcomes resulting from operational conditions are user-dependent and can as well be detected once the model is implemented. It is possible, for instance, that a healthcare practitioner relies too much on the input of a ML model without questioning its predictions in light of her expertise and experience; or that one tends to–consciously or not–ignore input from a ML tool, for example by dismissing alerts that pop up too frequently in what is known as alarm fatigue [34]. Moreover, end-users have their own biases too. It is thus possible that automation and dismissal bias is more prevalent in the case of certain patient groups than others, with obvious consequences in terms of uneven health outcomes. Only the nature of the clinical task at hand can suggest the risk of biased user-dependent outcomes (see Box 7). Therefore => [A4.3] Developers should document the anticipated risk of user-dependent outcome biases. and => [A4.4] Regular monitoring and statistical trend analysis should be in place to measure differences in how the model is operated for different clinical and socio-demographic patient groups. Data should be collected on how practitioners use a model - meaning how often they follow their advice and what courses of action result from its use - and compute use-patterns with patient characteristics and health outcomes, so as to monitor whether and to which extent the model produces fair health outcomes for all patient types. There is ample room to automate such monitoring to spot potential sources of outcome bias during clinical use across different socio-demographic and clinical classes, so as to enable prompt corrective actions in case uneven outcomes or deteriorating trends are detected. Box 7. Outcome fairness Given the absence of socio-demographic specifications in our original data, we could not control for minority bias. Moreover differences between training and validation data and the actual clinical population at the site of model implementation cannot be documented prior to model implementation. Ad hoc monitoring should be in place to compare health outcomes across different socio-demographic classes while the model is in use. We reasoned that an excessive false positive rate (FPR) can create inefficiencies in model implementation linked to dismissal bias (alarm fatigue) on the part of healthcare professionals. FPR can be checked during model development but proper adjustments also depend on the implementation context and can thus only be made ex post. This risk has to be reported and documented upon model implementation so that adequate monitoring and personnel training can be set up. Moreover, our model’s performance does not depend exclusively on a correct binary classification (whether or not the patient will experience circulatory failure), but also on the timing of the warning relative to the actual manifestation of circulatory failure. We can therefore check whether the model triggers alarms sufficiently in advance for different cohorts of patients by comparing time gaps between the first accurate alarm and circulatory failure. Such measurements should be reported and documented upon model implementation to allow clinical users to adapt their monitoring and intervention protocols. The risk that automation bias affects health outcomes for our model is low because a wrong prediction like a false alarm–while undesirable in terms of clinical workflow–is unlikely to endanger patients’ health.

[END]
---
[1] Url: https://journals.plos.org/digitalhealth/article?id=10.1371/journal.pdig.0000728

Published and (C) by PLOS One
Content appears here under this condition or license: Creative Commons - Attribution BY 4.0.

via Magical.Fish Gopher News Feeds:
gopher://magical.fish/1/feeds/news/plosone/