[HN Gopher] We investigated Amsterdam's attempt to build a 'fair...
___________________________________________________________________
We investigated Amsterdam's attempt to build a 'fair' fraud
detection model
Author : troelsSteegin
Score : 41 points
Date : 2025-06-12 13:54 UTC (2 days ago)
(HTM) web link (www.lighthousereports.com)
(TXT) w3m dump (www.lighthousereports.com)
| djoldman wrote:
| "Unbiased," and "fair" models are generally somewhat ironic.
|
| It's generally straightforward to develop one if we don't care
| much about the performance metric:
|
| If we want the output to match a population distribution, we just
| force it by taking the top predicted for each class and then
| filling up the class buckets.
|
| For example, if we have 75% squares and 25% circles, but circles
| are predicted at a 10-1 rate, who cares, just take the top 3
| squares predicted and the top 1 circle predicted until we fill
| the quota.
| Scarblac wrote:
| But that's a bias, if circles are actually more likely to be
| fraudulant.
| djoldman wrote:
| If the definition of "unbiased" and "fair" is that the model
| flags squares and circles at a rate or proportion equal to
| the population distribution of squares and circles, then the
| model is unbiased and fair.
|
| As noted above, this doesn't do anything for performance.
| wongarsu wrote:
| So if I want to make a model to recommend inkjet printers then
| a quarter of all recommendations should be for HP printers?
| After all, a quarter of all sold printers are HP.
|
| As you say, that would be a crappy model. But in my opinion
| that would also be hardly a fair or unbiased model. That would
| be a model unfairly biased in favor of HP, who barely sell
| anything worth recommending
| djoldman wrote:
| Yes, well there's the irony.
|
| "Unbiased" and "fair" are quite overloaded here, to borrow a
| programming term.
|
| I think it's one of those times where single words should
| expressly NOT be used to describe the intent.
|
| The intent of this is to presume that the rate of the thing
| we are trying to detect is constant across subgroups. The
| definition of a "good" model therefore is one that
| approximates this.
|
| I'm curious if their data matches that assumption. Do
| subgroups submit bad applications at the same rate?
|
| It may be that they don't have the data and therefore can't
| answer that.
| teekert wrote:
| I know a cop, they do public searchings for weapons or
| drugs. Our law dictates fairness. So every now and then
| they search an elderly couple. You know how this goes and
| what the results are.
|
| Any model would be unfair, age-wise but also ethnically.
|
| To be most effective the model would have to be unfair. It
| would suck to be a law abiding young specific ethnic
| minority.
|
| But does it help to search elderly couples?
|
| I'm Genuinely curious what would be fair and effective
| here. You can't be a Bayesian.
| djohnston wrote:
| > Equal performance across groups requires the share of people
| who are flagged in each group is the same.
|
| Sorry, this is retarded right? Why would you assume that all
| groupings of people commit welfare fraud at the same rate?
| BonoboIO wrote:
| Yes it is. This is some ideal world thinking, that has nothing
| to do with reality and is easily falsifiable, but only if you
| want to see the real world.
| throwawayqqq11 wrote:
| Holy crap! Because it would be a predetermined bias and thus
| pretty much unfair. I hate to be explicit about basics on HN.
| djohnston wrote:
| No... the pre-determined bias in this story is obviously that
| all subgroups of people behave identically w.r.t. welfare
| applications, which the data itself did not support and a
| momentary consideration of socioeconomics would debunk. When
| they tried to cludge the weights to fit their predetermined
| bias, the model did so poorly on a pilot run that the city
| shut it down.
| throwawayqqq11 wrote:
| Being flagged as potential fraud based on eg. ethnicity is
| what you want to eliminate, so you have to start with the
| assumption of an even distristribution.
|
| From the article:
|
| > Deciding which definition of fairness to optimize for is
| a question of values and context.
|
| This optimization is the human feedback required to not
| have the model stagnate in a local optimum.
| andy99 wrote:
| The better definition of equal performance would obviously be
| that the metrics for the detector - accuracy or false positive
| rate etc would be the same for all groups.
|
| I won't comment on why it's defined the way that it is.
|
| Edit: it looks like they define several metrics, including ones
| like I mention above that consider performance and at least one
| based on what number or percentage is flagged in each group.
| parpfish wrote:
| Or that the error distributions are equal across groups. That
| way you could still detect that one group is committing fraud
| at a higher rate, but false positives/negatives occur at the
| same rate in each group
| ordu wrote:
| The goal is to avoid penalizing people for their skin color, or
| for gender/sex/ethnicity/whatever. If some group have higher
| rate of welfare fraud, the fair/unbiased system must keep false
| positives for that group at the same level as for general
| population. Ideally there should be no false positives at all,
| because they are costly for people, who were marked wrongly,
| but sadly real systems are not like that. So these false
| positives have to be spread over all groups proportionally to
| sizes of the groups.
|
| Though the situation is more complex than that. What I
| described is named "False Positive Share" in the article (or at
| least I think so), but the article discusses other metrics too.
|
| The problem is that the policy should make the world better,
| but if the policy penalizes some groups for law breaking, then
| it can push these groups to break the law even more. It is
| possible to create biases this way, and it is possible to do it
| accidentally. Or, rather, it is hard not to do it accidentally.
|
| I'd recommend to read "Against Prediction", it has a lot of
| examples how it works. For example, biased False Negatives are
| also bad, they make it easier for some groups to break the law.
| tbrownaw wrote:
| There are multiple different ways to measure performance. If
| different groups have different rates of whatever you're
| predicting, it is not possible to have all of the different
| ways of measuring performance agree on whether your model is
| fair or not.
| BonoboIO wrote:
| The article talks a lot about fairness metrics but never mentions
| whether the system actually catches fraud.
|
| Without figures for true positives, recall, or financial
| recoveries, its effectiveness remains completely in the dark.
|
| In short: great for moral grandstanding in the comments section,
| but zero evidence that taxpayer money or investigative time was
| ever saved.
| tomp wrote:
| Key point:
|
| _The model is considered fair if its performance is equal across
| these groups._
|
| One can immediately see why this is problematic, easily by
| considering equivalent example in less controversial (i.e.
| emotionally charged) situations.
|
| Should basketball performance be equal across racial, or sex
| groups? How about marathon performance?
|
| It's not unusual that relevant features are correlated with
| protected features. In the specific example above, being an
| immigrant is likely correlated with not knowing the local
| language, therefore being underemployed and hence more likely to
| apply for benefits.
| atherton33 wrote:
| I think they're saying something more subtle.
|
| In your basketball analogy, it's more like they have a model
| that predicts basketball performance, and they're saying that
| model should predict performance equally well across groups,
| not that the groups should themselves perform equally well.
| tomp wrote:
| You're right, I misinterpreted it.
| wongarsu wrote:
| A big part of the difficulty of such an attempt is that we don't
| know the ground truth. A model is fair or unbiased if its
| performance is equally good for all groups. Meaning e.g. if 90%
| of cases of Arabs committing fraud are flagged as fraud, then 90%
| of cases of Danish people committing fraud should be flagged as
| fraud. The paper agrees on this.
|
| The issue is that we don't know how many Danish commit fraud, and
| we don't know how many Arabs commit fraud, because we don't trust
| the old process to be unbiased. So how are we supposed to judge
| if the new model is unbiased? This seems fundamentally impossible
| without improving our ground truth in some way.
|
| The project presented here instead tries to do some mental
| gymnastics to define a version of "fair" that doesn't require
| that better ground truth. They were able to evaluate their
| results on the false-positive rate by investigating the flagged
| cases, but they were completely in the dark about the false-
| negative rate.
|
| In the end, the new model was just as biased, but in the other
| direction, and performance was simply worse:
|
| > In addition to the reappearance of biases, the model's
| performance in the pilot also deteriorated. Crucially, the model
| was meant to lead to fewer investigations and more rejections.
| What happened instead was mostly an increase in investigations ,
| while the likelihood to find investigation worthy applications
| barely changed in comparison to the analogue process. In late
| November 2023, the city announced that it would shelve the pilot.
| zeroCalories wrote:
| Does anyone know what they mean by reweighing demographics? Are
| they penalizing incorrect classifications more heavily for those
| demographics, or making sure that each demographic is equally
| represented, or something else? Putting aside the model's
| degraded performance, I think it's fair to try and make sure the
| model is performing well for all demographics.
| 3abiton wrote:
| > A more concerning limitation is that when the city re-ran parts
| of its analysis, it did not fully replicate its own data and
| results. For example, the city was unable to replicate its train
| and test split. Furthermore, the data related to the model after
| reweighting is not identical to what the city published in its
| bias report and although the results are substantively the same,
| the differences cannot be explained by mere rounding errors.
|
| Very well written, but that last part id concerning and point to
| one part: did they hire interns? How cone they do not have
| systems? It just cast a big doubt on the whole experiment.
| tbrownaw wrote:
| > _But the model designers were aware that features could be
| correlated with demographic groups in a way that would make them
| proxies._
|
| There's a huge problem with people trying to use umbrella usage
| to predict flooding. Some people are trying to develop a computer
| model that uses rainfall instead, but watchdog groups have raised
| concerns that rainfall may be used as a proxy for umbrella usage.
|
| (It seems rather strange to expect a statistical model trained
| for accuracy to infer and indirect through a shadow variable that
| makes it _less_ accurate, simply because it 's something easy for
| humans to observe directly and then use as a lossy shortcut or to
| promote alternate goals that aren't part of the labels being
| trained for or whatever.)
|
| > _These are two sets of unavoidable tradeoffs: focusing on one
| fairness definition can lead to worse outcomes on others.
| Similarly, focusing on one group can lead to worse performance
| for other groups. In evaluating its model, the city made a choice
| to focus on false positives and on reducing ethnicity
| /nationality based disparities. Precisely because the reweighting
| procedure made some gains in this direction, the model did worse
| on other dimensions._
|
| Nice to see an investigation that's serious enough to acknowledge
| this.
| talkingtab wrote:
| Is this crazy or what? My take away is that the factors the city
| of Amsterdam is using to predict fraud are probably not actually
| predictors. For example if you use the last digit of someones
| phone number as a fraud predictor, you might discover there is a
| bias against low numbers. So you adjust your model to make it
| less likely that low numbers generate investigations. It is
| unlikely that your model will be any more fair after your
| adjustment.
|
| One has to wonder if the study is more valid a predictor of the
| implementers' biases than that of the subjects.
| thatguymike wrote:
| Congrats Amsterdam: they funded a worthy and feasible project;
| put appropriate ethical guardrails in place; iterated
| scientifically; then didn't deploy when they couldn't achieve a
| result that satisfied their guardrails. We need more of this in
| the world.
| tbrownaw wrote:
| What were the error rates for the various groups with the old
| process? Was the new process that included the model actually
| _worse_ for any group, or was it just uneven in how much better
| it was?
___________________________________________________________________
(page generated 2025-06-14 23:00 UTC)