[HN Gopher] Explainability is not a game
___________________________________________________________________
Explainability is not a game
Author : mitchbob
Score : 14 points
Date : 2024-07-02 21:18 UTC (1 hours ago)
(HTM) web link (cacm.acm.org)
(TXT) w3m dump (cacm.acm.org)
| brigadier132 wrote:
| ML models are just statistical function approximators right? So
| isn't asking for "explainability" from ML models equivalent to
| asking "why is fire hot?"? If an ML model makes a prediction the
| reason why is because it was trained on something that made the
| prediction more likely than others.
| lgas wrote:
| Sure, but there are different variables in the training data
| that may influence each inference more or less. So if you can
| observe which variables influenced the inference and how much
| then you can (at least attempt to) construct an explanation.
| mvanveen wrote:
| In this case let's assume that the weights and biases of,
| say, a neural network are fixed and the model is already
| trained.
|
| One way of thinking about explainability is that it deals
| with determining for some input data how much each feature is
| contributing to the final outcome (e.g. variable 1 and 2
| contributed x% and y% to the final inferred value).
|
| You're correct to suggest that when you backpropogate
| residual error there are also non-linear interactions between
| features and that will affect how much each variable is
| contributing to the updated weight value (in fact that's kind
| of the point of a deep network ;).
| taeric wrote:
| At very abstract points, I think you have a point; but that is
| largely a failure to scale for any system. Consider holding a
| rock is not dangerous. Holding a coal is. Why do you think
| that?
|
| Take that further and consider why people are hesitant to touch
| cooking surfaces? The answer isn't "because fire is hot." The
| answer is "because the purpose of cooking surfaces is to get
| hot enough to cook things and without knowing how long it has
| been unused, it is not safe to touch."
| Vecr wrote:
| That's a big problem though, mechanistic
| interpretability/explainability is really important for the
| whole "not wrecking everything" part of AI. You could also just
| hope AI stalls out or goes back to more hand-tuned Bayesian
| stuff, but hope is not a plan. Progress past standard analysis
| has been made, so I'm not sure how important this paper is.
| tomxor wrote:
| > ML models is most often inscrutable, with the consequence that
| decisions taken by ML models cannot be fathomed by human decision
| makers
|
| Yup, James Mickens planted this seed 6 years ago, well worth the
| watch, even if you already agree it's brilliant cathartic humour
| (the subject isn't really "security").
|
| https://www.youtube.com/watch?v=ajGX7odA87k
| mvanveen wrote:
| I am a co-author of a patent for model explainability for credit
| risk underwriting applications using Shapley values.
|
| In fairness I haven't given this article a thorough read but my
| initial impression is that I'm finding myself frustrated by the
| FUD this article is attempting to spread. As my boss would often
| remark to remind us all: _model explainability is an under-
| constrained optimization problem._ By definition there isn 't a
| unique explanation decomposition unless you further constrain the
| problem.
|
| Therefore, I personally find that hand-wringing around there not
| being 100% agreement around different explanations for a model
| inference, while definitely thought provoking and worth
| considering, should at least account for this reality. For some
| reason a lot of folks in the ML community seem to have come the
| opinion that because the problem is under-constrained that means
| that explanations shouldn't be calculated or have no utility.
|
| Would you prefer a model that examines which features are driving
| the model to deny a disproportionate number of folks of a
| particular race or ethnicity or not, all things being equal? My
| point is even if there are limitations to explainability I think
| there are a lot of very real, critical scenarios where applying
| SHAP can be of actual, real world utility.
|
| Furthermore, it's not clear that LIME or other explainability
| methods will provide better or more robust explanations than
| Shapley values. As someone that has looked at this pretty
| extensively in credit underwriting I'd personally feel most
| comfortable computing SHAP values while acknowledging some of the
| limitations and risks this article calls out.
|
| Axioms such as completeness are also pretty reasonable and I
| think there is a fair amount of real world utility to
| explainability algorithms that derive from such an axiomatic
| basis.
___________________________________________________________________
(page generated 2024-07-02 23:00 UTC)