[HN Gopher] Explaining Large Language Models Decisions Using Sha...
___________________________________________________________________
Explaining Large Language Models Decisions Using Shapley Values
Author : veryluckyxyz
Score : 80 points
Date : 2024-12-28 00:44 UTC (22 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| goldemerald wrote:
| While I love XAI and am always happy to see more work in this
| area, I wonder if other people use the same heuristics as me when
| judging a random arxiv link. This paper has one author, was not
| written in latex, and no comment referencing a peer reviewed
| venue. Do other people in this field look at these same signals
| and pre-judge the paper negatively?
|
| I did attempt to check my bias and skim the paper, it does seem
| well written and takes a decent shot towards understanding LLMs.
| However, I am not a fan of black-box explanations, so I didn't
| read much (I really like Sparse autoencoders). Has anyone else
| read the paper? How is the quality?
| refulgentis wrote:
| > I wonder if other people use the same heuristics as me when
| judging a random arxiv link.
|
| My prior after the header was the same as yours. The fight and
| interesting part is in the work past the initial reaction.
|
| i.e. if I react with my first order, least effort, reaction,
| your comment leaves the reader with a brief, shocked, laugh at
| you seemingly doing performance art. A seemingly bland
| assessment and overly broad question...only to conclude with
| "Has anyone else read the paper? Do you like it?"
|
| But that's not what you meant. You're geniunely curious if its
| a long tail, inappropriate, reaction to have that initial
| assessment based on pattern matching. And you didn't mean "did
| anyone else read it", you meant "Humbly, I'm admitting I'm
| skimmed, but I wasn't blown away for reasons X, Y, and Z. What
| do you all think? :)"
|
| The paper is superb and one of the best I recall reading in
| recent memory.
|
| It's a much whiter box than Spare Autoencoders. Handwaving what
| a bag of floats _might_ do in _general_ is much less
| interesting or helpful than being able to statistically
| quantify the behavior of the systems we 're building.
|
| The author is a PhD candidate at the Carnegie Mellon School of
| Business, and I was quite taken with their ability to hop
| across fields to get a rather simple and important way to
| systematically and statistically review the systems we're
| building.
| apstroll wrote:
| This paper is doing exactly that though, handwaving with a
| couple of floats. The paper is just a collection of
| observations about what their implementation of shapley value
| analysis gives for a few variations of a prompt.
| cauliflower2718 wrote:
| It looks like it's written in latex to me. Standard formatting
| varies across departments, and the author is in the business
| school at CMU.
|
| In some fields, single author papers are more common. Also,
| outside of ML conference culture, the journal publication
| process can be pretty slow.
|
| Based on the above (which is separate from an actual evaluation
| of the paper), there are no immediate red flags.
|
| Source: I am a PhD student and read papers across stats/CS/OR.
| woolion wrote:
| The Latex feel comes in good part from the respect for
| typographical standards that is encoded as default behaviour.
| In this document, so many spacings are just flat-out wrong,
| first paragraph indents, etc. If it's indeed Latex (it kinda
| looks like it), someone worked hard to make it look bad.
|
| The weirdest thing is that copy-paste doesn't work; if I copy
| the "3.1" of the corresponding equation, I get " . "
| ersiees wrote:
| Another clue: there is no way to download the latex, while
| you can if someone uploaded the latex on arxiv.
| cauliflower2718 wrote:
| There's a lazy way to submit to arxiv, which is to submit
| just the PDF, even if you did it in latex. Sometimes it can
| be annoying to organize the tex files to submit to arxiv.
| It's uncommon, but the font and math rendering are the
| standard latex font.
| mnky9800n wrote:
| I think that we should not accept peer review as some kind of
| gold standard anymore for several reasons. These are my
| opinions based on my experience as a scientist for the last 11
| years.
|
| - its unpaid work and often you are asked to do it too much and
| therefore may not give your best effort
|
| - editors want to have high profile papers and minimise review
| times so glossy journals like nature or science often reject
| things that require effort on the review
|
| - the peers doing a review are often anything but. I have seen
| self professed machine learning "experts" not know the
| difference between regression and classification yet proudly
| sign their names to their review. I've seen reviewers ask you
| to write prompts that are mean and cruel to an LLM to see if it
| would classify test data the same (text data from geologists
| writing about rocks). As an editor I have had to explain to
| adult tenured professor that she cannot write in her review
| that the authors were "stupid" and "should never be allowed to
| publish again".
| 3abiton wrote:
| Scientific peer review is another facit of civilization that
| its current design does not allow it to scale well. More and
| more people are being involved in the process, but the
| qualityis forever going down.
| mnky9800n wrote:
| Yes that's right. It's a scaling problem and there isn't a
| clear answer. It's easy to complain about it though haha. I
| think what is happening is science is atomitizing. People
| are publishing smaller amounts or simply creating ideas
| from nothing (like that science advances paper on hacker
| news a couple days ago that created a hydrogen rich crust
| from thin air).
| chongli wrote:
| A further issue is peer review quid pro quo corruption. The
| reviewer loves your paper but requests one small change: cite
| some of his papers and he'll approve your paper.
|
| I don't know how prevalent this sort of corruption is (I
| haven't read any statistical investigations) but I have heard
| of researchers complaining about it. In all likelihood it's
| extremely prevalent in less reputable journals but for all we
| know it could be happening at the big ones.
|
| The whole issue of citations functioning like a currency
| recalls Goodhart's Law [1]:
|
| _"When a measure becomes a target, it ceases to be a good
| measure."_
|
| [1] https://en.wikipedia.org/wiki/Goodhart's_law
| mnky9800n wrote:
| Tbh I used to have an issue with that but these days it
| really is a small issue in the grand scheme of things. You
| can say No but also, there are larger systemic problems in
| science.
| chongli wrote:
| You're right. It's more of a symptom of the systemic
| problems than the main problem itself. But it still
| contributes to my distrust in science.
| chongli wrote:
| I didn't even read the paper, I just read the abstract. I was
| really impressed by the idea of using Shapley values to
| investigate how each token in a prompt affects the output,
| including order-based effects.
|
| Even if the paper itself is rubbish I think this approach to
| studying LLMs at least warrants a second look by another team
| of researchers.
| johndough wrote:
| Two more heuristics:
|
| 1. The figures are not vectorized (text in figures can not be
| selected). All it takes is to replace "png" in
| `plt.savefig("figure.png")` with "pdf", so this is a very easy
| fix. Yet the author did not bother, which shows that he either
| did not care or did not know.
|
| 2. The equations lack punctuation.
|
| Of course you can still write insightful papers with low
| quality figures and unusual punctuation. This is just a
| heuristic after all.
| xianshou wrote:
| This doesn't replicate using gpt-4o-mini, which always picks
| Flight B even when Flight A is made somewhat more attractive.
|
| Source: just ran it on 0-20 newlines with 100 trials apiece,
| raising temperature and introducing different random seeds to
| prevent any prompt caching.
| yorwba wrote:
| The newline thing is the motivating example in the
| introduction, using Llama 3 8B Instruct with up to 200 newlines
| before the question. If you want to reproduce this example with
| another model, you might have to increase the number of
| newlines all the way to the context limit. (If you ask the API
| to give you logprobs, at least you won't have to run mutiple
| trials to get the exact probability.)
|
| But the meat of the paper is the Shapley value estimation
| algorithm in appendix A4. And in A5 you can see that different
| models giving different results is to be expected.
| scottiescottie wrote:
| explainable AI just ain't there yet.
|
| I wonder if the author took a class with Lipton, since he's at
| CMU. We literally had a lecture about Shapley Values "explaining"
| AI. It's BS.
___________________________________________________________________
(page generated 2024-12-28 23:01 UTC)