[HN Gopher] To Believe or Not Believe Your LLM
___________________________________________________________________
To Believe or Not Believe Your LLM
Author : josh-sematic
Score : 50 points
Date : 2024-06-05 13:04 UTC (9 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| cs702 wrote:
| I've only briefly skimmed the paper, so I can't comment on its
| information-theoretic aspects yet, but I found the example given
| on Section 3 very helpful and insightful:
|
| The key insight is that if you give a pretrained LLM a question
| prompt such as "What is the capital of the UK?" and then
| concatenate to the prompt a string with a different answer, such
| as "Another possible answer is Paris," again and again, the model
| will continue to output its original answer (say, "London") _only
| when it has low "epistemic uncertainty,"_ i.e., when the model's
| parameters and state encode sufficient knowledge to answer the
| question the same way, again and again, despite the repeated
| addition of wrong answers to the prompt.
|
| If the model quickly starts changing its original answer, by
| implication, its answer has high epistemic uncertainty, i.e., the
| model's parameters and state do not encode sufficient knowledge
| to answer the question the same way, again and again, as we add
| more wrong answers to the prompt. In other words, if we can
| quickly make the model change its answer by modifying the prompt,
| the model has high epistemic uncertainty.
|
| Figure 1 shows what happens when the authors add a wrong answer
| up to 100 times to a question prompt for which the model answers
| correctly with low epistemic uncertainty. Figure 3 shows that
| happens when the authors add a wrong answer up to 8 times to a
| question prompt for which the model exhibits high epistemic
| uncertainty.
|
| This strikes me as a simple, intuitive way of detecting epistemic
| uncertainty.
|
| Thank you for sharing this paper on HN. I've added it to my
| reading list.
| voiceblue wrote:
| For NLP tasks it's unlikely that "epistemic uncertainty" is a
| useful metric, though it's interesting to ask an LLM "what is
| the third word in this sentence?" and then suggest alternative
| answers. It's a good way to demonstrate that an LLM is not
| really a "thinking" machine in the way laymen might assume.
|
| Example:
|
| https://chatgpt.com/share/e8743fe2-a604-4cf0-8c10-aa863a67a5...
| mdp2021 wrote:
| (Of the two links, I get a 404 and a blank...)
| voiceblue wrote:
| Thanks, I expected that deleting the conversation on my end
| would have no effect on a permalink but I was wrong.
|
| Nothing I can do about the blank, however, it works for me.
| Manabu-eo wrote:
| There are at least two reasons for transformers poor
| performance on that prompt:
|
| - Transformers view the word as tokens, not words or
| characters. - The positional encoding might be holding them
| back. See this recent paper discussed here: Transformers Can
| Do Arithmetic with the Right Embeddings [1]
|
| [1] https://news.ycombinator.com/item?id=40497379
| maCDzP wrote:
| I haven't read the paper yet but I want to share my immediate
| thought.
|
| So the harder it is to "convince" the LLM on a wrong answer,
| that's a proxy for low epistemic uncertainty?
|
| I know you shouldn't view the LLM as a "mind" - but I can't
| help myself!
| 3abiton wrote:
| Would you be so kind and share your thoughts about it once you
| give it a read?
| empath75 wrote:
| "Arguing" with chat gpt in general will give you a good idea of
| how certain it is with it's answers. It often has a very
| incomplete or wrong knowledge of various apis and command line
| arguments. Asking if it's sure or telling it that it's wrong or
| pasting it an error message will get it to correct it's answer,
| but if you tell it that "Paris is definitely the capital of the
| UK" will not convince it to agree with you.
|
| Given the probable fact that information about certainty about
| facts is contained within the model _somewhere_, it'd be nice if
| it could better incorporate that into the answer. Like -- "I
| don't know the exact api, but I think the correct yaml should be
| something like this" would be much more useful than just spitting
| out the wrong yaml with complete confidence.
| Manabu-eo wrote:
| Even if it is contained within the model _somewhere_, it might
| be encoded in such a way that it's impractical to extract.
| Might need an exponential time algorithm, for example. Or this
| proxy method of hundreds of deceiving attempts.
|
| And it's very difficult to train it as a next token predictor
| and at the same time make it say correctly "I don't know".
| simple_quest_9 wrote:
| Thus far, I have only found LLMs to be as good as StackExchange.
|
| Heck, maybe a little better than that.
|
| There's still some untapped potential.
|
| But, if I try to replace coding with an LLM, I find that,
| eventually, my directions become more and more specific until it
| becomes meaningless, if not outright counterproductive to use the
| LLM compared to just coding things myself.
|
| It's a good learning tool. But, that's where I stop.
___________________________________________________________________
(page generated 2024-06-05 23:02 UTC)