[HN Gopher] The LLMentalist Effect
___________________________________________________________________
The LLMentalist Effect
Author : zahlman
Score : 81 points
Date : 2025-02-08 15:30 UTC (7 hours ago)
(HTM) web link (softwarecrisis.dev)
(TXT) w3m dump (softwarecrisis.dev)
| zahlman wrote:
| Original title (too long for submission):
|
| > The LLMentalist Effect: how chat-based Large Language Models
| replicate the mechanisms of a psychic's con
| JKCalhoun wrote:
| > 1 The tech industry has accidentally invented the initial
| stages a completely new kind of mind, based on completely unknown
| principles...
|
| > 2) The intelligence illusion is in the mind of the user and not
| in the LLM itself.
|
| I've felt as though there is something in between. Maybe:
|
| 3) The tech industry invented the initial stages a kind of mind
| that, though misses the mark, is approaching something not too
| dissimilar to how an _aspect_ of human intelligence works.
|
| > By using validation statements, ... the chatbot and the psychic
| both give the impression of being able to make extremely specific
| answers, but those answers are in fact statistically generic.
|
| "Mr. Geller, can you write some Python code for me to convert a
| 1-bit .bmp file to a hexadecimal string?"
|
| Sorry, even if you think the underlying mechanisms have some sort
| of analog there's real value in LLM's, not so psychics doing
| "cold readings".
| ianbicking wrote:
| Yeah, the basic premise is off because LLM responses are
| regularly tested against ground truth (like running the code
| they produce), and LLMs don't get to carefully select what
| requests they fulfill. To the contrary they fulfill requests
| even when they are objectively incapable of answering
| correctly, such as incomplete or impossible questions.
|
| I do think there is a degree of mentalist-like behavior that
| happens, maybe especially because of the RLHF step, where the
| LLM is encouraged to respond in ways that seem more truthful or
| compelling than is justified by its ability. We appreciate the
| LLM bestowing confidence on us, and rank an answer more highly
| if it gives us that confidence... not unlike the person who
| goes to a spiritualist wanting to receive comforting news of a
| loved one who has passed. It's an important attribute of LLMs
| to be aware of, but not the complete explanation the author is
| looking for.
| prideout wrote:
| I lost interest fairly quickly because the entire article seems
| to rely on a certain definition of "intelligent" that is not made
| clear in the beginning.
| dist-epoch wrote:
| Yeah, when I read about AI solving international math olympiad
| problems it's not intelligence, it's just me projecting my math
| skills upon the model.
|
| > LLMs are a mathematical model of language tokens. You give a
| LLM text, and it will give you a mathematically plausible
| response to that text.
|
| > The tech industry has accidentally invented the initial stages
| a completely new kind of mind, based on completely unknown
| principles, using completely unknown processes that have no
| parallel in the biological world.
|
| Or maybe our mind is based on a bunch of mathematical tricks too.
| pona-a wrote:
| > AI solving international math olympiad problems is not
| intelligence
|
| But couldn't it be overfitting? LLMs are very good at deriving
| patterns, many of which humans simply can't tell apart from
| noise. With a few billion parameters and whatever black magic
| is going on inside CoT, it's not unreasonable to think even
| small amounts of fine-tuning combined with many epochs of
| training would be enough for it to conjure a compressed
| representation of that problem type.
|
| Without an extensive audit, I'd be skeptical of OpenAI's
| claims, especially given how o1 is often wrong on much more
| trivial compositional questions.
|
| What defines intelligence is generalization, the ability to
| learn new tasks from few examples, and while LLMs have made
| some significant progress here, they are still many orders
| below a child and arguably even many animals.
| svachalek wrote:
| I suspect that's actually what's going on, LLMs are finding
| patterns that apply to their question and figure out how to
| combine them in the correct way. However, I'd also say this
| is how vast majority of humans solve math problems. What I've
| seen from o1/R1 is that they are more capable at this process
| than the average human, more capable than the vast majority
| of humans.
|
| We can say that they're not "intelligent" because they're not
| capable of solving problems they can't map to something in
| their training at all, but that would also put 99.9% of
| humanity in the unintelligent bucket.
| dist-epoch wrote:
| A trained LLM can learn from a few examples.
|
| A human takes 14+ years until it's intelligent, also requires
| extensive training.
| s1mplicissimus wrote:
| 6 year olds can fairly reliably count the number of
| occurrences of a letter in a word, at least according to
| the school system I attended. LLMs will never be able to do
| it due to their inherent limitations (being statistical
| next-word predictors)
| vharuck wrote:
| >Or maybe our mind is based on a bunch of mathematical tricks
| too.
|
| Some people used to push the theory that quantum probability
| was where free will and the soul reside. That is to say, people
| will imagine how the hard questions of old neatly fit into the
| hard questions of today. Nothing won't with that, it's how we
| explore different paths and make progress. But I'm not one of
| those exploring experts, so I'll wait for stricter definitions
| and experimental data.
| swaraj wrote:
| You should try the arc agi puzzles yourself, and then tell me you
| think these things aren't intelligent
|
| https://arcprize.org/blog/openai-o1-results-arc-prize
|
| I wouldn't say it's full agi or anything yet, but these things
| can definitely think in a very broad sense of the word
| daveguy wrote:
| LLMs don't do too well on those ARC-AGI problems. Even though
| they're pretty easy for a person.
| cubefox wrote:
| https://arcprize.org/blog/oai-o3-pub-breakthrough
| daveguy wrote:
| Let me know when they can perform that well without a
| 300-shot. Or that well on unseen ARC-AGI-2.
| dosinga wrote:
| This feels rather forced. The article seems to claim both that
| LLMs don't actually work, it is all an illusion and that of
| course the LLMs know everything, they stole all our work from the
| last 20 years by scraping the internet and underpaying people to
| produce content. If it was a con, it wouldn't have to do that. Or
| in other words, if you had a psychic who actually memorized all
| biographies of all people ever, they wouldn't need their cons
| pona-a wrote:
| Why would it have to be one or the other? Yes, it's been proven
| LLMs do create world models, how good they are is a separate
| matter. There still could be goal misalignment, especially when
| it comes to RLHF.
|
| If the model has in its internal world model knowledge it
| likely does not know how to solve a coding question, but the
| RLHF stage has reviewers rate refusals lower, it would in turn
| force its hand when it comes to tricks it knows it can pull
| based on its model of human reviewers. It can only implement
| the surface level boilerplate and pass that off as a solution,
| write its code in APL to obfuscate its lack of understanding,
| or keep misinterpreting the problem into a simpler one.
|
| A psychic that read on ten thousand biographies might start to
| recall them, or he might interpolate the blanks with a generous
| dose of BS, or more likely do both in equal measure.
| EagnaIonat wrote:
| I was hoping it was talking about how it can resonate with users
| using those techniques. Or some experiments to prove the point.
| But it is not even that.
|
| There is nothing of substance in this and it feels like the
| author has a grudge against LLMs.
| manmal wrote:
| Well they have a book to sell, at the bottom of the article.
| s1mplicissimus wrote:
| > There is nothing of substance in this and it feels like the
| author has a grudge against LLMs.
|
| There is nothing of substance in your comment and it feels like
| you are venting cognitive dissonance. Looking in the mirror can
| be painful.
| pama wrote:
| This is from 2023 and is clearly dated. It is mildly interesting
| to notice how quickly things changed since then. Nowadays models
| can solve original math puzzles much of the time and it is harder
| to argue they cannot reason when we have access to R1, o1, and
| o3-mini.
| Terr_ wrote:
| > Nowadays models can solve original math puzzles much of the
| time
|
| Isn't that usually by _not even trying_ , and delegating the
| work regular programs?
| bbor wrote:
| In what way is your mathematical talent truly you, but a
| python tool called by an LLM-centric agent not truly that
| agent?
| Terr_ wrote:
| For starters, it means you should not take the success of
| the math and ascribe it to an advance in the _LLM_ , or
| whatever phrase is actually being used to describe the the
| _new_ fancy target of hype and investment.
|
| An LLM is at best, a _possible_ future component of the
| speculative future being sold today.
|
| How might future generations visualize this? I'm imagining
| some ancient Greeks, who have invented an inefficient
| reciprocating pump, which they declare is a heart and that
| means they've _basically_ built a person. (At the time,
| many believed the brain was just there to cool the blood.)
| Look! The fluid being pumped can move a lever: It 's
| _waving_ to us.
| jbay808 wrote:
| I was interested in this question so I trained NanoGPT from
| scratch to sort lists of random numbers. It didn't take long to
| succeed with arbitrary reliability, even given only an
| infinitesimal fraction of the space of random and sorted lists as
| training data. Since I can evaluate the correctness of a sort
| arbitrarily, I could be certain that I wasn't projecting my own
| beliefs onto its response, and reading more into the output than
| was actually there.
|
| That settled this question for me.
| dartos wrote:
| I don't really understand what you're testing for?
|
| Language, as a problem, doesn't have a discrete solution like
| the question of whether a list is sorted or not.
|
| Seems weird to compare one to the other, unless I'm
| misunderstanding something.
|
| What's more, the entire notion of a sorted list was provided to
| the LLM by how you organized your training data.
|
| I don't know the details of your experiment, but did you note
| whether the lists were sorted ascended or descended?
|
| Did you compare which kind of sorting was most common in the
| output and in the training set?
|
| Your bias might have snuck in without you knowing.
| tossandthrow wrote:
| Commenter is merely saying that LLMs indeed are able to
| approximate arbitrary functions exemplified through sorting.
|
| It is nothing new and has been well established in the
| literature since the 90s.
|
| The shared article really is not worth the read and mostly
| uncovers an author who does not know what he write about.
| dartos wrote:
| You're talking specifically about perceptrons and feed
| forward neural networks.
|
| LLMs didn't exist in then. Attention only came out in
| 2017...
| tossandthrow wrote:
| Yes? Are you saying that attention is less expressive?
| IshKebab wrote:
| A large number of commenters are under the illusion that LLMs
| are "just" stochastic parrots and can't generalise to inputs
| not seen in their training data. He was proving that that
| isn't the case.
| dartos wrote:
| Not saying I disagree with the thesis, but I don't think
| this proves anything.
|
| If every pair of digits appears sorted in the dataset, then
| that could still be "just" a stochastic parrot.
|
| I'm kind of interested to see if an LLM can sort when the
| dataset specifically omits comparisons between certain
| pairs of numbers.
|
| Also I don't think OC was responding to commenters, but the
| article
| jbay808 wrote:
| It might seem like you could sort with just pairwise
| correlations, but on closer analysis, you cannot.
| Generating the next correct token requires correctly
| weighing the entire context window.
| dartos wrote:
| Of course, that's how attention works, after all.
|
| But by specifically avoiding certain cases, wet could
| verify if the model is generalizing or not.
| jbay808 wrote:
| > I don't really understand what you're testing for?
|
| For this hypothesis: _The intelligence illusion is in the
| mind of the user and not in the LLM itself._
|
| The output lists were sorted in ascending order, the same way
| that I generated them for the training data.
| manmal wrote:
| Have you considered that the nature of numeric characters is
| just so predictable that they can be sorted without actually
| understanding their numerical value?
| Terr_ wrote:
| There's another illusory effect here: Humans are being encouraged
| to confuse a fictional character with the real-world "author"
| system.
|
| I can create a mad-libs program which dynamically reassembles
| stories involving a kind and compassionate Santa Claus, but that
| does not mean the program shares those qualities. I have not
| digitally reified the spirit of Christmas, not even if excited
| human kids contribute some of the words that shape its direction
| and clap with glee.
|
| P.S.: This "LLM just makes document bigger" framing is also very
| useful understanding how prompt injection and hallucinations are
| constant core behaviors, which we just ignore except when they
| inconvenience us The assistant-bot in the story can be twisted or
| vanish so abruptly because it's just something in a digital
| daydream.
| bloomingkales wrote:
| And only to your eyes and those you force your vision onto. The
| rest of the universe never sees it. You don't exist to much of
| the universe (if a tree falls and no one is around to hear it,
| you understand what I mean).
|
| So you simultaneously exist and don't exist. Sorry about this,
| your post took me on this tangent.
| karmakaze wrote:
| AlphaGo also doesn't reason. That doesn't mean it can't do things
| that humans do by reasoning. It doesn't make sense to make these
| comparisons. It's like saying that planes don't _really_ fly
| because they aren 't flapping their wings.
|
| _Edit: Don 't conflate mechanisms with capabilities._
| IshKebab wrote:
| > But there isn't any mechanism inherent in large language models
| (LLMs) that would seem to enable this
|
| Stopped reading here. What is the mechanism in _humans_ that
| enables intelligence? You don 't know? Didn't think so. So how do
| you know LLMs don't have the required mechanism?
| twobitshifter wrote:
| Lost me here - "LLMs are not brains and do not meaningfully share
| any of the mechanisms that animals or people use to reason or
| think."
|
| "the initial stages a completely new kind of mind, based on
| completely unknown principles, using completely unknown processes
| that have no parallel in the biological world."
|
| We just call it a neural network because we wanted to confuse
| biology with math for the hell of it?
|
| "There is no reason to believe that it thinks or reasons--indeed,
| every AI researcher and vendor to date has repeatedly emphasised
| that these models don't think."
|
| I mean just look at the Nobel Prize winners for counter examples
| to all of this https://www.cnn.com/2024/10/08/science/nobel-
| prize-physics-h...
|
| I don't understand the denialism behind replicating minds and
| thoughts with technology - that had been the entire point from
| the start.
| exclipy wrote:
| Yeah I was expecting the article to give an argument to back up
| this claim by talking about the mechanisms behind LLMs and the
| mechanisms behind human thought and demonstrating a lack of
| overlap.
|
| But I don't see any discussion of multilayer perceptrons or
| multi-head attention.
|
| Instead, the rest of the article is just saying "it's a con"
| with a lot of words.
| bloomingkales wrote:
| Why not make a simpler conclusion? It speaks to humans like
| people speak to Trump, a narcissist.
|
| Everything has to start with "you are great".
|
| All humans are huge narcissists and the AI is told so, and acts
| accordingly.
|
| "Isn't it weird how the beggar constantly kneels?"
|
| How silly an observation.
| olddustytrail wrote:
| > One of the issues in during this research--one that has
| perplexed me--has been that many people are convinced that
| language models, or specifically chat-based language models, are
| intelligent.
|
| Different people have different definitions of intelligence. Mine
| doesn't require thinking or any kind of sentience so I can
| consider LLMs to be intelligent simply because they provide
| intelligent seeming answers to questions.
|
| If you have a different definition, then of course you will
| disagree.
|
| It's not rocket science. Just agree on a definition beforehand.
| habitue wrote:
| This kind of "LLMs don't really do anything, it's all a trick" /
| "they're stochastic parrots" argument was kind of maybe
| defensible a year and a half ago. At this point, if you're making
| these arguments you're willfuly ignorant of what is happening.
|
| LLMs write code, today, that works. They solve hard PhD level
| questions, today.
|
| There is no trick. If anything, it's clear they haven't found a
| trick and are mostly brute forcing the intelligence they have.
| They're using unbelievable amounts of compute and are getting
| close to human level. Clearly humans still have some tricks that
| LLMs dont have yet, but that doesn't diminish what they can
| objectively do.
| habitue wrote:
| Apparently, this article was written almost exactly a year and
| a half ago so... I guess the author is forgiven!
| fleshmonad wrote:
| Unfounded cope. And I know this will get me downvoted, as these
| arguments seem to be popular among the intellectuals on this
| glorious page.
|
| The machanism of intelligence is not understood. There isn't even
| a rigorous definition of what intelligence is. "All it does is
| combine parts it has seen in its training set to give an answer",
| well then the magic lies in how it knows what parts to combine,
| if one wants to go with this argument. Also conveniently, the
| fact that we have millions of years of evolution behind us, plus
| exabytes of training data over the years in form of different
| stimuli since birth gets shoved under the rug. I don't want to
| say that the conclusion is necessarily wrong, but the argument is
| always bad. I know it is hard to come to terms with the thought
| that intelligence may be more fundamental in nature and not
| exclusively a capability of carbon based life forms.
| kelseyfrog wrote:
| Intelligence feels like a hard scientific concept, but scratch
| the surface and you find a circular definition: we measure it
| with tools we designed for the purpose, then declare it real
| because the tools say so. That's affirming the consequent.
|
| If intelligence were an objective property of the universe,
| we'd define it like mass or charge--quantifiable, invariant,
| fundamental. Instead, it shifts to match whatever we decide to
| measure. The instruments don't quantify intelligence; they
| create it.
| viach wrote:
| > 1 The tech industry has accidentally invented the initial
| stages a completely new kind of mind, based on completely unknown
| principles...
|
| > 2) The intelligence illusion is in the mind of the user and not
| in the LLM itself.
|
| 3) The intelligence of the users is illusion either?
| scandox wrote:
| You're right! AI makes us ask really important questions about
| our own intelligence. I think it will lead to greater
| recognition that we are first and foremost animals: creatures
| of intention and action. I think we've put way too much
| emphasis on our intellectual dimension in the last few hundred
| years. To the point that some people started to believe that
| was what we are.
| viach wrote:
| Yup. And the real danger of AI is not that it enslaves humans
| but in that it will bring great disillusionment and
| existential crisis.
|
| Someone should write a blog post about this to warn humanity.
| Earw0rm wrote:
| Perhaps we're confusing intelligence with awareness.
|
| What LLMs seem to emulate surprisingly well is something like a
| person's internal monologue, which is part of but not the whole
| of our mind.
|
| It's as if it has the ability to talk to itself extremely
| quickly and while plugged directly into ~all of the written
| information humanity has ever produced, and what we see is the
| output of that hidden, verbally-reasoned conversation.
|
| Something like that could be called intelligent, in terms of
| its ability to manipulate symbols and rearrange information,
| without having even a flicker of awareness, and entirely
| lacking the ability to synthesise new knowledge based on an
| intuitive or systemic understanding of a domain, as opposed to
| a complete verbal description of said domain.
|
| Or to put it another way - it can be intelligent in terms of
| its utility, without possessing even an ounce of conscious
| awareness or understanding.
| GuB-42 wrote:
| "Do LLMs think?" is a false problem outside of the field of
| philosophy.
|
| The real question that gets billions invested is "Is it useful?".
|
| If the "con artist" solves my problem, that's fine by me. It is
| like having a mentalist tell me "I see that you are having a
| leaky faucet and I see your future in a hardware store buying a
| 25mm gasket and teflon tape...". In the end, I will have my leak
| fixed and that's what I wanted, who care how it got to it?
| yannyu wrote:
| It has to both be useful and economical. If that answer cost $2
| to get you and a search+youtube video would have been just as
| effective and much cheaper, then it's possible that the new way
| to get the answer isn't significantly better than the old way.
|
| The graveyards of startups are littered with economically
| infeasible solutions to problems.
| cratermoon wrote:
| So far it seems LLM-based systems are still reaching for a use.
| crummy wrote:
| Does copilot not count?
| lukev wrote:
| I don't disagree that "is it useful" is the important question.
|
| The amount of money being invested is very clearly
| disproportionate to the current utility, and much of it is
| obviously based on the premise that LLMs can (or will soon) be
| able to think.
| ripped_britches wrote:
| This article confuses conscious, felt experience with
| intelligence.
| tomohelix wrote:
| On a bit of a tangent and hypothetical, but what if we pool eough
| resources together to do a training that includes everything a
| human can experience? I am thinking of all the five senses and
| all the data that comes with it, e.g. books, movies, songs,
| recitals, landscape, the wind brushing against the "skin", the
| pain of getting burned, the smell of coffee in the morning, the
| itchiness of a mosquito's bite, etc.
|
| It is not impossible I think, just require so much effort,
| talents, and funding that the last thing resembling such an
| endeavor was the Manhattan project. But if it succeeded, the
| impact could rival or even exceed what nuclear power had done.
|
| Or am I deluded and there is some sort of fundamental limit or
| restriction on the transformer that would completely prevent this
| from the start?
| cratermoon wrote:
| How would you model embodiment and embodied experience?
| s1mplicissimus wrote:
| Is there anything except sensory input that you assume part
| of the embodied experience? What would that be?
|
| Apart from that, I'm afraid that at this point research on
| sensory input apart from audio and visual needs much more
| advancement. For example, it's not clear to me what kind of
| data structure would be a good fit for olfactory or sensory
| training data
| tomohelix wrote:
| As mentioned above, olfactory data can be just chemical
| fingerprints. Mass spectrometers already do this and
| provide very distinct signals for every chemical component.
|
| Touch and such can have some approximation done through
| various sensors like temperature, force, humidity,
| electromagnetic, etc.
| tomohelix wrote:
| Sight, sound are quite obvious.
|
| Taste and olfactory are matters of chemical compositions. It
| will take an incredible effort but something similar to a
| mass spectrometer can be used to detect every taste and smell
| we can think of and beyond. How fast and how efficient they
| can be is probably the main challenge.
|
| Touch is difficult. We don't even know fully why or how does
| an itch "work". But force, temperature, atmospheric, humidity
| sensors, etc are widely available. They can provide a crude
| approximation, imo.
|
| Just off the top of my head. I am sure smarter people can
| come up with much more suitable ways to "embody" a machine
| learning model.
| tmnvdb wrote:
| I'm amazed people are upvoting this piece which does not grapple
| with any of the real issues in a serious way. I guess some folks
| just really want AI to go away and are longing to hear that it is
| just all newfangled nonsense from the city slickers!
| cratermoon wrote:
| I've never gotten a good answer to my question regarding why Open
| AI chose a chat UI for their gpt, but this article comes closest
| to explaining it.
| orbital-decay wrote:
| _> LLMs <snip> do not meaningfully share any of the mechanisms
| that animals or people use to reason or think._
|
| This seems to be a hard assumption the entire post, and many
| other similar ones, rely upon. But how do you know how people
| think or reason? How do you know human intelligence is not an
| illusion? Decades of research were unable to answer this. Now
| when LLMs are everywhere, suddenly everybody is an expert in
| human thinking with extremely strong opinions. To my vague
| intuition (based on understanding of how LLMs work) it's
| absolutely obvious they do share at least some fundamental
| mechanisms, regardless of vast low-level architecture/training
| differences. The entire discussion on whether it's real
| intelligence or not is based on ill-defined terms like
| "intelligence", so we can keep going in circles with it.
|
| By the way, OpenAI does nothing of this, see [1]:
|
| _> artificial general intelligence (AGI)--by which we mean
| highly autonomous systems that outperform humans at most
| economically valuable work_
|
| Neither do others. So the author describes "tech industry"
| unknown to me.
|
| [1] https://openai.com/charter/
| aaplok wrote:
| > Decades of research were unable to answer this.
|
| From the article:
|
| > The field of AI research has a reputation for disregarding
| the value of other fields [...] It's likely that, being unaware
| of much of the research in psychology on cognitive biases or
| how a psychic's con works, they stumbled into a mechanism and
| made chatbots that fooled many of the chatbot makers
| themselves.
|
| Just because cognitive scientists don't know everything about
| how intelligence works (or on what it is) doesn't mean that
| they know nothing. There has been a lot of progress in
| cognitive science, in the last decade in particular on
| reasoning.
|
| > based on ill-defined terms like "intelligence".
|
| The whole discussion is about "artificial _intelligence_ ".
| Arguably AI researchers ought to have a fairly well defined
| stance of what "intelligence" means and can't use a trick like
| "nobody knows what intelligence is" to escape criticism.
| mewpmewp2 wrote:
| As far as I know the best definition of intelligence is
| "ability to solve problems".
___________________________________________________________________
(page generated 2025-02-08 23:01 UTC)