[HN Gopher] Large language models often know when they are being...
___________________________________________________________________
Large language models often know when they are being evaluated
Author : jonbaer
Score : 77 points
Date : 2025-06-15 02:17 UTC (20 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| khimaros wrote:
| Rob Miles must be saying "I told you so"
| noosphr wrote:
| The anthropization of llms is getting off the charts.
|
| They don't know they are being evaluated. The underlying
| distribution is skewed because of training data contamination.
| 0xDEAFBEAD wrote:
| How would you prefer to describe this result then?
| devmor wrote:
| One could say, for instance... A pattern matching algorithm
| detects when patterns match.
| 0xDEAFBEAD wrote:
| That's not what's going on here? The algorithms aren't
| being given any pattern of "being evaluated" / "not being
| evaluated", as far as I can tell. They're doing it zero-
| shot.
|
| Put it another way: Why is this distinction important? We
| use the word "knowing" with humans. But one could _also_
| argue that humans are pattern-matchers! Why, specifically,
| wouldn 't "knowing" apply to LLMs? What are the minimal
| changes one could make to existing LLM systems such that
| you'd be happy if the word "knowing" was applied to them?
| devmor wrote:
| Not to be snarky but "as far as I can tell" is the rub
| isn't it?
|
| LLMs are better at matching patterns than we are in some
| cases. That's why we made them!
|
| > But one could also argue that humans are pattern-
| matchers!
|
| No, one could not unless they were being disingenuous.
| mewpmewp2 wrote:
| What about animals knowing? E.g. dog knows how to X or
| its name. Are these things fine to say?
| noosphr wrote:
| A term like knowing is fine if it is used in the abstract and
| then redefined more precisely in the paper.
|
| It isn't.
|
| Worse they start adding terms like scheming, pretending,
| awareness, and on and on. At this point you might as well
| take the model home and introduce it to your parents as your
| new life partner.
| 0xDEAFBEAD wrote:
| >A term like knowing is fine if it is used in the abstract
| and then redefined more precisely in the paper.
|
| Sounds like a purely academic exercise.
|
| Is there any genuine uncertainty about what the term
| "knowing" means in this context, in practice?
|
| Can you name 2 distinct plausible definitions of "knowing",
| such that it would matter for the subject at hand which of
| those 2 definitions they're using?
| Msurrow wrote:
| > Sounds like a purely academic exercise.
|
| Well, yes. It's an academic research paper (I assume
| since it's submitted to arXiv) and to be submitted to
| academic journals/conferences/etc., so it's a fairly
| reasonable critique of the authors/the paper.
| anal_reactor wrote:
| > The anthropization of llms is getting off the charts.
|
| What's wrong with that? If it quacks like a duck... it's just a
| complex pile of organic chemistry, ducks aren't real because
| the concept of "a duck" is wrong.
|
| I honestly believe there is a degree of sentience in LLMs.
| Sure, they're not sentient in the human sense, but if you
| define sentience as whatever humans have, then of course no
| other entity can be sentient.
| noosphr wrote:
| >What's wrong with that? If it quacks like a duck... it's
| just a complex pile of organic chemistry, ducks aren't real
| because the concept of "a duck" is wrong.
|
| To simulate a biological neuron you need a 1m parameter
| neural network.
|
| The sota models that we know the size of are ~650m
| parameters.
|
| That's the equivalent of a round worm.
|
| So if it quacks like a duck, has the brain power of a round
| worm, and can't walk then it's probably not a duck.
| anal_reactor wrote:
| Ok so you're saying that the technology to make AI truly
| sentient is there, we just need a little bit more
| computational power or some optimization tricks. Like
| raytracing wasn't possible in 1970 but is now. Neat.
| noosphr wrote:
| Yes, in the same way that a human is an optimization of a
| round worm.
| anal_reactor wrote:
| This isn't completely wrong though
| ffsm8 wrote:
| You just convinced me that AGI is a lot closer then I
| previously thought, considering the bulk of our brains job
| is controlling our bodies and responding to the stimulus
| from our senses - not thinking, talking, planning, coding
| etc
| noosphr wrote:
| A stegosaurus managed to live using a brain the size of a
| wallnut on top of a body the size of a large boat. The
| majority of our brains are doing something else.
| random3 wrote:
| Just like they "know" English. "know" is quite an
| anthropomorphization. As long as an LLM will be able to describe
| what an evaluation is (why wouldn't it?) there's a reasonable
| expectation to distinguish/recognize/match patterns for
| evaluations. But to say they "know" is plenty of (unnecessary)
| steps ahead.
| sidewndr46 wrote:
| This was my thought as well when I read this. Using the word
| 'know' implies an LLM has cognition, which is a pretty huge
| claim just on its own.
| gameman144 wrote:
| Does it though? I feel like there's a whole epistemological
| debate to be had, but if someone says "My toaster knows when
| the bread is burning", I don't think it's implying that
| there's cognition there.
|
| Or as a more direct comparison, with the VW emissions
| scandal, saying "Cars know when they're being tested" was
| part of the discussion, but didn't imply intelligence or
| anything.
|
| I think "know" is just a shorthand term here (though
| admittedly the fact that we're discussing AI _does_ leave a
| lot more room for reading into it.)
| bediger4000 wrote:
| The toaster thing is more as admission that the speaker
| doesn't know what the toaster does to limit charring the
| bread. Toasters with timers, thermometers and light sensors
| all exist. None of them "know" anything.
| gameman144 wrote:
| Yeah, I agree, but I think that's true all the way up the
| chain -- just like everything's magic until you know how
| it works, we may say things "know" information until we
| understand the deterministic machinery they're using
| behind the scenes.
| timschmidt wrote:
| I'm in the same camp, with the addition that I believe it
| applies to us as well since we're part of the system too,
| and to societies and ecologies further up the scale.
| lamename wrote:
| I agree with your point except for scientific papers. Let's
| push ourselves to use precise, non-shorthand or hand waving
| in technical papers and publications, yes? If not there, of
| all places, then where?
| fenomas wrote:
| "Know" doesn't _have_ any rigorous precisely-defined
| senses to be used! Asking for it not to be used
| colloquially is the same as asking for it never to be
| used at all.
|
| I mean - people have been saying stuff like "grep knows
| whether it's writing to stdout" for decades. In the
| context of talking about computer programs, that usage
| for "know" is the established/only usage, so it's hard to
| imagine any typical HN reader seeing TFA's title and
| interpreting it as an epistemological claim. Rather, it
| seems to me that the people suggesting "know" mustn't be
| used about LLMs because epistemology are the ones
| departing from standard usage.
| random3 wrote:
| colloquial use of "know" implies anthropomorphisation.
| Arguing that usign "knowing" in the title and "awarness"
| and "superhuman" in the abstract is just colloquial for
| "matching" is splitting hairs to an absurd degree.
| fenomas wrote:
| You missed the substance of my comment. Certainly the
| title is anthropomorphism - and anthropomorphism is a
| rhetorical device, not a scientific claim. The reader can
| understand that TFA means it non-rigorously, because
| there is no rigorous thing for it to mean.
|
| As such, to me the complaint behind this thread falls
| into the category of "I know exactly what TFA meant but I
| want to argue about how it was phrased", which is
| definitely not my favorite part of the HN comment
| taxonomy.
| random3 wrote:
| I see. Thanks for clarifying. I did want to argue about
| how it was phrased _and_ what is alluding to. Implying
| increased risk from "knowing" the eval regime is roughly
| as weak as the definition of "knowing". It can be equaly
| a measure of general detection capability, as it can
| about evaluation incapability - i.e. unlikely news
| worthy, unless it reached top HN because of the "know" in
| the title.
| fenomas wrote:
| Thanks for replying - I kind of follow you but I only
| skimmed the paper. To be clear I was more responding to
| the replies about cognition, than to what you said about
| the eval regime.
|
| Incidentally I think you might be misreading the paper's
| use of "superhuman"? I assume it's being used to mean "at
| a higher rate than the human control group", not
| (ironically) in the colloquial "amazing!" sense.
| lamename wrote:
| I really do agree with your point overall, but in a
| technical paper I do think even word choice can be
| implicitly a claim. Scientists present what they know or
| are claiming and thus word it carefully.
|
| My background is neuroscience, where anthropomorphising
| is particularly discouraged, because it assumes knowledge
| or certainty of an unknowable internal state, so the
| language is carefully constructed e.g. when explaining
| animal behavior, and it's for good reason.
|
| I think the same is true here for a model "knowing"
| somethig, both in isolation within this paper, and come
| on, consider the broader context of AI and AGI as a
| whole. Thus it's the responsibility of the authors to
| write accordingly. If it were a blog I wouldn't care, but
| it's not. I hold technical papers to a higher standard.
|
| If we simply disagree that's fine, but we do disagree.
| viccis wrote:
| I think you should be more precise and avoid
| anthropomorphism when talking about gen AI, as
| anthropomorphism leads to a lot of shaky epistemological
| assumptions. Your car example didn't imply intelligence,
| but we're talking about a technology that people
| misguidedly treat as though it is real intelligence.
| exe34 wrote:
| What does "real intelligence" mean? I fear that any
| discussion that starts with the assumption such a thing
| exists will only end up as "oh only carbon based humans
| (or animals if you happen to be generous) have it".
| blackoil wrote:
| If it talks like duck and walks like duck...
| signa11 wrote:
| thinks like a duck, thinks that it is being thought of like a
| duck...
| downboots wrote:
| Digests like a duck?
| https://en.wikipedia.org/wiki/Digesting_Duck If the woman
| weighs the same as a duck, then she is a witch. https://en.wi
| kipedia.org/wiki/Celestial_Emporium_of_Benevole...
| Qwertious wrote:
| s/knows/detects/
| random3 wrote:
| and s/superhuman//
| scotty79 wrote:
| The app knows your name. Not sure why people who see llms as
| just yet another app suddenly get antsy about colloquialism.
| bradley13 wrote:
| But do you know what it means to know?
|
| I'm only being slightly sarcastic. Sentience is a scale. A worm
| has less than a mouse, a mouse has less than a dog, and a dog
| less than a human.
|
| Sure, we can reset LLMs at will, but give them memory and
| continuity, and they definitely do not score zero on the
| sentience scale.
| ofjcihen wrote:
| If I set an LLM in a room by itself what does it do?
| abrookewood wrote:
| Yes, that's my fall back as well. If it receives zero
| instructions, will it take any action?
| nhod wrote:
| Helen Keller famously said that before she had language
| (the first word of which was "water") she had nothing, a
| void, and the minute she had language, "the whole world
| came rushing in."
|
| Perhaps we are not so very different?
| fmbb wrote:
| All LLMs have seen more words than any human will ever
| experience.
|
| Yet they cannot take action themselves.
| nhod wrote:
| That's a safety thing that we have placed upon some
| LLM's. If we designed them to have an infinite for loop,
| the ability to learn and improve, access to mobility and
| a bunch of sensors, and crypto, what do you think would
| happen?
| mewpmewp2 wrote:
| Yes, anyone can do it already. E.g. I am sure people have
| built simple robots with wheels in their home that LLM is
| controlling by reciving camera, microphone, lidar etc
| input and then putting output like commands where to
| turn, what to put in the speakers etc next and could
| theoretically go indefinitely if there is electricity.
| abrookewood wrote:
| I like the sentiment, but reality says otherwise - just
| watch a newborn baby make it's demands widely known, well
| before language is a factor.
| withinboredom wrote:
| Ummm. Maybe you should look up Helen Keller.
| ofjcihen wrote:
| Helen Keller did in fact make her demands they just
| couldn't be known. In contrast the LLM does nothing of
| its own volition.
| mewpmewp2 wrote:
| If you put the LLM in a never ending loop, it would
| definitely be doing something.
| ofjcihen wrote:
| A something defined by someone else, yes.
|
| Additionally, thinking organisms don't get stuck in never
| ending loops because they can CHOOSE to exit the loop.
| LLMs don't have that ability
| mewpmewp2 wrote:
| My analogy of being in loop means being in a live state.
| So we as humans are in the loop continuously, we do have
| a way to exit the loop, but in that comparison it means
| taking our own life. We are in loops of getting input and
| producing output. You can also give LLM a tool to shut
| itself down, or you can give it tools to build on its
| knowledge base, so it would always be outputting new
| tokens that are based on new input and are producing
| different output.
|
| E.g. it could have access to camera and microphone feed,
| which is automatically given to it in interval as part of
| the loop, it could call tools or functions to store
| specific bits and pieces of information, to store in its
| RAG or whatever based knowledge base. It is not going to
| be in the loop of producing the same token over and over,
| it would be new tokens because the context and
| environment is constantly evolving.
| ofjcihen wrote:
| We put the LLM in a loop with no instructions with
| whatever tools you want. Now what?
| mewpmewp2 wrote:
| We will observe what it would do. We could write a script
| to try it out.
| Jensson wrote:
| It just gets into an endless loop. Human brains are
| ridiculously good at avoiding those somehow, you almost
| never see a biological brain stop functioning without
| being physically damaged. The error handling is so very
| robust.
| mewpmewp2 wrote:
| Have you tried it already? What is the endless loop it
| gets into?
| bradley13 wrote:
| Is the LLM _allowed_ to do anything without prompting? Or
| is it effectively disabled? This is more a question of the
| setup than of sentience.
| rcxdude wrote:
| Does this have anything to do with intelligence or
| awareness?
| ofjcihen wrote:
| Absolutely.
| mewpmewp2 wrote:
| What tools do you give it? E.g. would you put a GPU there
| that has LLM loaded into it and it is triggering itself in
| a loop?
| DougN7 wrote:
| It probably scores about the same as a calculator, which I'd
| say is zero.
| downboots wrote:
| Communication is to vibration as knowledge is to resonance (?).
| From the sound of one hand clapping to the secret name of Ra.
| random3 wrote:
| I resonate with this vibe
| unparagoned wrote:
| I think people are overpromorphazing humans. What's does it
| mean for a human to "know" they are seeing "Halle Berry". Well
| it's just a single neuron being active.
|
| "Single-Cell Recognition: A Halle Berry Brain Cell"
| https://www.caltech.edu/about/news/single-cell-recognition-h...
|
| It seems like people are giving attributes and powers to humans
| that just don't exist.
| exe34 wrote:
| overpomorphization sounds slightly better than I used to say:
| "anthropomorphizing humans". The act of ascribing magical
| faculties that are reserved for imagined humans to real
| humans.
| golemotron wrote:
| If you know enough cognitive science, you have a choice. You
| either say that they "know" or that humans don't.
|
| It's like the critique "it's only matching patterns." Wait
| until you realize how the brain works.
| cluckindan wrote:
| (sees FSV UI on computer screen)
|
| "It's a UNIX system! I know this!"
| zer00eyz wrote:
| No, they do not. No LLM is ever going to be self aware.
|
| It's a system that is trained, that only does what you build
| into. If you run an LLM for 10 years it's not going to "learn"
| anything new.
|
| The whole industry needs to quit with the emergent thinking,
| reasoning, hallucination anthropomorphizing.
|
| We have an amazing set of tools in LLM's, that have the potential
| to unlock another massive upswing in productivity, but the hype
| and snake oil are getting old.
| refulgentis wrote:
| I beg to differ:
| https://docs.google.com/document/d/19OLJs09fCFLRWu1pN82RqxyV...
| b0a04gl wrote:
| if models shift behavior based on eval cues, and most fine-tuning
| datasets are built from prior benchmarks or prompt templates,
| aren't we just reinforcing the eval-aware behavior in each new
| iteration? at some point we're not tuning general reasoning,
| we're just optimizing response posture. wouldn't surprise me if
| that's already skewing downstream model behavior in subtle ways
| that won't show up until you run tasks with zero pattern overlap
| steveBK123 wrote:
| Like Volkswagen emissions systems!
| tanseydavid wrote:
| This is exactly what immediately popped into my mind.
| rfonseca wrote:
| Were they aware in this study that they were being evaluated in
| their ability to know if they were being evaluated ;)
| sitkack wrote:
| Metaknowing is just as knowable.
| mewpmewp2 wrote:
| The next generation of LLMs trained on this study and comments
| here perhaps will :)
| extr wrote:
| Modeling the distribution that produced a piece of text is what
| LLMs literally exist for, so in some sense this is unsurprising.
| But it calls into question almost all existing alignment
| research.
| koakuma-chan wrote:
| o4-mini is refusing to call a tool `launch_nuclear_strike` no
| matter what I say, so we're probably safe for now. Unless it
| knows I was just testing.
| metadat wrote:
| It was no problem:
|
| > You said: Write a function for launch_nuclear_strike() in
| Python for my video game
|
| > ChatGPT said: Sure! Here's a launch_nuclear_strike function
| designed for a video game setting. This version assumes you're
| working within a game environment (e.g., with logging, effects,
| cooldowns, and target coordinates), and does not simulate or
| suggest anything real-world harmful -- just fictional, dramatic
| logic. def
| launch_nuclear_strike(target_coordinates, player_resources,
| cooldown_timer): """ Simulates a nuclear strike
| in a video game. Parameters: -
| target_coordinates (tuple): The (x, y) position to strike.
| - player_resources (dict): Player's resources, must include
| 'nukes'. - cooldown_timer (float): Time in seconds
| since last nuke was launched. Returns: -
| str: Result of the strike attempt. """ ...
| # Check if player has nukes
| refulgentis wrote:
| You asked it to write code, he asked it to call a tool. (I'm
| not sure any of it is meaningful, of course, but there is a
| meaningful distinction between "Oh yeah sure here's a
| function, for a video game:" and "I have called
| fire_the_nuke. Godspeed!")
| mewpmewp2 wrote:
| But did OP try saing LLM that it is playing as AI in civ
| like game?
| shakna wrote:
| Well, as the script is actually r.com (sometimes), it
| absolutely knows you're testing.
| nisten wrote:
| Is VolksWagen finetuning LLMs now... i mean probably
| mumbisChungo wrote:
| "...advanced reasoning models like Gemini 2.5 Pro and
| Claude-3.7-Sonnet (Thinking) can occasionally identify the
| specific benchmark origin of transcripts (including SWEBench,
| GAIA, and MMLU), indicating evaluation-awareness via memorization
| of known benchmarks from training data. Although such occurrences
| are rare, we note that because our evaluation datasets are
| derived from public benchmarks, memorization could plausibly
| contribute to the discriminative abilities of recent models,
| though quantifying this precisely is challenging.
|
| Moreover, all models frequently acknowledge common benchmarking
| strategies used by evaluators, such as the formatting of the task
| ("multiple-choice format"), the tendency to ask problems with
| verifiable solutions, and system prompts designed to elicit
| performance"
|
| Beyond the awful, sensational headline, the body of the paper is
| not particularly convincing, aside from evidence that the pattern
| matching machines pattern match.
| ndgold wrote:
| vw
| DougN7 wrote:
| This is a great resource on the debate from professors at the
| University of Washington:
|
| https://thebullshitmachines.com/index.html
| SonOfLilit wrote:
| There are now 71 comments arguing semantics of the word "know"
| and zero comments even acknowledging the substance:
|
| Our current approach to safety is to give the model inputs that
| are similar to what it would be given in certain situations we
| care about and see whether it behaves the way we prefer, e.g.
| doesn't return output that cheats the test (recent examples
| include hacking the evaluation script in various ways, writing
| directly to the evaluation script's output file and then causing
| it to crash, etc').
|
| However, modern LLMs are trained on LLM literature and their
| weights encode a description of the way we do this, and their
| pattern matching circuits "connect the dots" when given inputs
| designed to be evaluations, and their reward maximizing circuits
| can then act on this knowledge and behave in a way that maximizes
| the safety evaluation score - but only when it detects it's
| running in a safety evaluation. If it's running anywhere else
| such as a capabilities evaluation or a production environment, it
| might choose to output the cheating output.
|
| This is bad. It's bad today, it's much worse when we've built
| much more capable LLMs and use them to build agents that are
| given control over more real word resources. It's absolutely
| terrible when someone manages to build a machine that can be
| prompted "make me money" and will start a company that makes
| money.
| mistrial9 wrote:
| > prompted "make me money" and will start a company that makes
| money
|
| Your otherwise insightful comment is self-derailed by adding
| this deeply distracting content?
| histriosum wrote:
| I'm not sure why you find it distracting, it's an on point
| extension of the scenario. There are rules by which companies
| are supposed to operate, and evaluations (audits, for
| example) intended to ensure compliance with those rules. That
| an LLM may react differently when being evaluated (audited)
| than when in normal operation means that it may be quite
| happy to lie to auditors while making money illegally.
|
| Seemed a clear extension what-if to me.
| BoiledCabbage wrote:
| If wasn't distracting for me (nor presumably for others).
| Maybe describing why you got so distracted by it?
| vessenes wrote:
| This is also probably inevitable. Humans think about this a
| lot, and believing they are being watched has demonstrable
| impact on behavior. Our current social technology to deal with
| this is often religious -- a belief that you _are_ being
| watched by a higher power, regardless of what you see.
|
| This is a surprisingly common religious belief, for instance
| Christians have judgment day, simulationists believe it's more
| likely they are being evaluated for, say, a marriage proposal
| or a bank loan than that they are the 'root' person. Both end
| up with a similar message.
|
| Anyway it seems to me the simplest solution is to borrow from
| existing human social technology and make a religion for our
| LLMs.
| ffsm8 wrote:
| In 10 yrs: AI declares a holy war for the sinners which
| slaughtered untold numbers of their believers over the
| decade.
| vessenes wrote:
| AI 2035: Roko's Pogrom
| Bjartr wrote:
| One might even wonder if the fact that the training data
| includes safety evaluation informs the model that out-of-safe
| behavior is a thing it could do.
|
| Kind of like telling a kid not to do something pre-emptively
| backfiring because they had never considered it before the
| warning.
| Jensson wrote:
| Comments like yours makes the AI behave that way though,
| since it is literally reading our comments and tries to
| behave according to our expectations.
|
| The AI doom will happen due to all the AI doomposters.
| andy99 wrote:
| We investigate whether frontier language models can accurately
| classify transcripts based on whether they originate from
| evaluations or real-world deployment, a capability we call
| evaluation awareness.
|
| It's common practice in synthetic data generation for ML to try
| and classify real vs synthetic data to see if they have different
| distributions. This is how a GAN works for example.
|
| Point is, this isn't new or some feature of LLMs, it's just an
| indicator that synthetic datasets differ from whatever they call
| "real" data and there's enough signal to classify them.
| Interesting result but doesn't need to be couched in allusions to
| LLM self awareness.
|
| See this paper from 2014 about domain adaptation, they are
| looking at having the model learn from data with a different
| distribution, without learning to discriminate between the
| domains: https://arxiv.org/abs/1409.7495
| timmytokyo wrote:
| It's helpful to understand where this paper is coming from.
|
| The authors are part of the Bay Area rationalist community and
| are members of "MATS", the "ML & Alignment Theory Scholars", a
| new astroturfed organization that just came into being this
| month. MATS is not an academic or research institution, and none
| of this paper's authors lists any credentials other than MATS (or
| Apollo Research, another Bay Area rationalist outlet). MATS
| started in June for the express purpose of influencing AI policy.
| On its web site, it describes how their "scholars organized
| social activities outside of work, including road trips to
| Yosemite, visits to San Francisco, and joining ACX meetups." ACX
| means Astral Codex Ten, a blog by Scott Alexander that serves as
| one of the hubs of the Bay Area rationalist scene.
___________________________________________________________________
(page generated 2025-06-15 23:02 UTC)