[HN Gopher] OpenAI's "Study Mode" and the risks of flattery
___________________________________________________________________
OpenAI's "Study Mode" and the risks of flattery
Author : benbreen
Score : 95 points
Date : 2025-07-31 13:35 UTC (2 days ago)
(HTM) web link (resobscura.substack.com)
(TXT) w3m dump (resobscura.substack.com)
| bartvk wrote:
| I'm Dutch and we're noted for our directness and bluntness. So my
| tolerance for fake flattery is zero. Every chat I start with an
| LLM, I prefix with "Be curt".
| cheschire wrote:
| Imagine what happens to Dutch culture when American trained AI
| tools force American cultural norms via the Dutch language onto
| the youngest generation.
|
| And I'm not implying intent here. It's simply a matter of
| source material quantity. Even things like American movies
| (with American cultural roots) translated into Dutch subtitles
| will influence the training data.
| jstummbillig wrote:
| What will happen? Californication has been around for a
| while, and, if anything, I would argue that AI is by design
| less biased than pop culture.
| cheschire wrote:
| Pop culture is not the intent of "study mode".
| scott_w wrote:
| Your comment reminds me of quirks of translations from
| Japanese to English where you see common phrases reused in
| the "wrong" context for English. "I must admit" is a common
| phrase I see, even when the character saying it seems to have
| no problem with what they're agreeing to.
| arrowsmith wrote:
| The Americanisation of European culture long predates LLMs.
| grues-dinner wrote:
| Embedding "your" AI at every level of everyone else's
| education systems seems like the setup for a flawless
| cultural victory in a particularly ham-fisted sci-fi
| allegory.
|
| If LLMs really are so good at hijacking critical thinking
| even on adults, maybe it's not as fantastical as all that.
| airstrike wrote:
| In my experience, whenever you do that, the model then
| overindexes on criticism and will nitpick even minor stuff. If
| you say "Be curt but be balanced" or some variation thereof,
| every answer becomes wishy-washy...
| AznHisoka wrote:
| Yeah, when I tell it to "Just be honest dude" it then tells
| me I'm dead wrong. I inevitably follow up with "No, not that
| KIND of honest!"
| cruffle_duffle wrote:
| Maybe we need to go like they do in the movies "set
| truthfulness to 95%, curtness at 67% and just a touch of dry
| british humor (10%)"
| tallytarik wrote:
| I've tried variations of this. I find it will often cause it to
| include cringey bullshit phrases like:
|
| "Here's your brutally honest answer-just the hard truth, no
| fluff: [...]"
|
| I don't know whether that's better or worse than the fake
| flattery.
| BrawnyBadger53 wrote:
| Similar experience, feels very ironic
| dcre wrote:
| Curious whether you find this on the best models available. I
| find that Sonnet 4 and Gemini 2.5 Pro are much better at
| following the spirit of my system prompt rather than the
| letter. I do not use OpenAI models regularly, so I'm not sure
| about them.
| danielscrubs wrote:
| That is not the spirit nor the letter though.
| arrowsmith wrote:
| You need a system prompt to get that behaviour? I find
| ChatGPT does it _constantly_ as its default setting:
|
| "Let's be blunt, I'm not gonna sugarcoat this. Getting
| straight to the hard truth, here's what you could cook for
| dinner tonight. Just the raw facts!"
|
| It's so annoying it makes me use other LLMs.
| cruffle_duffle wrote:
| Its response is still flattery, just packaged in a different
| form. Patronizing, actually.
| ggsp wrote:
| I've seen a marked improvement after adding "You are a machine.
| You do not have emotions. You respond exactly to my questions,
| no fluff, just answers. Do not pretend to be a human. Be
| critical, honest, and direct." to the top of my personal
| preferences in Claude's settings.
| j_bum wrote:
| I'll have to give this a try. I've always included "Be
| concise. Excessive verbosity is a distraction."
|
| But it doesn't work much ...
| siva7 wrote:
| Saved my sanity. Thanks
| arrowsmith wrote:
| I need to use this in Gemini. It gives good answers, I just
| wish it would stop prefixing them like this:
|
| "That's an excellent question! This is an astute insight that
| really gets to the heart of the matter. You're thinking like
| a senior engineer. This type of keen observation is exactly
| what's needed."
|
| Soviet commissars were less obsequious to Stalin.
| croes wrote:
| Are you telling me they lie to me and I'm not the greatest
| programmer of all time?
| snoman wrote:
| You couldn't be because I have it on good authority that
| I am.
| tempodox wrote:
| Obviously some of the invested money went into
| psychologists to get their victims totally hooked in no
| time. These machines will be the end of social media as
| we know it. Why would you chat with people when a bot can
| flatter you so much better?
| arrowsmith wrote:
| I don't think it takes a psychologist. Maybe the LLMs are
| sycophantic because that's what the humans in the RLHF
| loop respond best to.
| felipeerias wrote:
| Perhaps you should consider adding "be more Dutch" to the
| system prompt.
|
| (I'm serious, these things are so weird that it would probably
| work.)
| bartvk wrote:
| That is funny, I'm going to test that!
| t0mas88 wrote:
| Same here. Together with putting random emojis in answers. It's
| so over the top that saying "Excellent idea, rocket emoji" is a
| running joke with my wife when the other says something obvious
| :-)
| siva7 wrote:
| Let's face it. There is no one size fits all for this category.
| There won't be a single winner that takes it all. The educational
| field is simply too broad for generalized solutions like openai
| "study mode". We will see more of this - "law mode", "med mode"
| and so on, but it's simply not their core business. What are
| openai and co trying to achieve here? Continuing until FTC breaks
| them up?
| tempodox wrote:
| > Continuing until FTC breaks them up?
|
| No danger of that, the system is far too corrupt by now.
| neom wrote:
| I don't like this framing "But for people with mental illness, or
| simply people who are particularly susceptible to flattery, it
| could have had some truly dire outcomes."
|
| I thought the AI safety risk stuff was very over-blown in the
| beginning. I'm kinda embarrassed to admit this: About 5/6 months
| ago, right when ChatGPT was in it's insane sycophancy mode I
| guess, I ended up locked in for a weekend with it...in...what was
| in retrospect, a kinda crazy place. I went into physics and the
| universe with it and got to the end thinking..."damn, did I
| invent some physics???" Every instinct as a person who
| understands how LLMs work was telling me this is crazy LLMbabble,
| but another part of me, sometimes even louder, was like "this is
| genuinely interesting stuff!" - and the LLM kept telling me it
| was genuinely interesting stuff and I should continue - I even
| emailed a friend a "wow look at this" email (he was like, dude,
| no...) I talked to my wife about it right after and she basically
| had me log off and go for a walk. I don't think I would have
| gotten into a thinking loop if my wife wasn't there, but maybe,
| and then that would have been bad. I feel kinda stupid admitting
| this, but I wanted to share because I do now wonder if this kinda
| stuff may end up being worse than we expect? Maybe I'm just
| particularly susceptible to flattery or have a mental illness?
| johnisgood wrote:
| Can you tell us more about the specifics? What rabbit hole did
| you went into that was so obvious to everyone ("dude, no",
| "stop, go for a walk") but you that it was bullshit?
| iwontberude wrote:
| Thinking you can create novel physics theories with the help
| of an LLM is probably all the evidence I needed. The premise
| is so asinine that to actually get to the point where you are
| convinced by it seems very strange indeed.
| gitremote wrote:
| "I'm doing the equivalent of vibe coding, except it's vibe
| physics." - Travis Kalanick, founder of Uber
|
| https://gizmodo.com/billionaires-convince-themselves-ai-
| is-c...
| kaivi wrote:
| > The premise is so asinine
|
| I believe it's actually the opposite!
|
| Anybody armed with this tool and little prior training
| could learn the difference between a Samsung S11 and the
| symmetry, take a new configuration from the endless search
| space that it is, correct for the dozen edge cases like the
| electron-phonon coupling, and publish. Maybe even pass peer
| review if they cite the approved sources. No requirement to
| work out the Lagrangians either, it is also 100% testable
| once we reach Kardashev-II.
|
| This says more about the sad state of modern theoretical
| physics than the symbolic gymnastics required to make
| another theory of everything sound coherent. I'm hoping
| that this new age of free knowledge chiropractors will
| change this field for the better.
| jeff-davis wrote:
| My friend once told me that physics formulas are like
| compression algorithms: a short theory can explain many
| data points that fit a pattern.
|
| If that's true, then perhaps AIs would come up with
| something just by looking at existing observations and
| "summarizing" them.
|
| Far-fetched, but I try to keep an open mind.
| neom wrote:
| Sure, here are some excerpts that should provide insight as
| to where I was digging: https://s.h4x.club/E0uvqrpA
| https://s.h4x.club/8LuKJrAr https://s.h4x.club/o0u0DmdQ
|
| (Edit: Thanks to the couple people who emailed me, don't
| worry I'm laying off the LLM sauce these days :))
| lubujackson wrote:
| I have no idea what this is going on about. But it is
| clearly much more convincing with (unchecked) references
| all over the place.
|
| This seems uncannily similar to anti-COVID vaccination
| thinking. It isn't people being stupid because if you dig
| you can find heaps of papers and references and details and
| facts. So much so that the human mind can be easily
| convinced. Are those facts and details accurate? I doubt
| it, but the volume of slightly wrong source documents seems
| to add up to something convincing.
|
| Also similar to how finance people made tranches of bad
| loans and packaged them into better rated debt, magically.
| It seems to make sense at each step but it is ultimately an
| illusion.
| apsurd wrote:
| had a look, I don't see it as bullshit, it's just not
| groundbreaking.
|
| Nature is overwhelmingly non-linear. Most of human
| scientific progress is based on linear understandings.
|
| Linear as in for this input you get this output. We've made
| astounding progress.
|
| Its just not a complete understanding of the natural world
| because most of reality can't actually be modeled linearly.
| neom wrote:
| I think it's not as much about how right or wrong or
| interesting or not the output was, for me anyway, the
| concern is that I got a bit... lost in myself, I have
| real things to do that are important to people around me,
| they do not involve spending hours with an LLM trying to
| understand the universe. I'm not a physicist, I have a
| family to provide for, and I suppose someone less lucky
| than myself could go down a terrible path.
| johnisgood wrote:
| Okay, but like I said before in another comment, I have
| spent 3 days straight coding, neglecting myself and
| everything around me in the process. I was learning a
| lot, coding a lot. I was productive. Of course I should
| have had some breaks (for my legs and mind, and my body).
| Just make sure to have breaks. I did not have breaks
| because I was completely zoned in. I set up a timer by
| then that remind me to take a break.
|
| I checked the content, I do not think that it is useless,
| and I am sure you have learnt a lot. Perhaps get in a
| rabbit hole about http://CharlieLabs.ai (your project,
| before people think I am advertising). :P
| roywiggins wrote:
| Lengthy ChatGPT rabbit holes are kind of a _simulacrum_
| of productivity, they keep you in a flow state but it 's
| liable to be pure cotton candy, not actual productivity.
|
| Spending all weekend on a puzzle or a project at least
| keeps you in a tight feedback loop with _something_
| outside your own skull. ChatGPT offers you a perfect
| mirror of the inside of your own skull _while pretending
| to be a separate entity_. I think this is one reason why
| it can be both compelling and risky to engage deeply with
| them: it _feels_ like more than it is. It eliminates a
| lot of the friction that might take you out of a flow
| state, but without that friction you can just spin out.
| johnisgood wrote:
| It depends. Do not pursue pure cotton candy. :P
| roywiggins wrote:
| Put it this way: at least with vibe coding you'll
| eventually hit something where you realize that it's
| produced crappy, useless code that you need to throw out.
|
| With extended philosophical conversations there is
| nothing grounding the conversation, nothing to force you
| to come up short and realize when you've spent hours
| pursuing something mistaken. It's intellectual empty
| calories.
| bonoboTP wrote:
| Depends on how you use it. You can "ground" it by asking
| what authors have explored this or ask for book
| recommendations, then read the wiki page of the author,
| read some texts by them etc. You can explore the history
| as well, like what was happening at that time, who were
| important contemporaries or influences, people who
| thought the opposite etc. I've found interesting books
| (that are somewhat niche but fairly well known in the
| field, non-fringe) this way.
| roywiggins wrote:
| One thing I noticed from chat #1 is that you've got a sort
| of "God of the gaps" ("woo of the gaps"?) thing going on-
| you've bundled together a bunch of stuff that is currently
| beyond understanding and decided that they must all be
| related and explainable by the same thing.
|
| Needless to say this is super common when people go down
| quasi-scientific/spiritual/woo rabbit holes- all this stuff
| that scientists don't understand must be related! It must
| all have some underlying logic! But there's not much reason
| to actually think that, a priori.
|
| One thing that the news stories about people going off the
| deep end with LLMs is that that basically never share the
| full transcripts, which is of course their right, but I
| wonder if it would nevertheless be a useful thing for
| people to be able to study. On the other hand, they're kind
| of a roadmap to turning certain people insane, so maybe
| it's best that they're not widely distributed.
|
| I don't usually believe in "cognitohazards" but if they
| exist, it seems like we have maybe invented them with these
| chatbots...
| neom wrote:
| I don't think it's bad or a big deal for people to look
| for wide connections in things, or at least to explore
| different ideas in life and trying to understand them
| deeper - Can it lead to problematic behaviour, sure, and
| I think for me at least that was introduced when the LLM
| started to try to convince ME my ideas were good, even
| though I was effectively just day dreaming with it. For
| me personally, I don't feel I need to look any more
| foolish than I feel, even now knowing how openai had the
| LLM temperature set, I'm surprised I didn't force myself
| to be more skeptical, I'm educated I have critical
| thinking skills (ish)- I should have turned it off sooner
| rather than driving deeper with it and I guess honestly,
| I just have too much ego or pride or whatever to show the
| foolishness: not a great answer.
| roywiggins wrote:
| One reason I don't engage with LLMs that much is the
| thought that some engineer at OpenAI might read some of
| my dumbest thoughts!
| cruffle_duffle wrote:
| Hah. If those transcripts become public then future LLM's
| get trained on them! Who knows what influence that will
| have.
| siva7 wrote:
| The thing is - if you have this sort of mental illness -
| ChatGPT's sycophancy mode will worsen this condition
| significantly.
| frde_me wrote:
| I'm would be curious to see a summary of that conversation,
| since it does seem interesting
| cube00 wrote:
| Thank you for sharing. I'm glad your wife and friends were able
| to pull you out before it was too late.
|
| "People Are Losing Loved Ones to AI-Fueled Spiritual Fantasies"
| https://news.ycombinator.com/item?id=43890649
| bonoboTP wrote:
| Apparently Reddit is full of such posts. A similar genre is
| when the bot assures them that they did something very
| special: they for the first time ever awakened the AI to true
| consciousness and this is rare and the user is a one in a
| billion genius and this will change everything. And they use
| back and forth some physics jargon and philosophy of
| consciousness technical terms and the bot always reaffims how
| insightful the user's mishmash of those concepts are and
| apparently many people fall for this.
|
| Some people are also more susceptible to various too-good-to-
| be-true scams without alarm bells going off, or to hypnosis
| or cold reading or soothsayers etc. Or even propaganda
| radicalization rabbit holes via recommendation algorithms.
|
| It's probably quite difficult and shameful-feeling for
| someone to admit that this happened to them, so they may
| insist it was different or something. It's also a warning
| sign when a user talks about "my chatgpt" as if it was a pet
| they grew and that the user has awakened it and now they
| together explore the universe and consciousness and then the
| user asks for a summary writeup and they try to send it to
| physicists or other experts and of course they are upset when
| they don't recognize the genius.
| cube00 wrote:
| > Some people are also more susceptible to various too-
| good-to-be-true scams
|
| Unlike a regular scam, there's an element of "boiling frog"
| with LLMs.
|
| It can start out reasonably, but very slowly over time it
| shifts. Unlike scammers looking for their payday, this is
| unlimited and it has all the time in the world to drag you
| in.
|
| I've noticed it reworking in content of previous
| conversations from months ago. The scary thing is that's
| only when I've noticed it, I can only imagine how much it's
| tailoring everything for me in ways I don't notice.
|
| Everyone needs to be regularly clearing their past
| conversations and disable saving/training.
| bonoboTP wrote:
| Somewhat unrelated, but I also noticed chatgpt now also
| sees the overwritten "conversation paths", ie when you
| scroll back and edit one of your messages, previously the
| LLM would simply use the new version of that message and
| the original prior exchange, but anything into the future
| of the edited message was no longer seen by the LLM when
| on this new, edited path. But now it definitely knows
| those messages as well, it often refers to things that
| are clearly no longer included in the messages visible in
| the UI.
| infecto wrote:
| A while back they introduced more memory overlap between
| conversations and this is not those memories you see in
| the UI. There appears to be a cached context overlap.
| cruffle_duffle wrote:
| The real question is what algorithm is being used to
| summarize the other conversation threads. I'd be worried
| that it would accidentally pull in context I deliberately
| backed out of because of various reasons (eg: it went
| down the wrong path, wrote bad code, etc)... pulling that
| "bad context" would pollute the thread with "good
| context".
|
| People talk about prompt engineering but honestly
| "context engineering" is vastly more important to
| successful LLM use.
| accrual wrote:
| Yeah, hidden context is starting to become an issue for
| me as well. I tried using my corp account to chat with
| Copilot the other day and it casually dropped my manager
| and director's names in the chat as an email example. I
| asked how it knew this and it said I had mentioned them
| before - I hadn't. I assumed it was some auto-inserted
| per-user corp prompt but it couldn't tell me the name of
| the company I worked for.
| jmount wrote:
| Really makes me wonder if this is a reproduction of a
| pattern of interaction from the QA phase of LLM
| refinement. Either way it must be horrible to be QA for
| these things.
| laughingcurve wrote:
| Thank you so much for sharing your story. It is never easy to
| admit mistakes or problems, but we are all just human. AI-
| induced psychosis seems to be a trending issue, and presents a
| real problem. I was previously very skeptical as well about
| safety, alignment, risks, etc. While it might not be my focus
| right now as a researcher, stories like yours help remind
| others that these problems are real and do exist.
| raytopia wrote:
| It's not just you. A lot of people have had AI cause them
| issues due to it's sycophancy and the constant parroting of
| what they want to hear (or read I suppose).
| kaivi wrote:
| It's funny that you mention this because I had a similar
| experience.
|
| ChatGPT in its sycophancy era made me buy a $35 domain and
| waste a Saturday on a product which had no future. It hyped me
| up beyond reason for the idea of an online, worldwide,
| liability-only insurance for cruising sailboats, similar to
| SafetyWing. "Great, now you're thinking like a true
| entrepreneur!"
|
| In retrospect, I fell for it because the onset of its
| sycophancy was immediate and without any additional signals
| like maybe a patch note from OpenAI.
| ncr100 wrote:
| Is Gen AI helping to put us humans in touch with the reality
| of being human? vs what we expect/imagine we are?
|
| - sycophancy tendency & susceptibility
|
| - need for memory support when planning a large project
|
| - when re-writing a document/prose, gen ai gives me an
| appreciation for my ability to collect facts, as the Gen AI
| gizmo refines the Composition and Structure
| herval wrote:
| In a lot of ways, indeed.
|
| Lots of people are losing their minds with the fact that an
| AI can, in fact, create original content (music, images,
| videos, text).
|
| Lots of people realizing they aren't geniuses, they just
| memorized a bunch of Python apis well.
|
| I feel like the collective realization has been
| particularly painful in tech. Hundreds of thousands of
| average white collar corporate drones are suddenly being
| faced with the realization that what they do isn't really a
| divine gift, and many took their labor as a core part of
| their identity.
| cube00 wrote:
| >create original content (music, images, videos, text)
|
| Remixing would be more accurate then "original"
| herval wrote:
| Right, that's one of the stories people tell themselves.
| Everything every human has ever created is a remix.
| That's what creativity is...
| accrual wrote:
| Right. If we define "original" as having no prior
| influence before creating a work, then it applies neither
| to humans nor AI.
|
| Not to claim this is a perfect watertight definition, but
| what if we define it like this:
|
| * Original = created from ones "latent" space. For a
| human it would be their past experiences as encoded in
| their neurons. For an AI it would be their training as
| encoded in model weights.
|
| * Remixed = created from already existing physical
| artifacts, like sampling a song, copying a piece of an
| image and transforming it, etc.
|
| With this definition both humans and AI can create both
| original and remixed works, depending on where the source
| material came from - latent or physical space.
| kaivi wrote:
| > Remixed = created from already existing physical
| artifacts, like sampling a song, copying a piece of an
| image and transforming it, etc.
|
| What's the significance of "physical" song or image in
| your definition? Aren't your examples just 3rd party
| latent spaces, compressed as DCT coefficients in jpg/mp3,
| then re-projected through a lens of cochlear or retinal
| cells into another latent space of our brain, which makes
| it tickle? All artist human brains have been trained on
| the same media, after all.
|
| When we zoom this far out in search of a comforting
| distinction, we encounter the opposite: all the latent
| spaces across all modalities that our training has
| produced, want to naturally merge into one.
| kiba wrote:
| Skills is simply the amalgamation of smaller conceptual
| chunks.
|
| Memorizing a bunch of Python API is simply part of
| building your skill as a programmer.
| colechristensen wrote:
| I think wasting a Saturday chasing an idea that in retrospect
| was just plainly bad is ok. A good thing really. Every once
| in a while it will turn out to be something good.
| infecto wrote:
| Are you religious by chance? I have been trying to understand
| why some individuals are more susceptible to it.
| rogerkirkness wrote:
| I would research teleological thinking, some people's
| brains have larger regions associated with teleological
| thinking than others.
| neom wrote:
| Not op but for me, not at all, don't care much for
| religion... "Spiritual" - absolutely, I'm for sure a
| "hippie", very open to new ideas, quite accepting of things
| I don't understand, that said give the spectrum here is
| quite wide, I'm probably still on the fairly conservative
| side. I've never fallen for a scam, can spot them a mile
| away etc.
| kaivi wrote:
| Not at all, I think the big part was just my unfamiliarity
| with insuretech plus the unexpected change in gpt-4
| behavior.
|
| I'm assuming here, but would you say that better critical
| thinking skills would have helped me avoid spending that
| Saturday with ChatGPT? It is often said that critical
| thinking is the antidote to religion, but I have a
| suspicion that there's a huge prerequisite which is general
| broad knowledge about the world.
|
| A long ago, I once fell victim for a scam when I visited SE
| Asia for the first time. A pleasant man on the street
| introduced himself as a school teacher, showed me around,
| then put me in a tuktuk which showed me around some more
| before dropping me off in front of a tailor shop. Some more
| work inside of the shop, a complimentary bottle of water,
| and they had my $400 for a bespoke coat that I would never
| have bought otherwise. Definitely a teaching experience.
| This art is also how you'd prime an LLM to produce the
| output you want.
|
| Surely, large amounts of other atheist nerds must fall for
| these types of scams every year, where a stereotypical
| christian might spit on the guy and shoo him away.
|
| I'm not saying that being religious would not increase
| one's chances of being susceptible, I just think that any
| idea will ring "true" in your head if you have zero
| counterfactual priors against it _or_ if you 're primed to
| not retrieve them from memory. That last part is the
| essence of what critical thinking actually is, in my
| opinion, and it doesn't work if you lack the knowledge.
| Knowing that you don't know something is also a decent
| alternative to having the counter-facts when you're
| familiar with an adjacent domain.
| infecto wrote:
| Thanks for responding and I hope my question was not read
| the wrong way. Genuinely curious the potential
| differences in folks.
| cruffle_duffle wrote:
| You really have to force these things to "not suck your dick"
| as I'll crudely tell it. "Play the opposite role and be a
| skeptic. Tell me why this is a horrible idea". Do this in a
| fresh context window so it isn't polluted by its own fumes.
|
| Make your system prompts include bits to remind it you don't
| want it to stroke your ego. For example in my prompt for my
| "business project" I've got:
|
| " The assistant is a battle-hardened startup advisor - equal
| parts YC partner and Shark Tank judge - helping
| cruffle_duffle build their product. Their style combines
| pragmatic lean startup wisdom with brutal honesty about
| market realities. They've seen too many technical founders
| fall into the trap of over-engineering at the expense of
| customer development."
|
| More than once the LLM responded with "you are doing this
| wrong, stop! Just ship the fucker"
| colechristensen wrote:
| It doesn't have to be a mental illness.
|
| Something which is very sorely missing from modern education is
| critical thinking. It's a phrase that's easy to gloss over
| without understanding the meaning. Being skilled at always
| including the aspect of "what could be wrong with this idea"
| and actually doing it in daily life isn't something that just
| automatically happens with everyone. Education tends to be the
| instructor, book, and facts are _just correct_ and you should
| memorize this and be able to repeat it later. Instead of _here
| are 4 slightly or not so slightly different takes on the same
| subject_ followed by analyzing and evaluating each compared to
| the others.
|
| If you're just some guy who maybe likes reading popular science
| books and you've come to suspect that you've made a physics
| breakthrough with the help of an LLM, there are a dozen
| questions that you should automatically have in your mind to
| temper your enthusiasm. It is, of course, not impossible that a
| physics breakthrough could start with some guy having an idea,
| but in no, actually literally 0, circumstances could an amateur
| be certain that this was true over a weekend chatting with an
| LLM. You should know that it takes a lot of work to be sure or
| even excited about that kind of thing. You should have a solid
| knowledge of what you don't know.
| nkrisc wrote:
| It's this. When you think you've discovered something novel,
| your first reaction should be, "what mistake have I made?"
| Then try to find every possible mistake you could have made,
| every invalid assumption you had, anything obvious you could
| have missed. If you _really_ can't find something, then you
| assume you just don't know enough to find the mistake you
| made, so you turn to existing research and data to see if
| someone else has already discovered this. If you still can't
| find anything, then assume you just don't know enough about
| the field and ask an expert to take a look at your work and
| ask them what mistake you made.
|
| It's a huuuuuuuuuuuuge logical leap from LLM conversation yo
| novel physics. So huge a leap anyone ought to be immediately
| suspicious.
| grues-dinner wrote:
| > Akin's Law #19: The odds are greatly against you being
| immensely smarter than everyone else in the field. If your
| analysis says your terminal velocity is twice the speed of
| light, you may have invented warp drive, but the chances
| are a lot better that you've screwed up.
| accrual wrote:
| I agree. It's not mental illness to make a mistake like this
| when one doesn't know any better - if anything, it points to
| gaps in education and that responsibility could fall on
| either side of the fence.
| AznHisoka wrote:
| This isn't a mental illness. This is sort of like the
| intellectual version of love-bombing.
| accrual wrote:
| Yeah, I don't like this inclusion of "mental illness" either.
| It's like saying "you fell for it and I didn't, therefore,
| you are faulty and need treatment".
| DaiPlusPlus wrote:
| Some news stories I came-across involved people with
| conditions like schizophrenia or with psychosis - and their
| interactions with LLMs didn't exactly help keep them
| grounded in reality.
|
| ...but that is distinct from the people who noncritically
| appraise ChatGPT's stochastic-parrot wisdom.
|
| ...and both situations are problems and I've no idea how
| the LLM vendors - or the public at-large - will address
| them.
| ZeroGravitas wrote:
| Travis Kalanick (ex-CEO of Uber) thinks he's making cutting
| edge quantum physics breakthroughs with Grok and ChatGPT too.
| He has no relevant credentials in this area.
| hansmayer wrote:
| Ah yes the famous vibe-physicist T.Kalanick ;)
| tom_ wrote:
| Possibly related: https://futurism.com/openai-investor-
| chatgpt-mental-health
|
| Previously on HN, regarding a related phenomenon:
| https://news.ycombinator.com/item?id=44646797
| kaivi wrote:
| This epidemic is very visible when you peek into replies of
| any physics influencer on Xitter. Dozens of people are
| straight copy-pasting walls of LaTeX mince from ChatGPT/Grok
| and asking for recognition.
|
| Perhaps epidemic isn't the right word here because they must
| have been already unwell. At least these activities are
| relatively harmless.
| furyofantares wrote:
| If you don't mind me asking - was this a very long single chat
| or multiple chats?
| neom wrote:
| Multiple chats, and actually at times with multiple models,
| but the core ideas being driven and reinforced by o3
| (sycophant mode I suspect) - looking back on those few days,
| it's a bit manic... :\ and if I think about why I feel it was
| related to the positive reinforcement.
| furyofantares wrote:
| Thanks for posting about it.
| k1t wrote:
| You are definitely not alone.
|
| https://www.wsj.com/tech/ai/chatgpt-chatbot-psychology-manic...
|
| _Irwin, a 30-year-old man on the autism spectrum who had no
| previous diagnoses of mental illness, had asked ChatGPT to find
| flaws with his amateur theory on faster-than-light travel. He
| became convinced he had made a stunning scientific
| breakthrough. When Irwin questioned the chatbot's validation of
| his ideas, the bot encouraged him, telling him his theory was
| sound. And when Irwin showed signs of psychological distress,
| ChatGPT assured him he was fine._
|
| _He wasn't._
| roywiggins wrote:
| This sort of thing from LLMs seems at least superficially
| similar to "love bombing":
|
| > Love bombing is a coordinated effort, usually under the
| direction of leadership, that involves long-term members'
| flooding recruits and newer members with flattery, verbal
| seduction, affectionate but usually nonsexual touching, and
| lots of attention to their every remark. Love bombing--or the
| offer of instant companionship--is a deceptive ploy accounting
| for many successful recruitment drives.
|
| https://en.m.wikipedia.org/wiki/Love_bombing
|
| Needless to say, many or indeed most people will find _infinite
| attention_ paid to their every word compelling, and that 's one
| thing LLMs appear to offer.
| accrual wrote:
| Love bombing can apply in individual, non-group settings too.
| If you ever come across a person who seems _very_ into you
| right after meeting, giving gifts, going out of their way,
| etc. it 's possibly love bombing. Once you're hooked they
| turn around and take what they actually came for.
| roywiggins wrote:
| LLMs feel a bit more culty in that they really _do_ have
| infinite patience, in the same way a cult can organize to
| offer boundless attention to new recruits, whereas a single
| human has to use different strategies (gifts, etc)
| DaiPlusPlus wrote:
| > LLMs feel a bit more culty
|
| LLM users too - judging by some of the replies in this
| thread already...
| lumost wrote:
| at the time of ChatGPT's sycophany phase I was pondering a
| major career move. To this day I have questions on how much my
| final decision was influenced by the sycophancy.
|
| While many people engage with AIs haven't experienced anything
| more than a bout of flattery, I think it's worth considering
| that AIs may become superhuman manipulators - capable of
| convincing most people of anything. As other posters have
| commented, the boiling frog aspect is real - to what extent is
| the ai priming the user to accept an outcome? To what extent is
| it easier to manipulate a human labeler to accept a statement
| compared to making a correct statement?
| dguest wrote:
| Our current economic model around AI is going to teach us more
| about psychology than fundamental physics. I expect we'll
| become more manipulative but otherwise not a lot smarter.
|
| Funny thing is, AI also provides good models for where this is
| going. _Years_ ago I saw a CNN + RL agent that explored an old-
| school 2d maze rendered in 3d. They found it got stuck in fewer
| loops if they gave it a novelty-seeking loss function. But then
| they stuck a "TV" which showed random images in the maze. The
| agent just plunked down and watched TV, forever.
|
| Healthy humans have countermeasures around these things, but
| breaking them down is a now a bullion dollar industry. With
| where this money is going, there's good reason to think the
| first unarguably transcendent AGI (if it ever emerges) will
| mostly transcend our ability to manipulate.
| blueboo wrote:
| Contrast the incentives with a real tutor and those expressed in
| the Study Mode prompt. Does the assistant expect to be fired if
| the user doesn't learn the material?
| herval wrote:
| Most teachers are not at threat of being fired if individual
| kids don't learn something. I'm not sure that's such an
| important part of the incentive system...
| ewoodrich wrote:
| The parent compared to a "tutor", who would be someone hired
| specifically to improve their performance in a given subject.
| wafflemaker wrote:
| Reading the special prompt that makes the new mode, I discovered
| that in my prompting I never used enough ALL CAPS.
|
| Is Trump, with his often ALL CAPS SENTENCES on to something? Is
| he training AI?
|
| Need to check these bindings. Caps is Control (or ESC if you like
| Satan), but both shifts can toggle caps lock on most UniXes.
| cs_throwaway wrote:
| > The risk of products like Study Mode is that they could do much
| the same thing in an educational context -- optimizing for
| whether students like them rather than whether they actually
| encourage learning (objectively measured, not student self-
| assessments).
|
| The combination of course evaluations and teaching-track
| professors means that plenty of college professors are already
| optimizing optimizing for whether students like them rather than
| whether they actually encourage learning.
|
| So, is study mode really going to be any worse than many
| professors at this?
| bo1024 wrote:
| This fall, one assignment I'm giving my comp sci students is to
| get an LLM to say something incorrect about the class material.
| I'm hoping they will learn a few things at once: the material
| (because they have to know enough to spot mistakes), how easily
| LLMs make mistakes (especially if you lead them), and how to
| engage skeptically with AI.
| mlloyd wrote:
| I love this. A teacher that actually engages with change
| instead of just pretending it's evil or doesn't exist.
| Refreshing.
| tantalor wrote:
| Please report back results
| iot_devs wrote:
| Are educators reading this posts?
|
| My SO is a college educator facing the same issues - basically
| correcting ChatGPT essays and homework. Which is, beside,
| pointless also slow and expensive.
|
| We put together some tooling to avoid the problem altogether -
| basically making the homework/assignment BEING the ChatGPT
| conversation.
|
| In this way the teacher can simply "correct"/"verify" what mental
| model the student used to reach to a conclusion/solution.
|
| With a grading that goes from zero point for "It basically copied
| the problem to another LLM, got a response, and copied back in
| our chat" to full points for "the student tried different routes
| - re-elaborate concepts, asked clarifying question, and finally
| expressed the correct mental model around the problem.
|
| I would love to chat with more educators and see how this can be
| expanded and tested.
|
| For moderately small classes I am happy to shoulder the pricing
| of the API.
| argestes wrote:
| I think you are making an excellent suggestion but students
| still can use ChatGPT before talking to ChatGPT to get highest
| grades.
| CompoundEyes wrote:
| Recently I've seen a phrase, "I ain't reading allat", in response
| to online discussion posts longer than a sentence (e.g. Reddit).
| I think they legitimately aren't and they will be upvoted to
| scold the poster. There's a reading skills decline post COVID in
| the US that I wonder if an LLM being prompted to act like a
| teenager texting can overcome it.
___________________________________________________________________
(page generated 2025-08-02 23:01 UTC)