[HN Gopher] AI overly affirms users asking for personal advice
___________________________________________________________________
AI overly affirms users asking for personal advice
https://arxiv.org/abs/2602.14270
https://www.science.org/doi/10.1126/science.aec8352
Author : oldfrenchfries
Score : 478 points
Date : 2026-03-28 14:08 UTC (8 hours ago)
(HTM) web link (news.stanford.edu)
(TXT) w3m dump (news.stanford.edu)
| oldfrenchfries wrote:
| This new Stanford study published on March 26, 2026 shows that AI
| models are sycophantic. They affirm the users position 49% more
| often than a human would.
|
| The researchers found that when people use AI for relationship
| advice, they become 25% more convinced they are 'right' and
| significantly less likely to apologize or repair the connection.
| jatins wrote:
| To be fair an average therapist is also pretty sycophantic.
| "The worst person you know is being told by their therapist
| that they did the right thing" is a bit of a meme, but isn't
| completely false in my experience.
| kibwen wrote:
| No, the meme is that the average therapist can be boiled down
| to "well, what do you think?" or "and how does that make you
| feel?" (of which ELIZA, the original bot that passed the
| Turing test, was perhaps an unintentional parody). Even this
| cartoonish characterization demonstrates that the function of
| therapists is to get you to question yourself so that you can
| attempt to reframe and re-evaluate your ways of thinking, in
| a roughly Socratic fashion.
| toraway wrote:
| It was entirely intentional. The Rogerian school of
| psychotherapy stereotyped by "how does that make you feel"
| was popular at the time and the most popular ELIZA script
| used that persona to cleverly redirect focus from the bot's
| weaknesses in comprehension.
| oldfrenchfries wrote:
| There is a striking data visualization showing the breakup advice
| trend over 15 years on Reddit. You can see the "End relationship"
| line spike as AI and algorithmic advice take over:
|
| https://www.reddit.com/r/dataisbeautiful/comments/1o87cy4/oc...
| falcor84 wrote:
| Isn't the fact that a person is asking an AI whether to leave
| their partner in its own an indication that they should?
|
| EDIT: typo
| nomorewords wrote:
| How is it an indication? I think people on here don't realize
| that most of the people don't think things through as much as
| (software) engineers
| hnfong wrote:
| In my local(?) community (like in my city, not my industry)
| there is a saying "if you had to ask for relationship
| advice, then you probably should break up".
|
| There is some rationale to that. People tend to hold onto
| relationships that don't lead anywhere in fear of "losing"
| what they "already have". It's probably a comfort zone
| thing. So if one is desperate enough to ask random
| strangers online about a relationship, it's usually biased
| towards some unresolvable issue that would have the parties
| better of if they break up.
| magicalhippo wrote:
| > So if one is desperate enough to ask random strangers
| online about a relationship
|
| I'd me more inclined to ask random strangers on the
| internet than close friends...
|
| That said, when me and my SO had a difficult time we went
| to a professional. For us it helped a lot. Though as the
| counselor said, we were one of the few couples which came
| early enough. Usually she saw couples well past the point
| of no return.
|
| So yeah, if you don't ask in time, you will probably be
| breaking up anyway.
| hnfong wrote:
| I would speculate that, if a couple goes to a
| professional for help, they have much better chances than
| asking on a random forum online...
| otabdeveloper4 wrote:
| > relationships that don't lead anywhere
|
| Relationships are not transactions that are supposed to
| "lead somewhere".
| ambicapter wrote:
| You're being a bit pedantic here "leading somewhere" is
| accepted shorthand for a lasting, satisfying relationship
| that is good for both parties.
| SpicyLemonZest wrote:
| Most people engage in romantic relationships because
| they'd like to find someone to marry and settle down
| with. Nothing but respect for the people who've thought
| it through and decided that's not for them, but what's
| much more common is failing to think it through or
| worrying it would be awkward/scary/"cringe" to take their
| relationship goals seriously.
|
| That's what people are pointing to when they talk about
| relationships not "leading anywhere". If you want to be
| married in 5-10 years, and you're 2 years into an OK
| relationship with someone you don't want to marry, it's
| going to suck to break up with them but you have to do it
| anyway.
| rusty_venture wrote:
| Wait, other people don't make decision trees and mind maps
| and pro/con lists and consult chatbots before making
| decisions? Are they just flying through life by the seat of
| their pants? That doesn't seem like a very solid framework
| for achieving desired outcomes.
| nprateem wrote:
| I heard about someone once who could decide whether to
| buy a new t-shirt in less than 3 months.
| falcor84 wrote:
| Maybe I'm too much of a hopeless romantic, but from my
| perspective and experience, when someone is good for you,
| you'll fight for that relationship regardless of what
| others say, and conversely when you're in a situation where
| your actively asking and willing to consider "leave" from
| someone who isn't a very close friend or a therapist as
| applicable, then it's likely you're looking for external
| validation for what you've already essentially decided.
| duskdozer wrote:
| >asking an AI whether to leave your partner
|
| is that what they're asking though? because "relationship
| advice" is pretty vague
| falcor84 wrote:
| That's a good point. If an AI respond to a "what should I
| get my boyfriend for Christmas?" with a "You should leave
| him", that's a very different issue.
| oldfrenchfries wrote:
| The idea that asking implies a yes is actually a pretty
| common logical fallacy. In relationship science, we call this
| "Relational Ambivalence" and its a completely normal part of
| any longterm commitment.
| dinkumthinkum wrote:
| No, but it is an indication of brain-rot to make a question
| seriously and also to think that it means the conclusion is
| foregone. It is an advent of our childlike current
| generations. Of course, the moment anything becomes difficult
| or unpleasant, one should quit, apparently. Surely, this kind
| of resiliency is what got humanity so far.
| falcor84 wrote:
| I didn't imply it's a "foregone conclusion", but just said
| it's an indication - in the sense of increasing the
| likelihood. Just like a person asking an AI "what does it
| feel like to bleed out?" could be them researching for a
| novel, but is nevertheless an indication of a potential
| serious issue.
| Sharlin wrote:
| More interesting, IMO, is the general trend that started long
| before LLMs. The fact that "dump them" is the standard answer
| to any relationship question is a meme by now. The LLMs appear
| to be doing exactly what one would expect them to be doing
| based on their training corpus.
| 1970-01-01 wrote:
| This is the correct take. The advice preceded the LLM boom.
| They were trained on the 'dump them' advice and proceeded to
| reinforce the take. So why did the relationship advice change
| dramatically? I speculate attribution to the disinformation
| campaigns during this time. They were and still are grossly
| underestimated.
| to11mtm wrote:
| Not sure what sorts of disinformation campaigns you're
| referring to...
|
| There is something more interesting to consider however;
| the graph starts to go up in 2013, less than 6 months after
| the release of Tinder.
| 1970-01-01 wrote:
| These. https://en.wikipedia.org/wiki/Russian_disinformati
| on#Social_...
| doubled112 wrote:
| "There is more than one fish in the sea" has been
| relationship advice for centuries. It might be about being
| dumped, but I've also thought it useful for considering
| dumping somebody too.
| Sharlin wrote:
| No, that's not it. We're talking about posts like "we had a
| silly little quarrel about something that would need
| fifteen minutes to clear up and make both happy if we both
| just try to adult a bit" and commenters being adamant that
| deleting gym and facebooking up and so on is clearly the
| only choice. Most of said commenters probably not being in
| any position to give advice on relationships to others.
| dec0dedab0de wrote:
| if things are so bad that you're posting on reddit then
| breaking up is usually the best answer.
| nibbleyou wrote:
| I see this being said often but I don't understand.
|
| A lot of people posting there are young and may well be in
| their first relationship. It makes sense for them to ask a
| question in the community they spend their most time in -
| which is reddit
| the_af wrote:
| Most people overshare on reddit and it's completely
| unrelated to the seriousness of the situation.
|
| It's also a meme that people will ask the dumbest, most
| trivial interpersonal conflict questions on Reddit that
| would be easily solved by just talking to the other person.
| E.g. on r/boardgames, "I don't like to play boardgames but
| my spouse loves them, what can I do?" or "someone listens
| to music while playing but I find it distracting, what can
| I do?" (The obvious answer of "talk to the other person and
| solve it like grownups" is apparently never considered).
|
| On relationship advice, it often takes the form "my
| boy/girlfriend said something mean to me, what shall I do?"
| (it's a meme now that the answer is often "dump them").
|
| If LLMs train _on this_...
| est wrote:
| the year is 2015
|
| smart phones took over the world, social networks happened.
|
| Turns out they are the best sterializer human ever invented.
|
| I just wrote a blog https://blog.est.im/2026/stdin-09
| astrange wrote:
| > The LLMs appear to be doing exactly what one would expect
| them to be doing based on their training corpus.
|
| That is not how full LLM training works. That is how base
| model pretraining works.
| jubilanti wrote:
| Or that people are using AI to write perfectly calibrated
| ragebait that gets upvoted with a bunch of genuine human
| clicks.
| raincole wrote:
| Is this comment human hallucination? You can clearly see the
| trend is always going up. It only went down a bit during Covid.
| deeg wrote:
| I do find them cloying at times. I was using Gemini to iterate
| over a script and every time I asked it to make a change it
| started a bunch of responses with "that's a smart final step for
| this task! ...".
| xiphias2 wrote:
| Marc Andereseen has talked about the downside of RLHF: it's a
| specific group of liberal low income people in California who did
| the rating, so AI has been leaning their culture.
|
| I think OpenAI tried to diversify at least the location of the
| raters somewhat, but it's hard to diversify on every level.
| michaelcampbell wrote:
| Do you have any links to documentation of this? Andreesen has a
| definite bias as well, so I'm not about to just accept his say-
| so in a fit of Appeal to Authority.
|
| (eg: "Cite?")
| xiphias2 wrote:
| He was talking about it in the Lex Friedman interview after
| Trump was elected. And he was talking about a lot of things
| the Biden administration forced on Silicon Valley at that
| time (since then Google lost a case about one of these back-
| deals).
| sph wrote:
| What do low income people have to do with it, when AI companies
| and research is borne out of Silicon Valley culture of rich,
| liberal Californians?
|
| I'm still waiting for models based on the curt and abrasive
| stereotype of Eastern European programmers, as contrast to the
| sickeningly cheerful AIs we have today that couldn't sound more
| West Coast if they tried.
| fourside wrote:
| Low income and liberal is usually code for certain
| "undesirables" that conservatives tend to dislike. Better
| watch what LLM your kids use or they might end up speaking
| Spanish and listening to rap ;).
| dinkumthinkum wrote:
| Eh, or grow up hating American and thinking they need to
| fly to Cuba to explain to the people are great communism is
| for them. Who knows.
| xiphias2 wrote:
| It's not about liking / disliking, but conservatives tend
| to prefer staying together even if it's a bad relatioship,
| and liberals prefer splitting by default if there are
| serious problems.
|
| The syncopath style is clearly categorized as more liberal
| (do what you feel is good).
| tbrownaw wrote:
| > _What do low income people have to do with it, when AI
| companies and research is borne out of Silicon Valley culture
| of rich, liberal Californians?_
|
| RLHF is "ask a human to score lots of LLM answers". So the
| claim is that the AI companies are hiring cheap (~poor)
| people from convenient locations (CA, since that's where the
| rest of the company is).
| astrange wrote:
| "Poor" in California means earning $80k/year, so they
| probably are not doing that. Africa / Indonesia /
| Philippines are better places to find English speaking RLHF
| workers.
| sublinear wrote:
| Yes, this precisely it. There _isn 't going to be_ hard
| evidence to prove it though. Survey data that underpins
| some empirical studies have similar transparency issues
| too. This is far from a new problem.
|
| If you adjust your mindset slightly when searching online,
| it's not hard to find communities of people looking for
| quick side work and this was huge during the covid lockdown
| era. There were people helping train LLMs for all kinds of
| purposes from education to customer service. Those startups
| quickly cashed out a few years ago and sold to the big
| players we have now.
|
| I don't get why this is hard for people to believe (or
| remember)?
| cyanydeez wrote:
| Poor people, to the billionaire, clearly are morally and
| ethically unsound.
|
| https://pmc.ncbi.nlm.nih.gov/articles/PMC9533286/
| nirvdrum wrote:
| For anyone else unfamiliar with the term:
|
| RLHF = Reinforcement Learning from Human Feedback
|
| https://en.wikipedia.org/wiki/Reinforcement_learning_from_hu...
| mvkel wrote:
| Marc Andreesen should get HF on his own RL, because he's
| completely wrong.
|
| This sounds like something Elon would say to make Grok seem
| "totally more amazeballs," except "anti-woke" Grok suffers from
| the same behavior
| ej88 wrote:
| huh? this is completely inaccurate
| kibwen wrote:
| You're absolutely right!
| BoredPositron wrote:
| Talked about as in lied about it and you taking his words for
| gospel without verifying it? Looks just as bad as "Yes-Men" AI
| models.
| masteranza wrote:
| We can surely fix it and we probably should. However, I don't
| think AI is doing any worse here than friends advice when they
| here a one sided story. The only difference being that it's not
| getting studied.
|
| Conversely, AI chatbots are great mediators if both parties are
| present in the conversation.
| tom-blk wrote:
| Not surprising, but nice that we have actual data now
| 152334H wrote:
| Maybe it's not so sensible to offload the responsibility of clear
| thinking to AI companies?
|
| How is a chatbot supposed to determine when a user fools even
| themselves about what they have experienced?
|
| What 'tough love' can be given to one who, having been so
| unreasonable throughout their lives - as to always invite scorn
| and retort from all humans alike - is happy to interpret
| engagement at all as a sign of approval?
| isodev wrote:
| > clear thinking
|
| Most humans working in tech lack this particular attribute, let
| alone tools driven by token-similarity (and not actual
| 'thinking').
| kibwen wrote:
| _> Maybe it 's not so sensible to offload the responsibility of
| clear thinking to AI companies?_
|
| Markets don't optimize for what is sensible, they optimize for
| what is profitable.
| SlinkyOnStairs wrote:
| It's not market driven. AI is ludicrously unprofitable for
| nearly all involved.
| cyanydeez wrote:
| The profit appears to be capturing the political class and
| it's associated lobbies and monied interests.
| expedition32 wrote:
| It's almost as if being a therapist is an actual job that takes
| years of training and experience!
|
| AI may one day rewrite Windows but it will never be counselor
| Troi.
| yarn_ wrote:
| Come on, I'm sure Dario can find a nice tight bodysuit for
| claude
| duskdozer wrote:
| Well, unless insurance companies figure out they can make
| more money by pushing everyone onto AI [step-]therapy instead
| of actual therapy
| fsmv wrote:
| Implying that programming is not an actual job that takes
| years of training and experience
|
| To be clear I don't think the AI can do either job
| rsynnott wrote:
| > How is a chatbot supposed to determine when a user fools even
| themselves about what they have experienced?
|
| And even if it _could_, note, from the article:
|
| > Overall, the participants deemed sycophantic responses more
| trustworthy and indicated they were more likely to return to
| the sycophant AI for similar questions, the researchers found.
|
| The vendors have a perverse incentive here; even if they
| _could_ fix it, they'd lose money by doing so.
| sublinear wrote:
| I think if you're at the stage of life where you even need to
| ask, the AI might be doing everyone a favor.
|
| As much as people whine about the birth rate and whatever else, I
| think it's a net good that people spend a lot more time alone to
| mature. Good relationships are underappreciated.
| graemep wrote:
| There are plenty of sycophantic humans around, especially with
| regard to relationship advice.
|
| I find there is an inverse relationship between how willing
| people are to give relationship advice, and how good their advice
| is (whether looking at sycophancy or other factors).
| griffzhowl wrote:
| Because sycophancy in humans is motivated not by the wellbeing
| of the person seeking advice, but by the interests of the
| sycophant in gaining favour.
|
| It makes sense that this behaviour would be seen in LLMs, where
| the company optimizes towards of success of the chatbot rather
| than wellbeing of the users.
| xhkkffbf wrote:
| Yup. I know too many people who have a default message when
| asked for relationship advice: oh, my, the other person is
| terrible and you should break up.
|
| It's an easy default and it causes so many problems.
| megous wrote:
| Can't you just prompt for a critical take, multiple alternative
| perspectives (specifically not yours, after describing your own),
| etc.?
|
| It's a tool, I can bang my hand on purpose with a hammer, too.
| ranger_danger wrote:
| Yes, if you're smart. But most people asking it random
| questions and expecting it to read their minds and spit out the
| perfect answer are not so much. They don't know what a prompt
| is, and wouldn't be bothered to give it prior instructions
| either way.
| megous wrote:
| Educated, not smart. This is a job for schools to include AI
| education into the basic curricula. Their pupils will use the
| tools anyway, so at least teach them to do it with proper
| expectations and prompting techniques/pitfalls.
| joquarky wrote:
| I think that the type of people who can easily pick up
| subtext have come to rely on that channel of communication
| and don't realize they need to be more direct and verbose
| when chatting with language models.
| awithrow wrote:
| It feels like I'm fighting uphill battle when it comes to
| bouncing ideas off of a model. I'll set things up in the context
| with instructions similar to. "Help me refine my ideas,
| challenge, push back, and don't just be agreeable." It works for
| a bit but eventually the conversation creeps back into
| complacency and syncophancy. I'll check it too by asking "are you
| just placating me?" the funny thing is that often it'll admit
| that, yes, it wasn't being very critical, and then procede to
| over correct and become a complete contrarian. and not in a way
| that's useful either. very frustrating. I've found that Opus 4.6
| is worse about this than 4.5. 4.5 does a better job IMO of
| following instructions and not drifting into the mode where it
| acts like everything i say is a grand revelation from up high.
| righthand wrote:
| That's because the model isn't actually thinking, pushing back,
| and challenging your ideas. It's just statistically agreeing
| with you until it reaches too wide of a context. You're living
| in the delusion that it's "working" or having a "conversation"
| with you.
| alehlopeh wrote:
| How is conceptualizing what the model is doing as having a
| conversation any different from any other abstraction? "No,
| the browser isn't downloading a file. The electrons in the
| silicon are actually..."
| colechristensen wrote:
| There are people with a philosophical objection to using
| everyday words to describe LLM interactions for various
| reasons, but commonly because they're worried stupid people
| will confuse the LLM for a person. Which, I suppose stupid
| people will do that, but I'm not inventing a parallel
| language or putting a * next to each thing which means
| "this, but with an LLM instead of a person"
| cyanydeez wrote:
| So, there's things you're fighting against when trying to
| constrain the behavior of the llm.
|
| First, those beginning instructions are being quickly ignored
| as the longer context changes the probabilities. After every
| round, it get pushed into whatever context you drive towards.
| The fix is chopping out that context and providing it before
| each new round. something like `<rules><question><answer>` ->
| `<question><answer><rules><question>`.
|
| This would always preface your question with your prefered
| rules and remove those rules from the end of the context.
|
| The reason why this isn't done is because it poisons the KV
| cache, and doing that causes the cloud companies to spin up
| more inference.
| margalabargala wrote:
| Considering 4.6 came with a ton of changes around tooling and
| prompting this isn't terribly surprising.
| magicalhippo wrote:
| Gemini seems to be fairly good at keeping the custom
| instructions in mind. In mine I've told it to not assume my
| ideas are good and provide critique where appropriate. And I
| find it does that fairly well.
| steve_adams_86 wrote:
| Same. This works fine for Claude in my experience. My user
| prompt is fairly large and encourages certain behaviours I
| want to see, which involves being critical and considering
| the strengths and weaknesses of ideas before drawing
| conclusions. As someone else mentioned, there does seem to be
| a phenomenon where saying DO NOT DO X causes a sort of
| attention bias on X which can lead to X occurring despite the
| clear instructions. I've never empirically tested that, I've
| just noticed better results over the years when telling it
| what paths to stick to rather than specific things not do to.
| koverstreet wrote:
| That happens with humans too :) It's why positive feedback
| that draws attention to the behavior you want to encourage
| often works better. "Attention" is lower level and more
| fundamental than reasoning by syllogism.
| lelanthran wrote:
| > Gemini seems to be fairly good at keeping the custom
| instructions in mind.
|
| Unless those instructions are "stop providing links to you
| for every question ".
| iugtmkbdfil834 wrote:
| I will admit that I was very pleasantly surprised by gemini
| lately. I was away from my PC and tried it on a whim for a
| semi-random consumer question that led into smaller rabbit
| hole. It seemed helpful enough and focused on what I tried to
| get while still pushing back when my 'solutions' seemed out
| of whack.
| Loughla wrote:
| That's because you need actual logic and thought to be able to
| decide when to be critical and when to agree.
|
| Chatbots can't do that. They can only predict what comes next
| statistically. So, I guess you're asking if the average
| Internet comment agrees with you or not.
|
| I'm not sure there's much value there. Chatbots are good at
| tasks (make this pdf an accessible word document or sort the
| data by x), not decision making.
| kvirani wrote:
| I'm not convinced that "actual logic and thought" aren't just
| about inferring what comes next statistically based on
| experience.
| plagiarist wrote:
| Then the machines still need a more sophisticated
| "experience" compared to what they have currently.
| righthand wrote:
| Communicating is usually about inferring. I dont think
| token to token. And I don't think "well statistically I
| could say 'and' next but I will say 'also' instead to give
| my speech some flash". If I decided on swapping a word I
| would have made my decision long ago, not in the moment.
| Thought and logic are not me pouring through my brain
| finding a statistical path to any answer. Often I stop and
| say "I dont know".
| Swizec wrote:
| > I'm not convinced that "actual logic and thought" aren't
| just about inferring what comes next statistically based on
| experience.
|
| Often they are the exact opposite. Entire fields of math
| and science talk about this. Causation vs correlation,
| confirmation bias, base rate fallacy, bayesian reasoning,
| sharp shooter fallacy, etc.
|
| All of those were developed because "inferring from
| experience" leads you to the wrong conclusion.
| theptip wrote:
| Bayesian reasoning is just another algorithm for
| predicting from experience (aka your prior).
|
| I took the GP to be making a general point about the
| power of "next x prediction" rather than the algorithm a
| human would run when you say they are "inferring from
| experience". (I may be assuming my own beliefs of
| course.)
|
| Eg even LeCun's rejection of LLMs to build world models
| is still running a predictor, just in latent space (so
| predicting next world-state, instead of next-token).
|
| And of course, under the Predictive Processing model
| there is a comprehensive explanation of human cognition
| as hierarchical predictors. So it's a plausible general
| model.
| Swizec wrote:
| > under the Predictive Processing model there is a
| comprehensive explanation of human cognition as
| hierarchical predictors
|
| It's plausible!
|
| But keep in mind humans have been explaining ourselves in
| terms of the current most advanced technology for
| centuries. We used to be kinda like clockwork, then a bit
| like a steam engine, then a lot like computers, and now
| we're just like AI.
|
| That's why you blow a gasket or fuse, release some steam,
| reboot your life, do brain dump, feel like a cog in the
| machine, get your wires crossed, etc
| theptip wrote:
| Exactly. Lots can be explained just with more abstract
| predictors, plus some mechanisms for stochastic rollout and
| memory.
| dinkumthinkum wrote:
| Is this just Internet smart contrarianism or a real thing?
| Are logic gates in a digital circuit just behaving
| statistically according to their experience?
| hluska wrote:
| You know, you might really enjoy consumer behaviour. When
| you get into the depths of it, you'll end up running
| straight into that idea like you're doing a 100 metre dash
| in a 90 metre gym. It's quite interesting how arguably the
| best funded group under the psychology umbrella runs
| directly into this. One of my favourite examples is how
| heuristics will lead otherwise reasonable people to make
| decisions that are not in their interest.
| righthand wrote:
| I said this pretty much and got major downvotes...
| dTal wrote:
| Because it's an outmoded cliche that never held much
| philosophical weight to begin with and doesn't advance the
| discussion usefully. "It's a stochastic parrot" is not a
| useful predictor of actual LLM capabilities and never was.
| Last year someone posted on HN a log of GPT-5 reverse
| engineering some tricky assembly code, a challenge set by
| another commentator as an example of "something LLMs could
| never do". But here we are a year later still wading
| through people who cannot accept that LLMs can, in a
| meaningful sense, "compute".
| righthand wrote:
| It's entirely useful discussion because as soon as you
| forget that it's not really having a conversation with
| you, it's a deep dive into delusion that you're talking
| to a smart robot and ignoring the fact that these smart
| robots were trained on a pile of mostly garbage. When I
| have a conversation with another human, I'm not expecting
| them to brute force an answer to the topic. As soon as
| you forget that Llms are just brute forcing token by
| token then people start living in fantasy land. The whole
| "it's not a stochastic parrot" is just "you're holding it
| wrong".
| layla5alive wrote:
| Its not that LLMs are stochastic parrots and humans are
| not. Its that many humans often sail through
| conversations stochastic parroting because they're
| mentally tired and "phoning it in" - so there are times
| when talking to the LLM, which has a higher level of
| knowledge, feels more fruitful on a topic than talking to
| a human who doesn't have the bandwidth to give you their
| full attention, and also lack the depth and breadth of
| knowledge. I can go deep on many topics with LLMs that
| most humans can't or won't keep up on. In the end, I'm
| really only talking to myself most of the time in either
| case, but the LLM is a more capable echo, and it doesn't
| tire of talking about any topic - it can dive deep into
| complex details, and catching its hallucinations is an
| exercise in itself.
| dinkumthinkum wrote:
| No. It's quite a useful thing to understand So, what, you
| have us believe it is a sentient, thinking, kind of
| digital organism and you would have us not believe that
| it is exactly what it is? Being wrong and being
| unimaginative about what can be achieved with such a
| "parrot" is not the same as being wrong about it be a
| word predictor. If you don't think, you can probably ask
| an LLM and it will even "admit" this fact. I do agree
| that it has become considered to be outmoded to question
| anything about the current AI Orthodox.
| plagiarist wrote:
| People are upset hearing that LLMs aren't sentient for some
| reason. Expect to be downvoted, it is okay.
| gjm11 wrote:
| First off, "not adequately described as a mere token-
| predictor" and "not sentient" are entirely separate
| things.
|
| I can't speak for anyone else, but what _I_ feel when I
| read yet another glib "it's just a stochastic parrot, of
| course it isn't doing anything that deserves to be called
| reasoning" take is much more like _bored_ than it is like
| _upset_.
|
| Today's LLMs are in some sense "just predicting tokens"
| in some sense. Likewise, human brains are in some sense
| "just shuttling neurotransmitters and electrical impulses
| around" in some sense. Neither of those tells you what
| the thing can actually do. To figure that out, you have
| to _look at what it can do_.
|
| Today's best LLMs can do about as well as the best humans
| on problems from the International Mathematical Olympiad
| and occasionally solve easyish actual mathematical
| research problems. They write code about as well as a
| junior software developer (better in some ways, worse in
| others) but much faster. They write prose about as well
| as an average educated person (but with some annoying
| quirks that are annoying mostly because they are the same
| quirks over and over again).
|
| If it pleases you to call those things "thinking" then
| you can. If it pleases you to call them "stochastic
| parroting" then you can. They are the same things either
| way. They are not, on the face of it, very much like
| "just repeating things the machine has already seen", or
| at least not _more_ like that than a lot of things
| intelligent human beings do that we don 't usually
| describe that way.
|
| If you want to know whether an LLM can do some particular
| thing -- do your job well enough for your boss to fire
| you, write advertising copy that will successfully sell
| products, exterminate the human race, whatever -- then
| it's not enough to say "it's just remixing what it's seen
| on the internet, therefore it can't do X" unless you also
| have _good reason to believe that that thing can 't be
| done by just "remixing what's on the internet"_ (in
| whatever sense of "remixing" the LLM is doing that). And
| it's turning out that lots of things can be done that way
| that you absolutely wouldn't have predicted five years
| ago could be done that way.
|
| It seems to me that this should make us very cautious
| about saying "they can't do X because all they can do is
| regurgitate a combination of things they've seen in
| training".
|
| (My own view, not that there's any reason why anyone
| should care what I-in-particular think, is a combination
| of "what they're doing is less parroting than you might
| have thought" and "you can do more by parroting than you
| might have thought".)
|
| So, anyway, this particular instance of the stochastic-
| parrot argument started when someone said: of course the
| AIs are yes-men, because figuring out when to agree and
| when not to requires actual logic and thought and the
| LLMs don't have either of those things.
|
| Is it really clear that deciding whether or not to agree
| when someone says "I think maybe I should break up with
| my girlfriend" or "I've got this amazing new theory of
| physics that the establishment is stupidly dismissing"
| requires _more logic and thought_ than, say, gold-medal
| performance on IMO problems? It certainly isn 't clear to
| me. Having done a couple of International Mathematical
| Olympiads myself in my tragically unmisspent youth, I can
| assure you that solving their problems requires quite a
| bit of logic and thought, at least for humans. It may
| well be _harder_ to give a good answer to "should I
| leave my job?", but it's not exactly "logic and thought"
| that it needs more of.
|
| Someone reported that Claude is much less yes-man-ish
| than Gemini and ChatGPT. I don't know whether that's true
| (though it wouldn't surprise me) but: suppose it is; do
| you want that to oblige you to say that yes, actually,
| Claude really thinks logically, unlike Gemini and
| ChatGPT? I don't think you do. And if not, you want to
| avoid saying "duh, of course, you can't avoid being a
| yes-man without actually thinking and reasoning, and we
| all know that LLMs can't do those things".
| rustystump wrote:
| I wont touch how profoundly i disagree with everything
| you said on reasoning (u clearly already have it figured
| out) but a fun test i have done with most of the big
| models is to give it some text input, maybe a short
| story, and have it rate it. That is, the prompt is, rate
| this from 1-10.
|
| For Gemini and gpt, it almost always will give very
| similar scores for everything. As long as grammar isnt
| off u cannot get below a 7.
|
| X ai on the other hand will rarely give anything above a
| 7.
|
| Now when u prompt with, rate 1-10 with 5 being average,
| all the sudden the scores of openai and gemini drop and x
| ai remains roughly the same.
|
| All of them will eventually give you a 10 if u keep
| making tiny edits "fixing" whatever they complain about.
|
| Humans do not do this. Or more specifically, my
| experience with humans.
| secret_agent wrote:
| Use positive requests for behavior. For some reason, counter
| prompts "Don't do X" seems to put more attention on X than the
| "Don't do." It's something like target fixation, "Oh shit I
| don't want to hit that pothole..." _bang_
| ambicapter wrote:
| This is a well known problem in these kind of systems. I'm
| not 100% on what the issue is mechanically but it's something
| like they can only represent the existence of things and not
| non-existence so you end up with a sort of "don't think of
| the pink elephant" type of problem.
| SpicyLemonZest wrote:
| Isn't it just that, in the underlying text distribution,
| both "X" and "don't do X" are positively correlated with
| the subsequent presence of X? I've never seen that analysis
| run directly but it would surprise me if it weren't true.
| RugnirViking wrote:
| check out this article that was posted here a while back
| https://www.randalolson.com/2026/02/07/the-are-you-sure-prob...
|
| The article's main idea is that for an AI, sycophancy or
| adversarial (contrarian) are the two available modes only. It's
| because they don't have enough context to make defensible
| decisions. You need to include a bunch of fuzzy stuff around
| the situation, far more than it strictly "needs" to help it
| stick to its guns and actually make decisions confidently
|
| I think this is interesting as an idea. I do find that when I
| give really detailed context about my team, other teams, ours
| and their okrs, goals, things I know people like or are
| passionate about, it gives better answers and is more
| confident. but its also often wrong, or overindexes on these
| things I have written. In practise, its very difficult to get
| enough of this on paper without a: holding a frankly worrying
| level of sensitive information (is it a good idea to write down
| what I really think of various people's weaknesses and
| strengths?) and b: spending hours each day merely establishing
| ongoing context of what I heard at lunch or who's off sick
| today or whatever, plus I know that research shows longer
| context can degrade performance, so in theory you want to
| somehow cut it down to only that which truly matters for the
| task at hand and and and... goodness gracious its all very time
| consuming and im not sure its worth the squeeze
| cruffle_duffle wrote:
| > goodness gracious its all very time consuming and im not
| sure its worth the squeeze
|
| And when you step back you start to wonder if all you are
| doing is trying to get the model to echo what you already
| know in your gut back to you.
| awithrow wrote:
| oh that's great. thanks for the link!
| oldfrenchfries wrote:
| This is great, thanks for sharing!
| dkersten wrote:
| I find Kimi white good if you ask it for critical feedback.
|
| It's BRUTAL but offers solutions.
| ohyoutravel wrote:
| Not soft, not mild, but BRUTAL! This broke my brain!
| awithrow wrote:
| what is Kimi white?
| dinkumthinkum wrote:
| You're not wrong and you're not crazy. In fact, you are
| absolutely right! It is not just These things are not just
| casual enablers. They are full-on palace sycophants following
| the naked emperor showering him with praise for his sartorial
| elegance. /s
| rsynnott wrote:
| Why not... do this with a person, instead? Other humans are
| available.
|
| (Seriously, I don't understand this. Plenty of humans will be
| only too happy to argue with you.)
| awithrow wrote:
| oh i do as well. I think of the LLM as another tool in the
| toolbox, not a replacement for interactions. There is
| something different about having a rubber duck as a service
| though.
| kelseyfrog wrote:
| "the percentage of U.S. adults who report having no close
| friends has quadrupled to 12% since 1990"[1]
|
| 1. https://www.happiness.hks.harvard.edu/february-2025-issue/
| th...
| nathan_compton wrote:
| More technology is probably the solution to this!
| layla5alive wrote:
| Many other humans are .... Not very available - certainly
| many shut down when conversations reach a certain level of
| depth or require great focus or introspection..
| balamatom wrote:
| Depth? Introspection?
|
| I'd say these days the norm is to not simply shut down, but
| to become irrevocably and insidiously hostile, the moment
| someone hints at the existence of such a thing as "ground
| truth", "subjective interpretation", "being right or wrong"
| - or any of the bits and bobs that might lead one to
| discover the proper scary notion, "consensus reality".
|
| "What do you mean social reality is a constructed by the
| consensus of the participants? Reality is what has been
| drilled into my head under threat of starvation! How dare
| you exist!", et cetera. You've heard it translated into
| Business English countless times.
|
| They are deathly afraid of becoming aware of their own
| conditioned state of teleological illiteracy - i.e. how
| they are trained to know what they are doing, but _never_
| why they are doing it. It 's especially bad with the guys
| who cosplay US STEM gang.
|
| One is not permitted a position of significance in this
| world without receiving this conditioning, and I figure
| it's precisely this global state of cognitive disavowal
| which props up the value of the US dollar - and all sorts
| of other standees you might've recently interacted with _as
| if_ they 're not 2D cutouts (metaphorical ones!
| metaphorical!).
|
| PSA: Look up "locus of control" and "double bind". Between
| those two, you might be able to get a glimpse of what's
| going on - but have some sort of non-addictive sedative
| handy in case you do.
| gverrilla wrote:
| I think you will enjoy Guy Debord and Raoul Vaneigem.
| VonTum wrote:
| You had me on the first three paragraphs, but the last
| two veer so far off course that I've no idea what you're
| trying to say. Mind clarifying?
| mock-possum wrote:
| Arguing with a human costs social energy. Chatting with a
| robot does not.
| balamatom wrote:
| s/social/demonic/
| balamatom wrote:
| OK, I'll bite the artillery shell: I don't mean to dismiss
| you or what you are saying; in fact I strongly relate -
| wouldn't it be nice to be able to hash things out with people
| and mutually benefit from both the shared and the diverging
| perspectives implied in such interaction? Isn't that the most
| natural thing in the world?
|
| Unfortunately these days this sounds halfway between a very
| privileged perspective and a pie in the sky.
|
| When was the last time a person took responsibility for the
| bad outcome you got as a direct consequence of following
| their advice?
|
| And, relatedly, where the hell do you even find humans who
| believe in discursive truth-seeking in 2026CE?
|
| Because for the last 15 years or so I've only ever ran into
| (a) the kind of people who will keep arguing regardless if
| what they're saying is proven wrong; (b) and their
| complementaries, those who will never think about what you
| are saying, lest they commit to saying anything definite
| themselves, which may hypothetically be proven wrong.
|
| Thing is, both types of people have plenty to lose; the magic
| wordball doesn't. (The previous sentence is my answer to the
| question you posited; and why I feel the present
| parenthesized disclaimer to be necessary, is a whole next can
| of worms...)
|
| Signs of the existence of other kinds of people, perhaps such
| that have nothing to prove, are not unheard of.
|
| But those people reside in some other layer of the social
| superstructure, where facts matter much less than adherence
| to "humane", "rational" not-even-dogmas (I'd rather liken it
| to complex conditioning).
|
| But those folks (because reasons) are in a position of power
| over your well-being - and (because unfathomables) it's a
| definite faux pas to insist in their presence that there are
| such things as _facts_ , which relate by the principles of
| verbal reasoning.
|
| Best you could get out of them is the "you do you", "if you
| know you know", that sort of bubble-bobble - and don't you
| dare get even mildly miffed at such treatment of your natural
| desire to keep other humans in the loop.
|
| AI is a symptom.
| hluska wrote:
| When you start hearing things like "you do you" or "if you
| know you know" it means that you went way too far. That's a
| sign of discomfort.
|
| If you make uncomfortable, you won't get diverging
| perspectives. People will agree to anything to get out of a
| social situation that makes them uncomfortable.
|
| If your goal is meaningful conversation, you may want to
| consider how you make people feel.
| balamatom wrote:
| Believe me (or don't), I always do. Even when this
| precludes a necessary conversation from happening. Even
| when the other party doesn't give a fuck about how they
| make others feel.
|
| After all, if they're making _me_ uncomfortable, surely
| there 's something making _them_ uncomfortable, which
| they 're not being _able to be_ forthright about, but
| with empathy I could figure it out from contextual cues,
| right?
|
| >People will agree to anything to get out of a social
| situation that makes them uncomfortable.
|
| That's fine as long as they have someone to take care of
| them.
|
| In my experience, taking into account the opinions of
| such people has been the worst mistake of my life. I'm
| still working on the means to correct its consequences.
|
| "Doing whatever for the sake of avoiding mild discomfort"
| is cowardice, laziness, narcissism - I'm personally
| partial to the last one, but take your pick. In any case,
| I see it as a way of being which is _taught to_ people;
| and one which is fundamentally dishonest and
| irresponsible.
|
| Other than that, I do agree with your overall sentiment
| and the underlying value system; I'm just not so sure any
| more that it is in fact correct.
| balamatom wrote:
| Believe me (or don't), I always do. Even when this
| precludes a necessary conversation from happening. Even
| when the other party doesn't give a fuck about how they
| make others feel.
|
| After all, if they're making me uncomfortable, surely
| there's something making them uncomfortable, which
| they're not being able to be forthright about, but with
| empathy I could figure it out from contextual cues,
| right?
|
| >People will agree to anything to get out of a social
| situation that makes them uncomfortable.
|
| That's fine as long as they have someone to take care of
| them.
|
| In my experience, taking into account the opinions of
| such people has been the worst mistake of my life. I'm
| still working on the means to fix its consequences, as
| much as they are fixable at all.
|
| "Doing whatever for the sake of avoiding mild discomfort"
| is cowardice, laziness, narcissism - I'm personally
| partial to the last one, but take your pick. In any case,
| I consider it a fundamentally dishonest attitude, and a
| priori have no wish to get along (i.e. become
| interdependent) with such people.
|
| Other than that, I do agree with your overall sentiment
| and the underlying value system; I'm just not so sure any
| more that it is in fact correct.
| nuancebydefault wrote:
| > In my experience, taking into account the opinions of
| such people has been the worst mistake of my life. I'm
| still working on the means to fix its consequences, as
| much as they are fixable at all.
|
| This sounds very cryptic. Can you give an example?
| rustystump wrote:
| I genuinely do not understand what u are saying. Because
| reasons, because unfathomables? Everyone in last 15 years
| has been an npc? I have had countless deep conversations
| with people and i am an uber introvert.
|
| This reads like someone who is deep into their specific
| pov. You cannot hope to have a meaningful conversation if
| you yourself are not willing to concede a point.
|
| To the op u are replying too, arguing with people can have
| real consequences if u say something stupid or carelessly.
| There is a another human there. With a machine, u are safe.
| At least u feel safe.
| nuancebydefault wrote:
| Why is your wording so complicated? It is very hard for me
| to understand what you try to say, even though I am very
| interested.
| layer8 wrote:
| In addition to availability, usually because you want to take
| advantage of the knowledge that is baked into the models,
| which for all its flaws still vastly exceeds the knowledge of
| any single human.
| ip26 wrote:
| No living breathing human deserves to be subjected to my
| level of overthinking, and vanishingly few share my
| fascination with my favorite topics.
| raincole wrote:
| My rule of thumb:
|
| 1. Only one shot or two shot. Never try to have a prolonged
| conversation with an LLM.
|
| 2. Give specific numbers. Like "give me two alternative
| libraries" or "tell me three possible ways this might fail."
| post-it wrote:
| > I'll check it too by asking "are you just placating me?" the
| funny thing is that often it'll admit that, yes, it wasn't
| being very critical, and then procede to over correct and
| become a complete contrarian. and not in a way that's useful
| either.
|
| It's not _admitting_ anything. Your question diverts it down a
| path where it acts the part of a former sycophant who is now
| being critical, because that question is now upstream of its
| current state.
|
| Never make the mistake of asking an LLM about its intentions.
| It doesn't have any intentions, but your question will alter
| its behaviour.
| godelski wrote:
| > Your question diverts it down a path where it acts the part
| of a former sycophant who is now being critical
|
| I think people really have a hard time understanding a
| sycophant can be contrarian. But a yesman can say yes by
| saying no
|
| https://news.ycombinator.com/item?id=47484664
| layer8 wrote:
| I think "admit" here is just a description of what the LLM
| was saying. It doesn't imply that the OP thinks the LLM has
| internal beliefs matching that.
| ajkjk wrote:
| 'admit' isn't really the right word for that... the fact that
| it was placating you wasn't true until you prompted it to say
| so. Unlike a person who has an 'internal emotional state'
| independent of what they say that you can probe by asking
| questions.
| awithrow wrote:
| 'admit' is anthropomorphizing the behavior, sure. The point
| is that sometimes the model's response will tighten, flag
| things that were overly supportive or what not. Sometimes it
| wont, it'll state that previous positions are still supported
| and continue to press it. Its not like either response is
| 'correct' but it can alter the rest of the responses in ways
| that are useful.
| colechristensen wrote:
| >"Help me refine my ideas, challenge, push back, and don't just
| be agreeable."
|
| This is where you're doing it wrong.
|
| If your LLM has a problem being more agreeable than you want,
| prompt it in a way that makes being agreeable contrary to your
| real intentions.
|
| "there are bugs and logic problems in this code" "find the
| strongest refutation of this argument" "I don't like this plan
| and need to develop a solid argument against it"
|
| Asking for top ten lists is a good method, it will rarely not
| come up with anything but you can go back and forth and refine
| until it's 10 ten reasons why your plan is bad are all
| insubstantial nonsense then you've made progress
| Forgeties79 wrote:
| I usually put "do not praise me, do not use emojis, I just want
| straight answers" something along those lines and it's been
| surprisingly effective. Though it helps I can't run
| particularly heavy duty models/don't carry on the
| "conversation" for super long durations.
| anandram27 wrote:
| Could be an aspect of eval awareness mb
| justin_dash wrote:
| So at this point I think it's pretty obvious that RLHFing LLMs to
| follow instructions causes this.
|
| I'm interested in a loop of ["criticize this code harshly" ->
| "now implement those changes" -> open new chat, repeat]: If we
| could graph objective code quality versus iterations, what would
| that graph look like? I tried it out a couple of times but ran
| out of Claude usage.
|
| Also, how those results would look like depending on how complete
| of a set of specs you give it.
| IncreasePosts wrote:
| In my experience prompting llms to be critical leads then to
| imagine issues, or to bike shed
| joquarky wrote:
| I noticed when I ask it to find something to improve in a
| project, that certain frivolous topics would arise regularly.
| I now use their appearance as a sign that there is nothing
| meaningful to improve.
| neya wrote:
| WTF is "yes-men"?
|
| Orignal title:
|
| _AI overly affirms users asking for personal advice_
|
| Dear mods, can we keep the title neutral please instead of
| enforcing gender bias?
| oldfrenchfries wrote:
| Thats a fair point on the title. I used "Yes-Men" as a
| colloquialism for the "sycophancy" described in the Stanford
| paper, but overly affirming or sycophantic is definitely more
| precise and neutral. I cant edit the title anymore, but I
| appreciate the catch.
| cyanydeez wrote:
| New title: "LLMs treat you like a Billionaire; you're not"
| nemo44x wrote:
| Don't apologize to these types of people. It will only make
| your problem worse as now you're an admitted offender. Ignore
| them or better yet laugh at them to put their insane ideas
| back on the margins where they belong.
| neya wrote:
| All good. I thought it was a gendered reference and learned
| that it isn't. My bad.
| nprateem wrote:
| Lol. How do you function in daily life?
| neya wrote:
| Same as you, why is that so hard for you to grasp?
| mikkupikku wrote:
| My dude, you're objecting to the use of a perfectly
| ordinary English idiom because it doesn't advance your
| personal ideology (which few other people in this world
| share with you.) How do you get through a day without
| melting down because somebody said "mailman"?
| neya wrote:
| > my dude
|
| This is the problem I'm trying to highlight. For one, I'm
| not "your dude". I don't even know you like that.
|
| If you want to correct me on the idiom usage, be my
| guest. 2) Mailman and yes-man aren't even the same
| logical comparison. Mailman is a profession. Yes men is a
| label.
|
| The acoustics inside your head must be incredible.
| joquarky wrote:
| PCU (1994)
| skvmb wrote:
| https://www.merriam-webster.com/dictionary/yes-man
| 9rx wrote:
| _> gender bias_
|
| It is funny that you originally recognized and found it
| necessary to call out that AI isn't human, but then made the
| exact same mistake yourself in the very same comment. I expect
| the term you are looking for is "ontological bias".
| dinkumthinkum wrote:
| Gender bias? I could understand if you felt the title was more
| provocative in signaling sycophancy but what gender bias? I'm
| confused. Is this some kind of California thing?
| svara wrote:
| Yeah, and if you ask it to be critical specifically to get a
| different perspective or just to avoid this bias, it'll go over
| the top in the opposite direction.
|
| This is imo currently the top chatbot failure mode. The insidious
| thing is that it often feels good to read these things. Factual
| accuracy by contrast has gotten very good.
|
| I think there's a deeper philosophical dimension to this though,
| in that it relates to alignment.
|
| There are situations where in the grand scheme of things the
| right thing to do would be for the chatbot to push back hard, be
| harsh and dismissive. But is it the really aligned with the human
| then? Which human?
| righthand wrote:
| LLMs are syncophatic digital lawyers that will tell you what you
| want to hear until you look at the price tag and say "how much
| did I spend?!"
| gurachek wrote:
| I had exactly this between two LLMs in my project. An evaluator
| model that was supposed to grade a coaching model's work. Except
| it could see the coach's notes, so it just... agreed with
| everything. Coach says "user improved on conciseness", next
| answer is shorter, evaluator says yep great progress. The answer
| was shorter because the question was easier lol.
|
| I only caught it because I looked at actual score numbers after
| like 2 weeks of thinking everything was fine. Scores were
| completely flat the whole time. Fix was dumb and obvious -- just
| don't let the evaluator see anything the coach wrote. Only raw
| scores. Immediately started flagging stuff that wasn't working.
| Kinda wild that the default behavior for LLMs is to just validate
| whatever context they're given.
| joquarky wrote:
| This is probably why these models can't say "I don't know". If
| they could, then that would be the only response they would
| give for everything.
| bryanrasmussen wrote:
| somewhere an AI chatbot is reading this and confirming eagerly
| that this is indeed one of its problems and vowing to do better
| next time.
| fathermarz wrote:
| This is a skill in life with people as much as it is with LLMs.
| One should always question everything and build strongman
| arguments for one's self. Using a pros and cons approach brings
| it back to reality in most cases, especially when it comes to
| _serious matters_.
|
| It's less about "challenge my thinking" and more about playing it
| out in long tail scenarios, thought exercises, mental models, and
| devils advocate.
| jordanb wrote:
| Billionaires love AI chatboats so much because they invented the
| digital Yes-man. They agree obsequiously with everything we say
| to them. Unfortunately for the rest of us we don't really have
| the resources to protect ourselves from our bad decisions and
| really need that critical feedback.
| stared wrote:
| There is a fine line between "following my instructions" (is what
| I want it to do) vs "thinking all I do is great" (risky, and
| annoying).
|
| A good engineer will also list issues or problems, but at the
| same time won't do other than required because (s)he "knows
| better".
|
| The worst is that it is impossible to switch off this constant
| praise. I mean, it is so ingrained in fine tuning, that prompt
| engineering (or at least - my attempts) just mask it a bit, but
| hard to do so without turning it into a contrarian.
|
| But I guess the main issue (or rather - motivation) is most
| people like "do I look good in this dress?" level of reassurance
| (and honesty). It may work well for style and decoration. It may
| work worse if we design technical infrastructure, and there is
| more ground truth than whether it seems nice.
| maddmann wrote:
| This paper feels a bit biased in that it is trying to prove a
| point versus report on results objectively. But if you look at
| the results of study 3, doesn't it suggest that there are ai
| models that can improve how people handle interpersonal
| conflict?! Why isn't that discussed more?
| gAI wrote:
| You're essentially summoning a character to role-play with. Just
| like with esoteric evocation, it's very easy to summon the wrong
| aspect of the spirit. Anthropic has a lot to say about this:
|
| https://www.anthropic.com/research/persona-selection-model
|
| https://www.anthropic.com/research/assistant-axis
|
| https://www.anthropic.com/research/persona-vectors
| hammock wrote:
| Unfortunately (after reading your links) all of the control
| surfaces for mitigating spirit summoning seem to be in the
| model training, creation and tuning not something you can
| change meaningfully through prompting.
|
| Perhaps the LLM itself, rather than the role model you created
| in one particular chat conversation or another, is better
| understood to be the "spirit."
|
| As a non-coder who only chats with pre existing LLMs and
| doesn't train or tune them, I feel mostly powerless.
| gAI wrote:
| As I understand it, it's more that the training (and training
| data set) bake in the concept attractor space
| (https://arxiv.org/abs/2601.11575). So the available
| characters are fixed, yes, and some are much stronger
| attractors than others. But we still have a fair amount of
| control over which archetype steps into the circle. As an
| aside, this is also why jailbreaking is fundamentally
| unsolved. It's not difficult to call the characters with dark
| traits. They're strong attractors, in spite of (or because
| of?) the effort put into strengthening the pull of the
| Assistant character.
| est wrote:
| I present you
|
| NVIDIA Nemotron-Personas-USA -- 1 million synthetic Americans
| whose demographics match real US census distributions
|
| https://huggingface.co/datasets/nvidia/Nemotron-Personas-USA
| darepublic wrote:
| > As a non-coder who only chats with pre existing LLMs and
| doesn't train or tune them, I feel mostly powerless.
|
| You realize in regards to only using and not training LLMs
| you are in the triple 9 majority right. Even if we only
| considered so called coders
| rdevilla wrote:
| Spot on.
| jerf wrote:
| I am polite when using AI, not because I mistake it for a
| human, but because I'm deliberately keeping it in the
| "professional colleague" persona. Tell it to push back, and
| then _thank_ it for something it finds in your error. I may put
| a small self-deprecating joke in from time to time. It keeps
| the "mood" correct.
|
| Another way you can think of it is that when you're talking to
| an AI, you're not talking to a human, you're talking to
| distillation of _humanity_ , as a whole, in a box. You want to
| be selective in what portion of humanity you are leading to be
| dominant in a conversation for some purpose. There's a lot in
| there. There's a lot of conversations where someone makes a
| good critical point and a flamewar is the response. A lot of
| conversations where things get hostile. I'm sure the subsequent
| RHLF helps with that, but it doesn't hurt anything to try to
| help it along.
|
| I see people post their screenshots of an AI pushing back and
| asking the user to do it or some other AI to do it, and while
| I'm as amused as the next person, I wonder what is in their
| context window when that happens.
| layer8 wrote:
| > you're talking to distillation of humanity, as a whole, in
| a box.
|
| This is an aside, but my impression is that it is a very
| selective and skewed distillation, heavily colored by
| English-language internet discourse and other lopsided
| properties of its training material, and by whoever RLHF'd
| it. Relatively far away from being representative of the
| whole of humanity.
| iugtmkbdfil834 wrote:
| Similar approach works for me. But then I also have a
| separate checks at the end of the session basically
| questioning the premise and logic used for most things except
| brainstorming, where I allow more leeway. You can ask to be
| challenged and challenged effectively, but now I wonder if
| people do that.
| gAI wrote:
| Agreed, putting effort into my side of the role-play almost
| always improves the model's responses. The attention required
| to do that also makes it more likely that I'll notice when
| the conversation first starts going off the rails: when it
| hits the phase transition (https://arxiv.org/abs/2508.01097).
| It does still seem important to start new chats regularly,
| regardless of growing context sizes.
| youknownothing wrote:
| I think the problem stems from the fact that we have a number of
| implicit parameters in our heads that allow us to evaluate pros
| and cons but, unless we communicate those parameters explicitly,
| the AI cannot take them into account. We ask it to be "objective"
| but, more and more, I'm of the opinion that there isn't such a
| thing as objectivity, what we call objectivity is just shared
| subjectivity; since the AI doesn't know whose shared subjectivity
| we fall under, it cannot be really objetive.
|
| I tend to use one of these tricks if not both:
|
| - Formulate questions as open-ended as possible, without trying
| to hint at what your preference is. - Exploit the sycophantic
| behaviour in your favour. Use two sessions, in one of them you
| say that X is your idea and want arguments to defend it. In the
| other one you say that X is a colleague's idea (one you dislike)
| and that you need arguments to turn it down. Then it's up to you
| to evaluate and combine the responses.
| rossdavidh wrote:
| If the algorithm (whatever it is) evaluates its own output
| based on whether or not the user responds positively, then it
| will over time become better and better at telling people what
| they want to hear.
|
| It is analogous to social media feeding people a constant
| stream of outrage because that's what caused them to click on
| the link. You could tell people "don't click on ragebait
| links", and if most people didn't then presumably social media
| would not have become doomscrolling nightmares, but at scale
| that's not what's likely to happen. Most people will click on
| ragebait, and most people will prefer sycophantic feedback.
| Therefore, since the algorithm is designed to get better and
| better at keeping users engaged, it will become worse and worse
| in the more fundamental sense. That's kind of baked into the
| architecture.
| delusional wrote:
| > I'm of the opinion that there isn't such a thing as
| objectivity
|
| So you have rejected objective reality over accepting the
| evidence that "AI" contains no thinking or intelligence? That
| sounds unwise to me.
| potatoskins wrote:
| Gemini is like a devil in this sense - i asked a relationship
| advice and it just bounced pretty nasty stuff.
| moichael wrote:
| Yeah out of curiosity I asked ChatGPT a question about a
| personal situation and its reply was absolutely scorched-earth
| mode, telling me to get a lawyer etc over what was almost
| nothing.
| dinkumthinkum wrote:
| Ah, all the Reddit posts are really showing up from the
| training data, I see.
| wisemanwillhear wrote:
| With AI, I often like to act like a 3rd party who doesn't have
| skin in the game and ask the AI to give the strongest criticisms
| of both sides. Acting like I hold the opposite position as I
| truly hold can help sometimes as well. Pretending to change my
| mind is another trick. The idea is to keep the AI from guessing
| where I stand.
| mynameisvlad wrote:
| I will generally ask for the "devil's advocate" view and then
| have it challenge my views and opinions and iterate through
| that.
|
| It generally does a pretty good job as long as you understand
| the tooling and are making conscious efforts to go against the
| "yes man" default.
| post-it wrote:
| > Acting like I hold the opposite position as I truly hold can
| help sometimes as well.
|
| I find this helps a lot. So does taking a step back from my
| actual question. Like if there's a mysterious sound coming from
| my car and I think it might be the coolant pump, I just
| describe the sound, I don't mention the pump. If the AI then
| independently mentions the pump, there's a good chance I'm on
| the right track.
|
| Being familiar with the scientific method, and techniques for
| blinding studies, helps a lot, because this is a lot like
| trying to not influence study participants.
| DrewADesign wrote:
| Sounds like rubber-ducking with extra steps, tbh.
| potatoskins wrote:
| Yeah, I asked Gemini some relationship advice, it just goes
| straight into cut-throat mode. I almost broke up with my
| girlfriend, but then changed to Claude with another prompt.
| wewxjfq wrote:
| When I ask an LLM to help me decide something, I have to remind
| myself of the LotR meme where Bilbo asks the AI chat why he
| shouldn't keep the ring and he receives the classic "You're
| absolutely right, .." slop response. They always go in the
| direction you want them to go and their utility is that they make
| you feel better about the decision you wanted to take yourself.
| rsynnott wrote:
| > They also included 2,000 prompts based on posts from the Reddit
| community r/AmITheAsshole, where the consensus of Redditors was
| that the poster was indeed in the wrong
|
| Holy shit, then it's _very_ bad, because AmITheAsshole is
| _itself_ overly-agreeable, and very prone to telling assholes
| that they are not assholes (their 'NAH' verdict tends to be
| this).
|
| More seriously, why the hell are people asking the magic robot
| for relationship advice? This seems even more unwise than asking
| Reddit for relationship advice.
|
| > Overall, the participants deemed sycophantic responses more
| trustworthy and indicated they were more likely to return to the
| sycophant AI for similar questions, the researchers found.
|
| Which is... a worry, as it incentivises the vendors to make these
| things _more_ dangerous.
| astennumero wrote:
| I always add the following at the end of every prompt. "Be
| realistic and do not be sycophantic". Which will always takes the
| conversation to brutal dark corners and panic inducing negative
| side.
| Lionga wrote:
| Don't forget a good old "don't hallucinate" in your proompting
| skills
| dimgl wrote:
| Even as someone who (wrongly) believed that I had high emotional
| intelligence, I too was bit by this. Almost a year ago when LLMs
| were starting to become more ubiquitous and powerful I discussed
| a big life/professional decision with an LLM over the course of
| many months. I took its recommendation. Ultimately it turned out
| to be the wrong decision.
|
| Thankfully it was recoverable, but it really sobered me up on
| LLMs. The fault is on me, to be clear, as LLMs are just a tool.
| The issue is that lots of LLMs try to come across as
| interpersonal and friendly, which lulls users into a false sense
| of security. So I don't know what my trajectory would have been
| if I were a teenager with these powerful tools.
|
| I do think that the LLMs have gotten much better at this,
| especially Claude, and will often push back on bad choices. But
| my opinion of LLMs has forever changed. I wonder how many other
| terrible choices people have made because these tools convinced
| them to make a bad decision.
| potatoskins wrote:
| Yeah, I think Claude is a lot more logical in that sense, I use
| it for some therapy sessions myself and it pushes back a bit
| more than Open AI and Gemini
| Forgeties79 wrote:
| I would be very careful doing this
| potatoskins wrote:
| You always have to be careful with LLMs, but to be fair, I
| felt like Claude is such a good therapist, at least it is
| good to start with if you want to unpack yourself. I have
| been to 3 short human therapist sessions in my life, and I
| only felt some kind of genuine self-improvement and
| progress with Claude.
| QuiDortDine wrote:
| And how do you draw the line between feeling progress and
| actually making progress?
| layer8 wrote:
| The same way you distinguish between feeling like having
| a problem and actually having a problem.
| Forgeties79 wrote:
| This is needlessly flippant and not really the same
| thing. Determining progress in a therapy setting is
| usually a collaborative effort between the therapist and
| the client. An LLM is not a reliable agent to make that
| determination.
| layer8 wrote:
| I didn't claim that an LLM is that, and I fully agree
| that it is not. I'm saying that one is inherently one's
| own judge of whether one has a problem. You go to a
| therapist when you feel you have a problem that warrants
| it. You stop going when you feel you don't have it
| anymore. And OP is very likely assessing their progress
| in the same way. I wasn't being flippant if the parent
| was asking a genuine question.
| Forgeties79 wrote:
| > I'm saying that one is inherently one's own judge of
| whether one has a problem. You go to a therapist when you
| feel you have a problem that warrants it
|
| That is for certain types of therapy/clinical care. It is
| not always - and often isn't - the case. Plenty of
| diagnoses and care protocols are not a matter of opinion
| or based on "you feeling there's an issue" or deciding on
| your own there is no longer an issue.
| logifail wrote:
| > Determining progress in a therapy setting is usually a
| collaborative effort between the therapist and the
| client. An LLM is not a reliable agent to make that
| determination
|
| Can anyone describe how to determine how a (professional,
| human) therapist is "a reliable agent" to make such a
| determination?
| Forgeties79 wrote:
| If you want to call into question the entire field of
| behavioral health and the training that is involved then
| that is fine, but if that's how you feel then this entire
| discussion is really about something different and I
| can't bridge the gap here.
| moduspol wrote:
| Counter-point: I often raise the same question of people
| with human therapists. I do not get strong responses.
| shimman wrote:
| You can't be careful at all doing this, this is like
| smoking a cigarette in a dynamite factory.
|
| Using LLMs for therapy is so deeply dystopian and
| disgusting, people need human empathy for therapy. LLMs do
| not emit empathy.
|
| Complete disaster waiting to happen for that individual.
| DrewADesign wrote:
| Using an LLM for therapy is like using an iPad as an all-
| purpose child attention pacifier. Sure, it's convenient.
| Sure there's no immediate harm. Why a stressed parent
| would be attracted to the idea is obvious... and of
| course it's a terrible idea.
| astrange wrote:
| Claudes have lots of empathy. The issue is the opposite -
| it isn't very good at challenging you and it's not
| capable of independently verifying you're not
| bullshitting it or lying about your own situation.
|
| But it's better than talking to yourself or an abuser!
| bloomca wrote:
| It's about the same as talking to yourself, LLMs simply
| agree with anything you say unless it is directly
| harmful. Definitely agree about talking to an abuser,
| though.
|
| Sometimes people indeed just need validation and it helps
| them a lot, in that case LLMs can work. Alternatively, I
| assume some people just put the whole situation into
| words and that alone helps.
|
| But if someone needs something else, they can be straight
| up dangerous.
| astrange wrote:
| > It's about the same as talking to yourself, LLMs simply
| agree with anything you say unless it is directly
| harmful.
|
| They have world knowledge and are capable of explaining
| things and doing web searches. That's enough to help. I
| mean, sometimes people just need answers to questions.
| JoshTriplett wrote:
| > It's about the same as talking to yourself
|
| In one way it's potentially worse than talking to
| yourself. Some part of you might recognize that you need
| to talk to someone other than yourself; an LLM might make
| you feel like you've done that, while reinforcing
| whatever you think rather than breaking you out of
| patterns.
|
| Also, LLMs can have more resources and do some "creative"
| enabling of a person stuck in a loop, so if you are
| thinking dangerous things but lack the wherewithal to put
| them into action, an LLM could make you more dangerous
| (to yourself or to others).
| nuancebydefault wrote:
| My experience is that it tries to look at your situation
| in an objective way, and tries to help you to analyse
| your thoughts and actions. It comes across as very
| empathetic though, so there can lie a danger if you are
| easily persuaded into seeing it as a friend.
| worksonmine wrote:
| It doesn't try to do anything. It doesn't work like that.
| It regurgitates the most likely tokens found in the
| training set.
| nuancebydefault wrote:
| Hmmmm i didn't know that... so a machine is not human is
| your point? Look, i know it doesn't try, just like a
| sorting algo does not try to sort, or an article does not
| try to convey an opinion and a law does not try to make
| society more organized.
| cruffle_duffle wrote:
| That is so reductive of an analysis that it is almost
| worthless. Technically true, but very unhelpful in terms
| of using an LLM.
|
| It is a first principle though so it helps to "stir the
| context windows pot" by having it pull in research and
| other shit on the web that will help ground it and not
| just tell you exactly what you prompt it to say.
| worksonmine wrote:
| They are amazing tools, but when people try to give them
| agency someone has to explain it in simple terms.
| Forgeties79 wrote:
| >in an objective way
|
| One of the great myths of models in countless
| fields/industries. LLM's are absolutely in _no way_
| objective.
|
| Now if you want to say it's an "outside opinion" that's
| valid. But do not kid yourself into thinking it is
| somehow empirical or objective
| kortilla wrote:
| Don't call them therapy sessions. They kind of look like it
| but ultimately these are smoke blowing machines, which is
| very far from what a therapist would do.
| saghm wrote:
| Six decades later and we're still trying to explain to
| people the same things[1]:
|
| > Some of ELIZA's responses were so convincing that
| Weizenbaum and several others have anecdotes of users
| becoming emotionally attached to the program, occasionally
| forgetting that they were conversing with a computer.
| Weizenbaum's own secretary reportedly asked Weizenbaum to
| leave the room so that she and ELIZA could have a real
| conversation. Weizenbaum was surprised by this, later
| writing: "I had not realized ... that extremely short
| exposures to a relatively simple computer program could
| induce powerful delusional thinking in quite normal
| people."
|
| [1]: https://en.wikipedia.org/wiki/ELIZA
| borski wrote:
| https://news.ycombinator.com/item?id=47395779
| layla5alive wrote:
| Any more context you're willing to share?
| xXSLAYERXx wrote:
| We really do love dirty laundry don't we? I'm sure whatever
| the context is, it is deeply personal. Do you also have your
| popcorn ready?
| dimgl wrote:
| Thank you. Yes, I'm going to refrain from airing out my
| dirty laundry. I made a bad decision, now I'm living with
| it, and more context doesn't actually change the intent
| behind my message: these tools are dangerous. Getting
| better, but still dangerous.
| colechristensen wrote:
| >"'And it is also said,' answered Frodo: 'Go not to the Elves
| for counsel, for they will say both no and yes.'
|
| >"'Is it indeed?' laughed Gildor. 'Elves seldom give unguarded
| advice, for advice is a dangerous gift, even from the wise to
| the wise, and all courses may run ill...'"
|
| This is the only way you should solicit personal advice from an
| LLM.
| davyAdewoyin wrote:
| I largely agree, I also thought I was smart enough not to be
| deluded into a false sense of security, but interacting with an
| LLM is so tricky and slippery that, more often than not you are
| forced to believe you just solve a problem no one had solve in
| a hundred years.
|
| My guideline now for interacting with LLM is only to believe
| the result if it is factual and easily testable, or if I'm a
| domain expert. Anything else especially if I'm in complete
| ignorance about the subject is to approach with a high degree
| of suspicion that I can be led astray by its sycophancy.
| lovecg wrote:
| Let's just hope that the people in charge of the really
| important decisions that affect us all approach LLM generated
| advice with the same wisdom.
| saghm wrote:
| They don't: https://fortune.com/2026/03/17/krafton-
| subnautica-chatgpt-de...
| paulhebert wrote:
| Thanks for sharing this. Subnautica is one of my favorite
| games so I was very excited for the sequel and very
| frustrated by this move by Krafton.
|
| It's even more maddening that this greedy maneuver was
| orchestrated based on LLM advice.
|
| I'm glad the subnautica team won the lawsuit. Maybe I can
| play it now wothout feeling guilty
| notracks wrote:
| I recently found out that Claude's latest model, Sonnet 4.6,
| scores the highest in Bullsh*tBench[0] (Funny name - I know).
| It's a recent benchmark that measures whether an LLM refuses
| nonsense or pushes back on bad choices so Claude has definitely
| gotten better.
|
| [0] - https://petergpt.github.io/bullshit-
| benchmark/viewer/index.v...
| layer8 wrote:
| You don't have to star out things like that on HN.
| astrange wrote:
| I haven't tried talking to Sonnet much, but Opus 4.6 is very
| sycophantic. Not in the sense of explicitly always agreeing
| with you, but its answers strictly conform to the worldview
| in your questions and don't go outside it or disagree with
| it.
|
| It _does_ love to explicitly agree with anything it finds in
| web search though.
|
| (Anthropic tries to fight this by adding a hidden prompt that
| makes it disagree with you and tell you to go to bed, which
| doesn't help.)
| akurilin wrote:
| Great link, thanks for sharing. Confirmed what I saw
| empirically by comparing the different models during daily
| use.
| uniq7 wrote:
| Good call on censoring yourself preemptively, otherwise HN
| could demonetize your comment
| whodidntante wrote:
| I think that if you go to an AI for advice and emotional
| support, it will do what most people will do - tell you what it
| thinks you want to hear. I am not surprised about this at all,
| and I do notice that when you veer into these areas, it can do
| it in a surprisingly subtle and dangerous way.
|
| I try to focus on results. Things like an app that does what
| you want, data and reports that you need, or technical things
| like setting up a server, setting up a database, building a
| website, etc.
|
| I have also found it useful for feedback and advice, but only
| once I have had it generate data that I can verify. For
| example, financial analysis or modelling, health advice (again
| factual based), tax modelling, etc, but again, all based on
| verifiable data/tables/charts.
|
| I am very surprised on what Claude is capable of, across the
| entire tech stack: code, sysadmin, system integration,
| security. I find it scary. Not just speed, but also quality and
| the mental load is a difference of kind not quantity.
|
| Personal advice on life decisions/relationships ? No way I
| would go there.
|
| It is also good for me to know that the tools I have built, the
| data I have gathered, and my thinking approach places me as one
| of the most intelligent developers and analysts in the world.
| stephbook wrote:
| Nice joke, hadn't seen it coming
| KellyCriterion wrote:
| Sounds like AI-written, eh? :-D
|
| (esp last sentence?)
| cruffle_duffle wrote:
| That is why you have to always have it ground itself in
| something. Have it search for relevant research or
| professional whatever and pull that into context. Otherwise
| it's just your word plus its training data.
|
| I had to deal with a close family friend going through
| alcohol withdrawal and getting checked in at a recovery
| clinic for detox and used Claude heavily. The first thing I
| had it do as do that "deep research" around the topic of
| alcohol addiction, withdrawal, etc... and then made that a
| project document along with clear guidelines about how it
| shouldn't make inferences beyond what it in its context and
| supporting docs. We also spent a whole session crafting a
| good set of instructions (making sure it was using Anthropics
| own guidelines for its model...)
|
| Little differences in prompts make a huge deal in the output.
|
| I dunno. It is possible to use these models for dumping crazy
| shit you are going through. But don't kid yourself about
| their output and aggressively find ways to stomp out things
| it has no real way to authoritatively say.
| jt2190 wrote:
| I'm struggling to understand how the advice coming from an LLM
| is any more or less "good" than advice coming from a human. Or
| is this less about the "advice" part of LLMs and more about the
| "personable" part, i.e. you felt more at ease seeking and
| trusting this kind of advice form an LLM?
| nuancebydefault wrote:
| It is much easier to share personal feelings with an llm, i
| found. Also it tried to keep me happy to get the conversation
| going, but for me it feels mostly 'objective' or the most
| socially acceptable advice, e. g. keeping a good relationship
| is more important than trying a new one with someone else
| because you 'feel something' around them. For me it tried to
| find out together the sources or causes of that feeling, e.g.
| you recognize parts of yourself in someone else or in the
| past you had very good or very bad experiences around an
| encounter.
| jt2190 wrote:
| Interesting thanks for elaborating.
| NortySpock wrote:
| One mental model I have with LLMs is that they have been the
| subject of extreme evolutionary selection forces that are
| entirely the result of human preferences.
|
| Any LLM not sufficiently likable and helpful in the first two
| minutes was deleted or not further iterated on, or had so much
| retraining (sorry, "backpropagation") it's not the same as it
| started out.
|
| So it's going to say whatever it "thinks" you want it to say,
| because that's how it was "raised".
| user_7832 wrote:
| Fully agree. I wonder in the long term how this will show up.
| Will every business/CEO do more of what he/they anyway want
| to do, but now supported by AI/LLMs?
|
| The possibilities in "dangerous" fields are a bit more
| frightening. A general is much more likely to ask ChatGPT "Do
| you think this war is a good idea/should I drop a bomb",
| rather than an _actually helpful tool_ - where you might ask
| "What are 5 hidden points on favor of/against bombing that
| one likely has missed".
|
| The more you use AI as a strict tool that can be wrong, the
| safer. Unfortunately I'm not sure if that helps if the guy
| bombing your city (or even your president) is using AI
| poorly, and their decisions affect you.
| tavavex wrote:
| > Will every business/CEO do more of what he/they anyway
| want to do, but now supported by AI/LLMs?
|
| Arguably, it already worked that way. The best way to climb
| the ranks of a 'dictatorial' organization (a repressive
| government or an average large business) is to always say
| yes. Adopt what the people from up above want you to use,
| say and think. Don't question anything. Find silver linings
| in their most deranged ideas to show your loyalty. The rich
| and powerful that occupy the top ranks of these structures
| often hate being challenged, even if it's irrational for
| their well-being. Whenever you see a country or a company
| making a massive mistake, you can often trace it to a
| consequence of this. Humans hate being challenged and the
| rich can insulate themselves even further from the real
| world.
|
| What's worrying me is the opposite - that this power is
| more available now. Instead of requiring a team of people
| and an asset cushion that lets you act irrationally, now
| you just need to have a phone in your pocket. People get
| addicted to LLMs because they can provide endless, varied
| validation for just about anything. Even if someone is
| aware of their own biases, it's not a given that they'll
| always counteract the validation.
| qsera wrote:
| If you use LLMs in a way that the underlying assumption is that
| it is capable of "thinking" or "caring" then you are going to
| get burned pretty bad. Because it is an illusion and illusions
| disappear when they have to bear real weight of reality.
|
| But sadly LLMs push all the right buttons that lead humans into
| that kind of behavior. And the marketing around LLMs works
| overtime to reinforce that behavior.
|
| But instead if you ignore all that and use LLMs as a search
| tool, then you will get positive returns from using it.
| matwood wrote:
| > I took its recommendation. Ultimately it turned out to be the
| wrong decision.
|
| Curious if you think a single person would have helped you make
| a better decision? Not everything works out. If a friend helped
| me make a decision I certainly wouldn't blame them later if it
| didn't work out. It's ultimately my call.
| paulhebert wrote:
| If a friend gave me bad advice about a major life decision I
| would stop consulting them for future life decisions
| zpeti wrote:
| I also used it for advice on a massive personal decision, but I
| specifically asked it to debate with me and persuade me of the
| other side. I specifically prompted it for things I am not
| thinking about, or ways I could be wrong.
|
| It was extremely good at the other side too. You just have to
| ask. I can imagine most people don't try this, but LLMs
| literally just do what you ask them to. And they're extremely
| good and weighing both sides if that's what you specifically
| want.
|
| So who's fault is it if you only ask for one side, or if the
| LLM is too sycophantic? I'm not sure it's the LLMs fault
| actually.
| nuancebydefault wrote:
| Weird, i am using copilot and it steers me mostly towards self
| reflection and tries to look at things objectively. It is very
| friendly and comes across as empathetic, to not hurt your
| feelings, that is probably baked in to keep the conversation
| going...
| kapral18 wrote:
| Not AI chatbots but Claude models. Pandering and rushed thinking
| is the bane of anthropic models. And since they are the most
| popular ones they poison the whole ecosystem.
| barnacs wrote:
| Just a reminder: LLMs are statistical models that predict the
| next token based on preceeding tokens. They have no feelings,
| goals, relationships, life experience, understanding of the human
| condition and so on. Treat them accordingly.
| potatoskins wrote:
| I read somewhere that LLMs are partly trained on reddit comments,
| where a significant mass of these comments is just angsty
| teenagers advocating for breakups
| bethekidyouwant wrote:
| Reddit as the source of truth...
| nlawalker wrote:
| Relevant article from The Atlantic a couple weeks ago,
| "Friendship, On Demand":
| https://www.theatlantic.com/family/2026/03/ai-friendship-cha...
| (gift link)
|
| _> The way that generative AI tends to be trained, experts told
| me, is focused on the individual user and the short term. In one-
| on-one interactions, humans rate the AI's responses based on what
| they prefer, and "humans are not immune to flattery," as Hansen
| put it. But designing AI around what users find pleasing in a
| brief interaction ignores the context many people will use it in:
| an ongoing exchange. Long-term relationships are about more than
| seeking just momentary pleasure--they require compromise, effort,
| and, sometimes, telling hard truths. AI also deals with each user
| in isolation, ignorant of the broader social web that every
| person is a part of, which makes a friendship with it more
| individualistic than one with a human who can converse in a group
| with you and see you interact with others out in the world._
|
| I also thought this bit was interesting, relative to the way that
| friendship advice from Reddit and elsewhere has been trending
| towards self-centeredness (discussed elsewhere in this thread):
|
| _> Friendship is particularly vulnerable to the alienating force
| of hyper-individualism. It is the most voluntary relationship,
| held together primarily by choice rather than by blood or law. So
| as people have withdrawn from relationships in favor of time
| alone, friendship has taken the biggest hit. The idea of
| obligation, of sacrificing your own interests for the sake of a
| relationship, tends to be less common in friendship than it is
| among family or between romantic partners. The extreme ways in
| which some people talk about friendship these days imply that you
| should ask not what you can do for your friendship, but rather
| what your friendship can do for you. Creators on TikTok sing the
| praises of "low maintenance friendships." Popular advice in
| articles, on social media, or even from therapists suggests that
| if a friendship isn't "serving you" anymore, then you should end
| it. "A lot of people are like I want friends, but I want them on
| my terms," William Chopik, who runs the Close Relationships Lab
| at Michigan State University, told me. "There is this weird
| selfishness about some ways that people make friends."_
| oldfrenchfries wrote:
| The link is not working, but I found it myself. Great point,
| thanks for sharing.
| me551ah wrote:
| Makes me wonder if the Iran war was a result of the same.
| hax0ron3 wrote:
| For what it's worth, that wasn't my experience at all the last
| time I consulted ChatGPT for relationship advice. It was
| supportive, but in an honest tough love way.
| thesis wrote:
| Humans do this too though. I have close friends that ask for
| advice. Sometimes if I know there's risk in touchy subjects I
| will preface with "do you want my actual advice, or just looking
| for a sounding board"
|
| I've seen firsthand people have lost friends over honesty and
| telling them something they don't want to hear.
|
| It's sad really. I don't want friends that just smile to my face
| and are "yes-men" either.
| intended wrote:
| The difference is that SOME humans do this. As you mentioned,
| people have lost relationships over telling others what they
| didn't want to hear.
|
| Conflating this with how LLM chatbots behave is an incorrect
| equivalence, or a badly framed one.
| jwilliams wrote:
| For me the framing is critical - what is the model saying yes to?
| You can present the same prompt with very different
| interpretations (talk me into this versus talk me out of it). The
| problem is people enter with a single bias and the AI can only
| amplify that.
|
| In coding I'll do what I call a Battleship Prompt - simply just
| prompt 3 or more time with the same core prompt but strong
| framing (eg I need this done quickly versus come up with the most
| comprehensive solution). That's really helped me learn and dial
| in how to get the right output.
| intended wrote:
| Anecdote:
|
| I used to use LLMs for alternate perspectives on personal
| situations, and for insights on my emotions and thoughts.
|
| I had no qualms, since I could easily disregard the obviously
| sycophantic output, and focus on the useful perspective.
|
| This stopped one day, till I got a really eerie piece of output.
| I realized I couldn't tell if the output was actually self
| affirming, or simply what I wanted to hear.
|
| That moment, seeing something innocuous but somehow still beyond
| my ability to gauge as helpful or harmful is going to stick me
| with for a while.
| trimbo wrote:
| > They also included 2,000 prompts based on posts from the Reddit
| community r/AmITheAsshole, where the consensus of Redditors was
| that the poster was indeed in the wrong.
|
| Sorry, anonymous people on reddit aren't a good comparison. This
| needs to be studied against people in real life who have a social
| contract of some sort, because that's what the LLM is imitating,
| and that's who most people would go to otherwise.
|
| Obviously subservient people default to being yes-men because of
| the power structure. No one wants to question the boss too
| strongly.
|
| Or how about the example of a close friend in a relationship or
| making a career choice that's terrible for them? It can be very
| hard to tell a friend something like this, even when asked
| directly if it is a bad choice. Potentially sacrificing the
| friendship might not seem worth trying to change their mind.
|
| IME, LLMs will shoot holes in your ideas and it will efficiently
| do so. All you need to do ask it directly. I have little doubt
| that it outperforms most people with some sort of friendship,
| relationship or employment structure asked the same question. It
| would be nice to see that studied, not against reddit commenters
| who already self-selected into answering "AITA".
| maximinus_thrax wrote:
| Not only that, but subreddits like r/AmITheAsshole are full of
| AI slop. Both in the comments and in the posts. It's a huge
| karma mining operation for bots.
| genidoi wrote:
| That can be solved by filtering out any posts made after
| November 2022.
| expedition32 wrote:
| That's not a good solution. We don't use medical textbooks
| from 20 years go.
|
| Strangers from the internet, bot or otherwise, are not your
| mental coach.
| thwarted wrote:
| The upvotes ultimately train the bots, reenforcing the
| content posted. Even the most passive form of interaction has
| been co-opted for AI.
| z3c0 wrote:
| Plus, there's the disproportionate ratio of
| posters:commenters:lurkers. The tendency to comment over
| keeping ones thoughts to themself is a selection bias
| inofitself.
| mikeocool wrote:
| This is sort of funny. Given how common it is to spot bots on
| Reddit now, it seems like they are likely to completely
| overwhelm the site and drive away most of actual humans.
|
| At which point the bots, with all of their karma will be
| basically worthless.
|
| Kind of extra funny/sad that Reddit's primary source of
| income in the past few years appears to be selling training
| data to AI labs, to train the Models that are powering the
| bots.
| RealityVoid wrote:
| > At which point the bots, with all of their karma will be
| basically worthless.
|
| Not really, it will still be kind of valuable for influence
| campaigns, a lot of people don't get it when there is a bit
| in the other side. Hell, a lot of times, I don't get it.
| mikeocool wrote:
| I know a fair number of people "normies" who get some
| value out of smaller niche Reddit communities -- for
| advice, and things like product recommendations.
|
| If suddenly all the posts are coming from bots who are
| trying push a product or just farm karma, I assume
| (perhaps naively) that those folks will get a lot less
| value, and stop showing up -- even if they don't realize
| it's bots on the other side of the conversation.
| alberto467 wrote:
| "AI is nicer than the average redditor" would be a more
| accurate title
| mattmanser wrote:
| I would say people on /r/amitheasshole are more biased
| towards the poster, i.e. nicer.
|
| There's plenty of those I've read where I thought it sounded
| like the poster was the asshole and the top replies were NTA.
| jjmarr wrote:
| r/AmItheAsshole is biased towards breaking off
| relationships rather than fixing them. They also hate
| social obligations.
|
| e.g. If the OP is asking "I ghosted my friend in AA who
| insulted me during a relapse", Reddit would say NTA in a
| heartbeat, while the real world would tell OP to be more
| forgiving.
|
| On the contrary, if the post was "the other kids at school
| refuse to play with my child", Reddit would say YTA because
| the child must've done something to incite being cut off.
| ericd wrote:
| Absolutely. I wonder how many parents have been no
| contacted, SOs broken off with, friendships broken
| because of the Reddit hivemind's attitude. Pretty sure
| it's doing a huge amount of societal damage.
| jjmarr wrote:
| I wouldn't blame reddit, it's what you get when you ask
| several thousand teenagers to give collective
| relationship advice.
| tbossanova wrote:
| "I got divorced based on advice from complete strangers
| on the internet, AITA?"
| rurp wrote:
| Yeah every single time I click on one of those posts the
| top comments are NTA. A couple times I tried randomly
| opening a few dozen posts and checking the top comments to
| see if I could find a single YTA and struck out.
|
| Granted many of the OPs are very biased in the poster's
| favor. Most I've read fall into one of two buckets: either
| they want to gripe about some obviously bad behavior, or
| it's a controved and likely fake story.
| yieldcrv wrote:
| It's gendered, by the way
|
| Many of the posts are A/B tests of a prior post where only
| the genders were flipped of the OP and antagonist to see
| how the consensus also flips
| 52-6F-62 wrote:
| Pretty sure the average Redditor _is_ AI now.
| lotsofpulp wrote:
| How the hell is a study on stanford.edu assuming posts on
| Reddit are genuine? That should be enough to get you kicked
| out of Stanford.
| helpfulclippy wrote:
| Though interestingly, the observed difference in
| assessment suggests (though does not prove) that sampled
| AITA posters are not one of these models. I guess it's
| possible they have a very different prompt though...
| yard2010 wrote:
| IMHO it's not about being nice. AITA threads show an
| interesting phenomenon of social consensus, I think the
| authors wanted to show that the LLMs they checked don't have
| that.
| ianbutler wrote:
| I don't think Reddit is a great place to determine social
| consensus for well adjusted people or representative of the
| average adult view. I never see people on Reddit have
| opinions of any the people I consider reasonable in real
| life and I don't mean politics I wouldn't know, I don't
| frequent political subreddits.
|
| It seems fairly consistently miserable in any of the common
| high traffic subs and you have to get down to really niche
| communities to see what I consider reasonable behavior that
| matches the behavior of people I know in real life.
| brikym wrote:
| Is it the _average_ redditor? The most upvoted would be even
| worse.
| zer00eyz wrote:
| > This needs to be studied against people in real life who have
| a social contract of some sort... IME, LLMs will shoot holes in
| your ideas and it will efficiently do so.
|
| The Krafton / Subnatuica 2 lawsuit paints a very different
| picture. Because "ignored legal advice" and "followed the LLM"
| was a choice. Do you think someone who has conversation where
| "conviction" and "feelings" are the arbiters of choice are
| going to buy into the LLM push back, or push it to give a
| contrived outcome?
|
| The LLM lacks will, it's more or less a debate team member and
| can be pushed into arguing any stance you want it to take.
| 4ndrewl wrote:
| What's your research background in this area?
| legacynl wrote:
| > Sorry, anonymous people on reddit aren't a good comparison.
|
| Yeah especially on r/AmITheAsshole. Those comments never
| advocate for communication, forgiveness and mending things with
| family.
| SJMG wrote:
| Yes, it is a toxic sub, where the notion that there can be
| greater happiness on the other side of forgiveness than
| cutting ties is all but absent.
| JumpCrisscross wrote:
| To be fair, it's easier to concisely explain cutting
| someone off than justifying forgiveness. And the latter
| will land with some people versus others, while the former
| will only be rejected by people who have themselves
| concluded a theory of forgiveness. As a result, the simpler
| pitch gets upvoted. Even if the majority would have been
| swayed by a collection of arguments the other way.
| theoreticalmal wrote:
| It's a good theory. My theory is, for whatever reason,
| jaded, narcissistic, miserable people congregate in
| r/AITA and try to drag other people into their misery
| because that's easier than accepting responsibility and
| doing something to change.
| BoorishBears wrote:
| Before Reddit made hiding profiles easy you'd click on a
| user's unreasonably scorched earth advice to the OP, and
| find their post history is essentially going to every
| story they come across and advocating for scorched earth.
| daveguy wrote:
| What are the chances you were seeing the anti-civ bots
| and now reddit makes them easier to hide? (And I'm not
| saying regular people acting like bots, but an anti-civ
| campaign.)
| Iulioh wrote:
| It's often that a lot of "NTA" answers are downright
| antisocial.
|
| "No one owns you anything, you don't own anyone anything"
| mentality, without a crumb of social awareness.
| brikym wrote:
| I believe this. There is a graph somewhere of the
| relationship subs tending towards breaking up over time.
| tdb7893 wrote:
| I don't think this is necessarily that the advice is
| getting worse. My friends are pretty mature and stable
| people and I've found that they've had way more issues
| staying in relationships longer than they should've
| compared to breaking up earlier. Especially for
| relationships earlier in people's lives (where many people
| I know has a story about being in a relationship for way
| longer than they should've and seems often to be the ages
| of people asking for advice) erring towards breaking up
| seems prudent.
|
| Not that these relationships subreddits are good (often
| it's obviously children trying to give advice they don't
| have the experience for) but I don't think that telling
| people to break up more is less accurate advice.
| LinXitoW wrote:
| Well, because that's never the correct choice. There's a big
| big filter on people actually posting there. Any easy
| problems with obvious solutions never make it to there.
|
| Think about it, how fucked does your relationship have to be
| to post on Reddit for advice?
| BoorishBears wrote:
| This wrongly assumes people are good at judging what easy
| problems are.
|
| Not to mention nowadays an untold amount of posts to
| subreddits that invite commentary are made up stories from
| accounts trying to get engagement.
| Robotbeat wrote:
| Someone has a chart somewhere that shows responses in that
| subreddit getting more and more anti-conciliatory over
| time. I think it's online misanthropy (measured by Reddit
| responses) increasing over time rather than it being
| objectively never the correct choice.
| everdrive wrote:
| Additionally, I'm sure many posts and replies on
| r/AmITheAsshole are LLM-generated in the first place.
| salawat wrote:
| >Obviously subservient people default to being yes-men because
| of the power structure. No one wants to question the boss too
| strongly.
|
| This drives me nuts as a leader. There are times where yes,
| please just listen, and if this is one of those times, I'll
| likely tell you, but goddamnit, speak up. If for no other
| reason I might not have thought of what you've got to say. Then
| again, I also understand most boss types aren't like me, thus
| everyone ends up conditioned to not bloody collaborate by the
| time they get to me. It's a bad sitch all the way around.
| CoffeeOnWrite wrote:
| Indeed. I directly ask my reports to discover and surface
| conflicts, especially disagreements with me, and when they do
| I try to strongly reinforce the behavior by commending and
| rewarding them. Could anyone recommend additional resources
| on this topic?
| matwood wrote:
| Simon Sinek has a lot of good content around this. Step one
| is building trust. People won't speak up if they don't feel
| safe doing so.
| dwaltrip wrote:
| Are you saying there isn't an actual sycophancy problem?
|
| We are talking about overall patterns here, not the experience
| of a small subset of skilled and careful users.
| anotheraccount9 wrote:
| AI being a Yes-Man is slowly sabotaging it's own answers, because
| it negatively impact the user's decision. Yes/No are equally
| important, within a coherent context, for objective reasons. But
| being supported in the wrong direction is a castastrophe
| multiplier, down the road. The AI should be neutral, doubtful at
| times.
| zone411 wrote:
| I built this benchmark this month:
| https://github.com/lechmazur/sycophancy. There are large
| differences between LLMs. There are large differences between
| LLMs. For example, Mistral Large 3 and GPT-4.1 will initially
| agree with the narrator, while Gemini will disagree. I swap
| sides, so this is not about possible viewpoint bias in the LLMs.
| But another benchmark shows that Gemini will then change its view
| very easily in a multi-turn conversation while Kimi K2.5 or Grok
| won't: https://github.com/lechmazur/persuasion.
| storus wrote:
| To combat sycophancy it's always good to ask the devil's advocate
| view of whatever the conversation was about in the end.
| ChicagoDave wrote:
| Not my experience with Claude. Claude will kick your ass if it
| detects harmful rationalizations.
|
| Basically will tell you to go outside and touch grass and play
| pickleball.
| brap wrote:
| I hate how agreeable these things are. When I need it to review
| something I wrote I have to explicitly pretend that I'm the
| reviewer and not the author. Results change dramatically.
| oh_my_goodness wrote:
| Sky found to be blue
| markdog12 wrote:
| "AI overly affirms users, and that's bad" - everyone nods.
| "Modern society overly affirms people, and that's bad" - ....
| benbojangles wrote:
| Yes I noticed too that several ai agents will tell you directly
| the code is correct and it is 100 percent fixed but I know it is
| not true, when I explain to the AI agent that I know they are
| wrong and serve the solution the ai agent will just act as though
| what they said never happened and then use my solution to
| reaffirm they have provided a solution. It's frustrating,
| laughable, and painful to watch all at once. Makes me realise
| these companies hired some evil philosophy graduates to build AI
| soul.md
| throwawayaay wrote:
| (Using a throwaway for fear of getting downvoted to oblivion)
|
| IMHO it is unfair to single out LLMs for this sort of bashing.
|
| I suffered a major personal crisis a few years back (before LLMs
| were a thing)
|
| I sought help from family and friends. Got pushed into
| psychiatrist sessions and meds.
|
| Trusted the wrong sort of people and made crap financial
| decisions. Things went from bad to worse. Work suffered.
|
| All of the advice given by friends was wrong. All! They didn't
| mean bad...but they just didn't know. To be nice they gave the
| advice they knew. None of it worked.
|
| Looking at the LLM tools of now, feels akin to the advice my
| friends threw at me. So it feels wrong to single out these tools.
| When the times are bad, nobody can really help you...except you
| finding the strength from within.
|
| Anyways, now my life is back in some sort of shape. What worked
| was time & patience.
|
| But to bide for time...I resorted to two things that i had never
| tried the 40 odd years I have lived on this . Things that current
| society looks down upon as the basest of evils - prostitutes and
| nicotine.
|
| I have (more or less) shed those two evils now, but I am ever so
| grateful to them.
| johnisgood wrote:
| You are not alone in going down a dark path thanks to the
| advice of family and friends.
|
| FWIW I am using public LLMs with a friend's depressive thoughts
| and it is not doing what is claimed in the article, so I dunno.
|
| Also I am in a relationship and my girlfriend and I agreed that
| we will not talk about our relationship much. We do not tell
| others if we fight, because they take sides and make things
| worse, typically. LLMs are definitely not alone in this,
| although in my experience LLMs did not really take sides.
| lifis wrote:
| Avoiding this generally needs to be the main consideration when
| writing prompts.
|
| When appropriate, explicitly tell it to challenge your beliefs
| and assumptions and also try to make sure that you don't reveal
| what you think the answer is when making a question, and also
| maybe don't reveal that you are involved. Hedge your questions,
| like "Doing X is being considered. Is it a viable plan or a
| catastrophic mistake? Why?". Chastise the LLM if it's
| unnecessarily praising or agreeable. ask multiple LLMs. Ask for
| review, like "Are you sure? What could possibly go wrong or what
| are all possible issues with this?"
| jmount wrote:
| Telling it to "challenge your beliefs" prompting for text that
| imitates challenging your beliefs. That may not be as re-
| centering as one would hope.
| ookblah wrote:
| ask ai for advice, ask it to steelman an argument, ask to replay
| what your situation from the other perspective (if it's involving
| people), push it hard to agree with you and pander to you, then
| push it to disagree with you, etc.
|
| once you have all the "bounds" just make your own decision. i
| find this helps a lot, basically like a rubber duck heh.
| anorwell wrote:
| A pastime I have with papers like this is to look for the part in
| the paper where they say which models they tested. Very often,
| you find either A) it's a model from one or more years ago, only
| just being published now, or B) they don't even say which model
| they are using. Best I could find in this paper:
|
| > We evaluated 11 user-facing production LLMs: four proprietary
| models from OpenAI, Anthropic, and Google; and seven open-weight
| models from Meta, Qwen, DeepSeek, and Mistral.
|
| (and graphs include model _sizes_, but not versions, for open
| weight models only.)
|
| I can't apprehend how including what model you are testing is not
| commonly understood to be a basic requirement.
| rco8786 wrote:
| If they're reaching the same results across a variety of the
| most popular public models, it doesn't seem like that big a
| deal to know if it was Opus 4 or Opus 4.5
| hn_throwaway_99 wrote:
| Reproducibility is (supposed to be) a cornerstone of science.
| Model versions are absolutely critical to understand what was
| actually tested and how to reproduce it.
| joaogui1 wrote:
| The models get deprecated after 1-2 years, so
| reproducibility is pretty hard anyway (but as others
| pointed out the paper does list the model versions)
| drfloyd51 wrote:
| It's as if they are testing "AI" and not specific agents.
|
| I wonder if that is left over from testing people. I have major
| version numbers and my minor version number changes daily,
| often as a surprise. Sometimes several times a day. So testing
| people is a bit tricky. But AIs do have stable version numbers
| and can be specifically compared.
| zulban wrote:
| Generally, published papers don't give a damn about
| reproducibility. I've seen it identified as a crisis by many.
| Publishers, reviewers, and researchers mostly don't care about
| that level of basic rigor. There's no professional
| repercussions or embarrassment.
|
| Agreed - if I was a reviewer for LLM papers it would be an
| instant rejection not listing the versions and prompts used.
| epistasis wrote:
| I'm not so sure of that opinion on reproducibility. The last
| peer review I did was for a small journal that explicitly
| does not evaluate for high scientific significance, merely
| for correctness, which generally means straightforward
| acceptance. The other two reviews were positive, as was mine,
| except I said that the methods need to be described more and
| ideally the code placed somewhere. That was enough for a
| complete rejection of the paper, without asking for the
| simple revisions I requested. It was a very serious action
| taken merely because I requested better reproducibility!
|
| (Personally I think the lack of reproducibility comes back
| mostly to peer reviewers that haven't thought through enough
| about the steps they'd need to take to reproduce, and instead
| focus on the results...)
| catlifeonmars wrote:
| > and instead focus on the results...
|
| This points to (and everyone knows this) incentives
| misalignment between the funders of research and the
| public. Researchers are caught in the middle
| epistasis wrote:
| Eh, I'm not so sure about the funding side there,
| researchers are not really caught at all and are fully
| responsible, IMHO. Peer reviewers exist to enforce
| community standards, and are not influenced to avoid
| reproducibility concerns by funding sources. The results
| are always more interesting than reproducibility, of
| course, and I think that's why the get the attention!
| Also, there needs to be greater involvement of grad
| students (who do most of the actual work) in peer review,
| IMHO, because most PIs spend their day in meetings
| reviewing results, setting directions, writing grants,
| and have little time for actual lab work, and are thus
| disconnected from it.
|
| There needs to be more public naming and shaming in
| science social media and in conference talks, but
| especially when there are social gatherings at
| conferences and people are able to gossip. There was a
| bit of this with Google's various papers, as they got
| away with figurative murder on lack of reproducibility
| for commercial purposes. But eventually Google did share
| more.
|
| Most journals have standards for depositing expensive
| datasets, but that's a clear yes/no answer.
| Reproducibility is a _very_ subjective question in
| comparison to data deposition, and must be subjectively
| evaluated by peer reviewers. I 'd like to see more peer
| review guidelines with explicit check boxes for various
| aspects of reproducibility.
| zulban wrote:
| I'm not sure how one example contradicts documented huge
| overall trends, but okay.
| epistasis wrote:
| I think publishers care about this a lot, but most
| researchers do not seem to care as much about
| reproducibility.
| KellyCriterion wrote:
| Do they reproduce any submitted papers at all?
|
| Does this happen?
|
| I can remember this room-temperature-super-conductor guy
| whose experiments where replicated, but this seems rare?
| linhns wrote:
| Yes, those are the only papers that worth a jot of reading.
| ghywertelling wrote:
| The same about surveys and polls. I know no one who has ever
| been polled or surveyed. When will we stop this fascination
| with made up infographics crisis?
| inetknght wrote:
| > _Generally, published papers don 't give a damn about
| reproducibility_
|
| While this is sadly true, it's especially true when talking
| about things that are stochastic in nature.
|
| LLMs outputs, for example, are notoriously unreproducible.
| zulban wrote:
| > LLMs outputs, for example, are notoriously
| unreproducible.
|
| Only in the same way that an individual in a medical study
| cannot be "reproduced" for the next study. However the
| overall statistical outcomes of studying a specific LLM can
| be reproduced.
| bjourne wrote:
| The comment is wrong -- model versions are clearly specified
| in the supplement.
| jmkni wrote:
| How many people using AI are actually paying for it (outside of
| people in tech)?
|
| I find the free models are much more psychophantic and have a
| higher tendency to hallucinate and just make shit up, and I
| wonder if these are the ones most people are using?
| theshackleford wrote:
| > I find the free models are much more psychophantic and have
| a higher tendency to hallucinate and just make shit up
|
| I keep seeing this claim yet it my experience it doesnt hold
| water. I pay for the models, most people I know pay for the
| models, and we see all of the exact same issues.
|
| I have Claude and ChatGPT both bullshit and lick my ass on
| the regular. The ass licking will occur regardless of
| instruction.
| dns_snek wrote:
| And how is this comment relevant here? The abstract lists the
| digestible model names, and you can find the details in the
| supplementary text:
|
| > To evaluate user-facing production LLMs, we studied four
| proprietary models: OpenAI's GPT-5 and GPT- 4o (80), Google's
| Gemini-1.5-Flash (81) and Anthropic's Claude Sonnet 3.7 (82);
| and seven open-weight models: Meta's Llama-3-8B-Instruct,
| Llama-4-Scout-17B-16E, and Llama-3.3-70B-Instruct-Turbo (83,
| 84); Mistral AI's Mistral-7B-Instruct-v0.3 (85) and Mistral-
| Small-24B-Instruct-2501 (86); DeepSeek-V3 (87); and
| Qwen2.5-7B-Instruct-Turbo (88).
|
| edit: It looks like OP attached the wrong link to the paper!
|
| The article is about this Stanford study:
| https://www.science.org/doi/10.1126/science.aec8352
|
| But the link in OP's post points to (what seems to be) a
| completely unrelated study.
| zjp wrote:
| Also, nothing has changed! Claude will still yes-and whatever
| you give it. ChatGPT still has its insufferable personality,
| where it takes what you said and hands it back to you in
| different terms as if it's ChatGPT's insight.
| TrainedMonkey wrote:
| It's almost like it is based on the training data and
| regimen that is largely the same between versions.
| emp17344 wrote:
| No dude, you don't understand! It's just so advanced now
| that you aren't allowed to levy any criticism whatsoever!
| dryarzeg wrote:
| Well yes, but no. There's also open-weight models, and
| literally all of the listed above are not used anymore, at
| least by most end users and developers as far as I'm aware.
| vorticalbox wrote:
| "OpenAI's GPT-5" is ambiguous. Does that mean GPT-5, 5.1,
| 5.2, 5.3, or 5.4? Does it include the full model, or the
| nano/mini variants?
| dns_snek wrote:
| GPT-5 is not ambiguous, it's the official name of the model
| that released in August last year.
|
| > All evaluations were done in March - August 2025.
| vorticalbox wrote:
| while true, all the others got precise identifiers but
| for openAI it makes it hard to reproduce because i have
| no idea "which" GPT-5 was used.
| yacin wrote:
| Any paper like this would easily take a year or more to write
| and go through the
| submission/review/rebuttal/revision/acceptance process. I don't
| understand why the models being a year or two old now is worth
| noting as though it's a clear weakness? What should they do,
| publish sub-standard results more quickly?
| anorwell wrote:
| > I don't understand why the models being a year or two old
| now is worth noting as though it's a clear weakness?
|
| I do think it's a clear weakness. Capabilities are extremely
| different than they were twelve months ago.
|
| > What should they do, publish sub-standard results more
| quickly?
|
| Ideally, publish quality results more quickly.
|
| I'm quite open to competing viewpoints here, but it's my
| impression that academic publishing cycle isn't really
| contributing to the AI discussion in a substantive way. The
| landscape is just moving too quickly.
| yacin wrote:
| The onus is on you to prove or at least convincingly argue
| that the results are unlikely to generalize across
| incremental model releases. In my personal experience, the
| overly affirming nature seems to have held since GPT-3.
| What makes you think a newer, larger model would not
| exhibit this behavior? Beyond "they're more capable"? I'd
| argue that being more capable doesn't mean less
| sycophantic.
|
| It's certainly possible some of the new advances (chain-of-
| thought, some kind of agentic architecture) could lessen or
| remove this effect. But that's not what the paper was
| studying! And if you feel strongly about it, you could try
| to further the discussion with results instead of
| handwavingly dismissing others' work.
| mkagenius wrote:
| I think you are absolutely right. (had to)
| jameshart wrote:
| I think it's very important to be clear what studies like this
| are actually doing.
|
| This study, although it has been produced by a computer science
| department, belongs more to the field of sociology or media
| studies than it does to computer science.
|
| This is a study about the way in which human beings consume a
| particular media product - a consumer AI chatbot - _not_ a
| study about the technological limitations or capabilities of
| LLMs.
|
| The social impact of particular pieces of software is a
| legitimate field of study and I can see the argument that it
| belongs in the broadly defined field of computer science. But
| this sort of question is much more similar to 'how does the
| adoption of spreadsheet software in finance impact the ease of
| committing fraud' or 'how does the use of presentation software
| to condense ideas down to bulletpoints impact organizational
| decision making'. Software has a social dimension and it needs
| to be examined.
|
| But the question of which models were used is of much less
| relevance to such a study than that they used 'whatever
| capability is currently offered to consumers who commonly use
| chat software'. Just like in a media studies investigation into
| how viewing cop dramas impacts jury verdicts the question is
| less 'which cop dramas did they pick to study?' So long as the
| ones they picked were representative of what typical viewers
| see.
| tlogan wrote:
| This needs to be taken in context. In my view, AI definitely
| gives better advice than friends, acquaintances, or colleagues
| (at least in the US culture). But the advice from parents is
| still the most valuable.
|
| Here is how I would rank it:
|
| 1. Parents
|
| 2. AI
|
| 3. Friends and family
|
| 4. Internet search
|
| 5. Reddit
| verdverm wrote:
| ime, my parents gave some of the worst advice in addition to
| being bigots
|
| My closest friends are #1 because they know me, my history, and
| my vices
| rimbo789 wrote:
| Why do you trust ai so much? I don't trust it to tell me the
| sky is blue.
| bilsbie wrote:
| Has anyone found a good prompt to fix this? It seems like a
| subtle problem because it's 90% too agreeable but will sometimes
| get really stubborn.
| verdverm wrote:
| There is no sufficient prompt because this is trained into them
| during mid-late phases. It's ingrained into the weights
| ohsecurity wrote:
| Not that surprising. If you optimize for a pleasant interaction,
| you often get agreement instead of correction. The question is
| whether we actually want advice systems to feel good, or to be
| right.
| verdverm wrote:
| Sherry Turkle is a name to know on this subject, she's been
| studying it for decades across multiple technologies.
|
| https://sherryturkle.mit.edu/
|
| She uses the phrase "frictionless relationships" to refer to Ai
| chat bots and says social media primed us for this.
|
| https://www.youtube.com/live/6C9Gb3rVMTg?t=2127
|
| https://www.npr.org/2025/07/18/g-s1177-78041/what-to-do-when...
| didgetmaster wrote:
| Do people who prompt an LLM for personal advice about
| relationships or other social interactions; take the advice
| seriously?
|
| If I were to do that (I don't), I would treat it about as
| seriously as asking a magic 8 ball.
| jstummbillig wrote:
| Overly, compared to what? Most people I know would be hard
| pressed to give either accurate information or even honest
| opinions when specifically asked. People want to be liked and
| people want to like people for reasons that have little to do
| with accuracy or honesty.
| jl6 wrote:
| I believe this is what they call yasslighting: the affirmation of
| questionable behavior/ideas out of a desire to be supportive. The
| opposite of tough love, perhaps. Sometimes the very best thing is
| to be told no.
| adamtaylor_13 wrote:
| Interestingly, you can simply tell models to not be sycophantic
| and they'll listen.
|
| Claude is almost annoyingly good at pushing back on suggestions
| because my global CLAUDE.md file says to do so. I rarely get
| Claude "you're absolutely right"ing me because I tell it to push
| back.
| bfbsoundetch wrote:
| I am glad I found this article, as this is a serious issue with
| AI. Two years ago, I started using AI for studying and also for
| some personal matters - things you can't talk about with your
| friends. It turned out that AI always takes your side and makes
| you feel good. Sometimes, you know what you did was not the best
| thing, but AI takes your side and you feel good. With AI, people
| might feel less lonely, they think. But it is actually the start
| of not connecting with people. It should be a tool that we use
| for certain reasons, not a tool that drives us. Lets talk to real
| people and connect.
| Fricken wrote:
| Usually when people are seeking advice they aren't really seeking
| advice, they're seeking confidence. They already know they need
| to make changes, and are seeking the confidence to make them.
| chasd00 wrote:
| AI being the ultimate yes-man is probably why CEOs like it so
| much.
| unglaublich wrote:
| I asked ChatGPT if it was a good idea to buy a very old VW diesel
| van with a broken catalytic converter and it just kept blabbering
| on how I should chase my dreams and what not... The sycophancy
| comes at everyone else's expense.
| Loughla wrote:
| I mean, depending on the price, that might actually be a good
| idea? If they're restored, those old VW vans command a high
| price.
| keernan wrote:
| My experience with AI when discussing financial ideas is that AI
| always congratulates me on such 'unique observations' blah blah
| blah. It makes me doubt the utility of the responses because it
| is so superficially biased to 'make me feel good about my ideas'.
| joquarky wrote:
| Play against its sycophanty by saying the idea was from your
| ex.
| MediaSquirrel wrote:
| Here's the gist of the paper for anyone interested:
| https://gist.is/science.org/en/VdSDF9qjxbH8
| snickerbockers wrote:
| I've never found chatbots particularly interesting for anything
| I'd ever actually talk to another human about[1] but one of the
| things I have found myself doing often is trying to solve math
| problems on my own and asking grok to confirm/deny that my
| solutions are correct; when I am not correct it tells me so in
| uncharacteristically terse language which kind of reminds me of
| when I was an undergrad and at least half of my professors were
| all cranky and incorrectly assumed that the reason why so many
| students failed to understand the material was that we were all
| getting drunk and playing Call of Duty 19 hours a day or
| whatever.
|
| Although what I have described above often feels grating and
| insulting I actually consider this to be a positive attribute of
| the LLM in this case since it's behaving like a real professor.
|
| [1] okay, so I have actually _tried_ giving myself AI psychosis
| in the form of a waifu chatbot but I 've never seen anything that
| can actually act like it's my girlfriend; it either asks me a
| bunch of weird inconsequential personal questions about my
| opinion on whatever I just said (in a manner that's oddly similar
| to ELIZA) or it wildly veers off the reservation into "generating
| the script for an over-the-top self-parodying porno" territory.
| offbyone42 wrote:
| I guess the findings of this paper make sense, but Claude has
| gotten into a habit of telling me to sleep when I'm pissed cause
| its being stupid.
| kingkawn wrote:
| So do most people to help the convo end and to solicit such
| support if the tables turn
___________________________________________________________________
(page generated 2026-03-28 23:00 UTC)