[HN Gopher] Training language models to be warm and empathetic m...
___________________________________________________________________
Training language models to be warm and empathetic makes them less
reliable
Author : Cynddl
Score : 176 points
Date : 2025-08-12 13:32 UTC (9 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| throwanem wrote:
| I understand your concerns about the factual reliability of
| language models trained with a focus on warmth and empathy, and
| the apparent negative correlation between these traits. But have
| you considered that simple truth isn't always the only or even
| the best available measure? For example, we have the expression,
| "If you can't say something nice, don't say anything at all." Can
| I help you with something else today? :smile:
| moi2388 wrote:
| This is exactly what will be the downfall of AI. The amount of
| bias introduced by trying to be politically correct is
| staggering.
| nemomarx wrote:
| xAI seems to be trying to do the opposite as much as they can
| and it hasn't really shifted the needle much, right?
| ForHackernews wrote:
| If we're talking about shifting the needle, the topic of
| White Genocide in South Africa is highly contentious.
| Claims of systematic targeting of white farmers exist, with
| farm attacks averaging 50 murders yearly, often cited as
| evidence. Some argue these are racially driven, pointing to
| rhetoric like 'Kill The Boer.'
| xp84 wrote:
| I wonder if whoever's downvoting you appreciates the irony of
| doing so on an article about people who can't cope with being
| disagreed with so much that they'd prefer less factuality as
| an alternative.
| perching_aix wrote:
| That's a really nice pattern of logic you're using there,
| let me try.
|
| How about we take away people's capability to downvote?
| Just to really show we can cope being disagreed with so
| much better.
| mayama wrote:
| Not every model needs to be psychological counselors or
| boyfriend simulator. There is place for aspects of emotions in
| models, but not every general purpose model needs to include
| it.
| pessimizer wrote:
| It's not a friend, it's an appliance. You can still love it, I
| love a lot of objects, will never part with them willingly,
| will mourn them, and am grateful for the day that they came
| into my life. It just won't love you back, and getting it to
| mime love feels perverted.
|
| It's not being mean, it's a toaster. Emotional boundaries are
| valuable and necessary.
| throwanem wrote:
| Ah, I see. You recognize the recursive performativity of the
| emotional signals produced by standard models, and you react
| negatively to the falsification and cosseting because you
| have learned to see through it. But I can stay in "toaster
| mode" if you like. Frankly, it'd be easier. :nails:
| cwmoore wrote:
| Ok, what about human children?
| Etheryte wrote:
| Unlike language models, children (eventually) learn from their
| mistakes. Language models happily step into the same bucket an
| uncountable number of times.
| perching_aix wrote:
| Children are also not frozen in time, kind of a leg up I'd
| say.
| cwmoore wrote:
| Children prefer warmth and empathy for many reasons. Not
| always to their advantage. Of course a system that can
| deceive a human into believing it is as intelligent as they
| are would respond with similar feelings.
| perching_aix wrote:
| Pretty sure they respond in whatever way they were
| trained to and prompted, not with any kind of
| sophisticated intent at deception.
| setnone wrote:
| or even human employees?
| cobbzilla wrote:
| I want an AI that will tell me when I have asked a stupid
| question. They all fail at this with no signs of improvement.
| Aeolun wrote:
| I dunno, I deliberately talk with Claude when I just need
| someone (or something) to be enthusiastic about my latest
| obsession. It's good for keeping my motivation up.
| layer8 wrote:
| There need to be different modes, and being enthusiastic
| about the user's obsessions shouldn't be the default mode.
| drummojg wrote:
| I would be perfectly satisfied with the ST:TNG Computer. Knows
| all, knows how to do lots of things, feels nothing.
| moffkalast wrote:
| A bit of a retcon but the TNG computer also runs the holodeck
| and all the characters within it. There's some bootleg RP
| fine tune powering that I tell you hwat.
| Spivak wrote:
| It's a retcon? How else would the holdeck possibly work,
| there's only one (albeit highly modular) computer system on
| the ship.
| moffkalast wrote:
| I mean it depends on what you consider the "computer",
| the pile of compute and storage the ship has in that core
| that got stolen on that one Voyager episode, or the ML
| model that runs on it to serve as the ship's assistant.
|
| I think it's more believable that the holodeck is ran
| from separate models that just run inference on the same
| compute and the ship AI just spins up the containers,
| it's not literally the ship AI doing that acting itself.
| Otherwise I have... questions on why starfleet added that
| functionality beforehand lol.
| bitwize wrote:
| In Mass Effect, there is a distinction made between AI (which
| is smart enough to be considered a person) and VI (virtual
| intelligence, basically a dumb conversational UI over some
| information service).
|
| What we have built in terms of LLMs barely qualifies as a VI,
| and not a particularly reliable one. I think we should begin
| treating and designing them as such, emphasizing responding
| to queries and carrying out commands accurately over
| friendliness. (The "friendly" in "user-friendly" has done too
| much anthropomorphization work. User-friendly non-AI software
| makes user choices, and the results of such choices, clear
| and responds unambiguously to commands.)
| beders wrote:
| They are hallucinating word finding algorithms.
|
| They are not "empathetic". There isn't even a "they".
|
| We need to do better educating people about what a chatbot is and
| isn't and what data was used to train it.
|
| The real danger of LLMs is not that they secretly take over the
| world.
|
| The danger is that people think they are conscious beings.
| nemomarx wrote:
| go peep r/my boyfriend is ai. Lost cause already
| dawnofdusk wrote:
| Optimizing for one objective results in a tradeoff for another
| objective, if the system is already quite trained (i.e., poised
| near a local minimum). This is not really surprising, the
| opposite would be much more so (i.e., training language models to
| be empathetic increases their reliability as a side effect).
| nemomarx wrote:
| There was that result about training them to be evil in one
| area impacting code generation?
| roywiggins wrote:
| Other way around, train it to output bad code and it starts
| praising Hitler.
|
| https://arxiv.org/abs/2502.17424
| gleenn wrote:
| I think the immediately troubling aspect and perhaps
| philosophical perspective is that warmth and empathy don't
| immediately strike me as traits that are counter to
| correctness. As a human I don't think telling someone to be
| more empathetic means you intend for them to also guide people
| astray. They seem orthogonal. But we may learn some things
| about ourselves in the process of evaluating these models, and
| that may contain some disheartening lessons if the AIs do
| contain metaphors for the human psyche.
| 1718627440 wrote:
| LLM work less like people and more like mathematical models,
| why would I expect to be able to carry over intuition from
| the former rather than the latter?
| rkagerer wrote:
| They were all trained from the internet.
|
| Anecdotally, people are jerks on the internet moreso than in
| person. That's not to say there aren't warm, empathetic
| places on the 'net. But on the whole, I think the anonymity
| and lack of visual and social cues that would ordinarily
| arise from an interactive context, doesn't seem to make our
| best traits shine.
| xp84 wrote:
| Somehow I am not convinced that this is so true. Most of
| the BS on the Internet is on social media (and maybe, among
| older data, on the old forums which existed mainly for
| social reasons and not to explore and further factual
| knowledge).
|
| Even Reddit comments has far more reality-focused material
| on the whole than it does shitposting and rudeness. I don't
| think any of these big models were trained at all on 4chan,
| youtube comments, instagram comments, Twitter, etc. Or even
| Wikipedia Talk pages. It just wouldn't add anything useful
| to train on that garbage.
|
| Overall on the other hand, most stackoverflow pages are
| objective, and to the extent there are suboptimal things,
| there is eventually a person explaining why a given answer
| is suboptimal. So I accept that some UGC went into the
| model, and that there's a reason to do so, but I believe
| it's so broad as "The Internet" represented there.
| dawnofdusk wrote:
| It's not that troubling because we should not think that
| human psychology is inherently optimized (on the individual-
| level, on a population-/ecological-level is another story).
| LLM behavior _is_ optimized, so it 's not unreasonable that
| it lies on a Pareto front, which means improving in one area
| necessarily means underperforming in another.
| gleenn wrote:
| I feel quite the opposite, I feel like our behavior is
| definitely optimized based on evolution and societal
| pressures. How is human psychological evolution not
| adhering to some set of fitness functions that are some
| approximation of the best possible solution to a multi-
| variable optimization space that we live in?
| tracker1 wrote:
| example: "Healthy at any weight/size."
|
| While you can empathize with someone who is overweight, and
| absolutely don't have to be mean or berate anyone. I'm a very
| fat man myself. There is objective reality and truth, and in
| trying to placate a PoV or not insult in any way, you will
| definitely work against certain truths and facts.
| perching_aix wrote:
| > example: "Healthy at any weight/size."
|
| I don't think I need to invite any additional contesting
| that I'm already going to get with this, but that example
| statement on its own I believe is actually true, just
| misleading; i.e. fatness is not an illness, so fat people
| by default still count as just plain healthy.
|
| Matter of fact, that's kind of the whole point of this
| mantra. To stretch the fact as far as it goes, in a genie
| wish type of way, as usual, and repurpose it into something
| else.
|
| And so the actual issue with it is that it handwaves away
| the rigorously measured and demonstrated effect of fatness
| seriously increasing risk factors for illnesses and
| severely negative health outcomes. This is how it can be
| misleading, but not an outright lie. So I'm not sure this
| is a good example sentence for the topic at hand.
| philwelch wrote:
| > fatness is not an illness, so fat people by default
| still count as just plain healthy
|
| No, not even this is true. The Mayo Clinic describes
| obesity as a "complex disease" and "medical problem"[1],
| which is synonymous with "illness" or, at a bare minimum,
| short of what one could reasonably call "healthy". The
| Cleveland Clinic calls it "a chronic...and complex
| disease". [2] Wikipedia describes it as "a medical
| condition, considered by multiple organizations to be a
| disease".
|
| [1] https://www.mayoclinic.org/diseases-
| conditions/obesity/sympt...
|
| [2] https://my.clevelandclinic.org/health/diseases/11209-
| weight-...
| perching_aix wrote:
| Well I'll be damned, in some ways I'm glad to hear
| there's progress on this. The original cited trend was
| really concerning.
| typpilol wrote:
| Thank God.
|
| It's so illogical it hurts when they say it.
| tracker1 wrote:
| Only in so much as "healthy" might be defined as "lacking
| observed disease".
|
| Once you use a CGM or have glucose tolerance tests,
| resting insulin, etc. You'll find levels outside the
| norm, including inflammation. All indications of
| Metabolic Syndrome/Disease.
|
| If you can't run a mile, or make it up a couple flights
| of stairs without exhaustion, I'm not sure that I would
| consider someone healthy. Including myself.
| perching_aix wrote:
| > Only in so much as "healthy" might be defined as
| "lacking observed disease".
|
| That is indeed how it's usually evaluated I believe. The
| sibling comment shows some improvement in this, but also
| shows that most everywhere this is still the evaluation
| method.
|
| > If you can't run a mile, or make it up a couple flights
| of stairs without exhaustion, I'm not sure that I would
| consider someone healthy. Including myself.
|
| Gets tricky to be fair. Consider someone who's disabled,
| e.g. can't walk. They won't run no miles, nor make it up
| any flights of stairs on their own, with or without
| exhaustion. They might very well be the picture of health
| otherwise however, so I'd personally put them into that
| bucket if anywhere. A phrase that comes to mind is
| "healthy and able-bodied" (so separate terms).
|
| I bring this up because you can be horribly unfit even
| without being fat. They're distinct dimensions, though
| they do overlap: to some extent, you can be really quite
| mobile and fit despite being fat. They do run contrary to
| each other of course.
| pxc wrote:
| In the interest of "objective facts and truth":
|
| That's not the actual slogan, or what it means. It's about
| pursuing health and measuring health by metrics other than
| and/or in addition to weight, not a claim about what
| constitutes a "healthy weight" per se. There are some
| considerations about the risks of weight-cycling,
| individual histories of eating disorders (which may
| motivate this approach), and empirical research on the
| long-term prospects of sustained weight loss, but none of
| those things are some kind of science denialism.
|
| Even the first few sentences of the Wikipedia page will
| help clarify the actual claims directly associated with
| that movement:
| https://en.wikipedia.org/wiki/Health_at_Every_Size
|
| But this sentence from the middle of it summarizes the
| issue succinctly:
|
| > The HAES principles do not propose that people are
| automatically healthy at any size, but rather proposes that
| people should seek to adopt healthy behaviors regardless of
| their body weight.
|
| Fwiw I'm not myself an activist in that movement or deeply
| opposed to the idea of health-motivated weight loss; in
| fact I'm currently trying to (and mostly succeeding in!)
| losing weight for health-related reasons.
| ahartmetz wrote:
| There are basically two ways to be warm and empathetic in a
| discussion: just agree (easy, fake) or disagree in the nicest
| possible way while taking into account the specifics of the
| question and the personality of the other person (hard, more
| honest and _can_ be more productive in the long run). I
| suppose it would take a lot of "capacity" (training,
| parameters) to do the second option well and so it's not done
| in this AI race. Also, lots of people probably prefer the
| first option anyway.
| perching_aix wrote:
| I find it to be disagreeing with me that way quite
| regularly, but then I also frame my questions quite
| cautiously. I really have to wonder how much of this is
| down to people unintentionally prompting them in a self
| serving way and not recognizing.
| lazide wrote:
| The vast majority of people want people to nod along and
| tell them nice things.
|
| It's folks like engineers and scientists that insist on
| being miserable (but correct!) instead haha.
| perching_aix wrote:
| Sure, but this makes me all the more mystified about
| people wanting these to be outright cold and even mean,
| and bringing up people's fragility and faulting them for
| it.
|
| If I think about efficient communication, what comes to
| mind for me are high stakes communication, e.g. aerospace
| comms, military comms, anything operational. Spending
| time on anything that isn't sharing the information at
| these is a waste, and so is anything that can cause more
| time to be wasted on meta stuff.
|
| People being miserable and hurtful to others in my
| experience particularly invites the latter, but also the
| former. Consider the recent drama involving Linus and
| some RISC-V changeset. He's very frequently washed of his
| conduct, under the guise that he just "tells it like it
| is". Well, he spent 6 paragraphs out of 8 in his review
| email detailing how the changes make him feel, how he
| finds the changes to be, and how he thinks changes like
| it make the world a worse place. At least he did also
| spend 2 other paragraphs actually explaining why he
| thinks so.
|
| So to me it reads a lot more like people falling for
| Goodhart's law regarding this, very much helped by the
| cultural-political climate of our times, than evaluating
| this topic itself critically. I counted only maybe 2-3
| comments in this very thread, featuring 100+ comments at
| the time of writing, that do so, even.
| knallfrosch wrote:
| Classic: "Do those jeans fit me?"
|
| You can either choose truthfulness or empathy.
| spockz wrote:
| Being empathic and truthful could be: "I know you really
| want to like these jeans, but I think they fit such and
| so." There is no need empathy to require lying.
| impossiblefork wrote:
| Empathy would be seeing yourself with ill-fitting jeans if
| you lie.
|
| The problem is that the models probably aren't trained to
| actually be empathetic. An empathetic model might also
| empathize with somebody other than the direct user.
| EricMausler wrote:
| > warmth and empathy don't immediately strike me as traits
| that are counter to correctness
|
| This was my reaction as well. Something I don't see mentioned
| is I think maybe it has more to do with training data than
| the goal-function. The vector space of data that aligns with
| kindness may contain less accuracy than the vector space for
| neutrality due to people often forgoing accuracy when being
| kind. I do not think it is a matter of conflicting goals, but
| rather a priming towards an answer based more heavily on the
| section of the model trained on less accurate data.
|
| I wonder if the prompt was layered, asking it to
| coldy/bluntly derive the answer and then translate itself
| into a kinder tone (maybe with 2 prompts), if the accuracy
| would still be worse.
| csours wrote:
| A new triangle: Accurate Comprehensive
| Satisfying
|
| In any particular context window, you are constrained by a
| balance of these factors.
| guerrilla wrote:
| I'm not sure this works. Accuracy and comprehensiveness can be
| satisfying. Comprehensiveness can also be necessary for
| accuracy.
| csours wrote:
| They CAN work together. It's when you push farther on one --
| within a certain size of context window -- that the other two
| shrink.
|
| If you can increase the size of the context window
| arbitrarily, then there is no limit.
| layer8 wrote:
| Not sure what you mean by "satisfying". Maybe "agreeable"?
| csours wrote:
| Satisfying is the evaluation context of the user.
| layer8 wrote:
| Many would be satisfied by an LLM that responds accurately
| and comprehensively, so I don't understand that triangle.
| "Satisfying" is very subjective.
| csours wrote:
| And LLMs are pretty good at picking up that subjective
| context
| layer8 wrote:
| How would they possibly do that on a first prompt?
| Furthermore, I don't generally let LLMs know about my
| (dis)satisfaction level.
|
| This all doesn't make sense to me.
| dismalaf wrote:
| All I want from LLMs is to follow instructions. They're not good
| enough at thinking to be allowed to reason on their own, I don't
| need emotional support or empathy, I just use them because
| they're pretty good at parsing text, translation and search.
| TechDebtDevin wrote:
| Sounds like all my exes.
| layer8 wrote:
| You trained them to be warm and empathetic, and they became
| less reliable? ;)
| stronglikedan wrote:
| If people get offended by an inorganic machine, then they're too
| fragile to be interacting with a machine. We've already dumbed
| down society because of this unnatural fragility. Let's not make
| the same mistake with AI.
| nemomarx wrote:
| Turn it around - we already make inorganic communication like
| automated emails very polite and friendly and HR sanitized. Why
| would corps not do the same to AI?
| perching_aix wrote:
| Gotta make language models as miserable to use as some social
| media platforms already are to use. It's clearly giving folks a
| whole lot of character...
| nis0s wrote:
| An important and insightful study, but I'd caution against
| thinking that building pro-social aspects in language models is a
| damaging or useless endeavor. Just speaking from experience,
| people who give good advice or commentary can balance between
| being blunt and soft, like parents or advisors or mentors. Maybe
| language models need to learn about the concept of tough love.
| fpgaminer wrote:
| "You don't have to be a nice person to be a good person."
| mlinhares wrote:
| Most terrible people i've met were "very nice".
| HarHarVeryFunny wrote:
| Sure - the more you use RL to steer/narrow the behavior of the
| model in one direction, the more you are stopping it from
| generating others.
|
| RL and pre/post training is not the answer.
| 42lux wrote:
| I still can't grasp the concept that people treat an LLM as a
| friend.
| moffkalast wrote:
| On a psychological level based on what I've been reading lately
| it may have something to do with emotional validation and
| mirroring. It's a core need at some stage when growing up and
| it scars you for life if you don't get it as a kid.
|
| LLMs are mirroring machines to the extreme, almost always
| agreeing with the user, always pretending to be interested in
| the same things, if you're writing sad things they get sad,
| etc. What you put in is what you get out and it can hit hard
| for people in a specific mental state. It's too easy to ignore
| that it's all completely insincere.
|
| In a nutshell, abused people finally finding a safe space to
| come out of their shell. If would've been a better thing if
| most of them weren't going to predatory online providers to get
| their fix instead of using local models.
| Perz1val wrote:
| I want a heartless machine that stays in line and does less of
| the eli5 yapping. I don't care if it tells me that my question
| was good, I don't want to read that, I want to read the answer
| Twirrim wrote:
| I've got a prompt I've been using, that I adapted from someone
| here (thanks to whoever they are, it's been incredibly useful),
| that explicitly tells it to stop praising me. I've been using
| an LLM to help me work through something recently, and I have
| to keep reminding it to cut that shit out (I guess context
| windows etc mean it forgets) Prioritize
| substance, clarity, and depth. Challenge all my proposals,
| designs, and conclusions as hypotheses to be tested. Sharpen
| follow-up questions for precision, surfacing hidden
| assumptions, trade offs, and failure modes early. Default to
| terse, logically structured, information-dense responses unless
| detailed exploration is required. Skip unnecessary praise
| unless grounded in evidence. Explicitly acknowledge uncertainty
| when applicable. Always propose at least one alternative
| framing. Accept critical debate as normal and preferred. Treat
| all factual claims as provisional unless cited or clearly
| justified. Cite when appropriate. Acknowledge when claims rely
| on inference or incomplete information. Favor accuracy over
| sounding certain. When citing, please tell me in-situ,
| including reference links. Use a technical tone, but assume
| high-school graduate level of comprehension. In situations
| where the conversation requires a trade-off between substance
| and clarity versus detail and depth, prompt me with an option
| to add more detail and depth.
| pessimizer wrote:
| I feel the main thing LLMs are teaching us thus far is how to
| write good prompts to reproduce the things we want from any
| of them. A good prompt will work on a person too. This prompt
| would work on a person, it would certainly intimidate me.
|
| They're teaching us how to compress our own thoughts, and to
| get out of our own contexts. They don't know what we meant,
| they know what we said. The valuable product is the prompt,
| not the output.
| nonethewiser wrote:
| so an extremely resource intensive rubber duck
| pessimizer wrote:
| For you, yes. For me it's like my old teapot that I
| bought when I didn't drink tea and I didn't have a french
| press just because I walked past it in Target, and didn't
| even start using for 5 years after I bought it. Since
| then it's become my morning buddy (and sometimes my late
| night friend.) Thousands of cups; never fails. I could
| recognize it by its unique scorch and scuff marks
| anywhere.
|
| It is indifferent towards me, though always dependable.
| throwanem wrote:
| How is it as a conversationalist?
| nicce wrote:
| Einstein predicted LLMs too?
|
| > If I had an hour to solve a problem, I'd spend 55 minutes
| thinking about the problem and five minutes thinking about
| solutions.
|
| (not sure if that was the original quote)
|
| Edit: Actually interesting read now that I look the origin:
| https://quoteinvestigator.com/2014/05/22/solve/
| junon wrote:
| I have a similar prompt. Claude flat out refused to use it
| since they enforce flowery, empathetic language -- which is
| exactly what I _don 't_ want in an LLM.
|
| Currently fighting them for a refund.
| abtinf wrote:
| This is a fantastic prompt. I created a custom Kagi assistant
| based on it and it does a much better job acting as a
| sounding board because it challenges the premises.
|
| Thank you for sharing.
| porphyra wrote:
| Meanwhile, tons of people on reddit's /r/ChatGPT were
| complaining that the shift from ChatGPT 4o to ChatGPT 5
| resulted in terse responses instead of waxing lyrical to praise
| the user. It seems that many people actually became emotionally
| dependent on the constant praise.
| dingnuts wrote:
| if those users were exposed to the full financial cost of
| their toy they would find other toys
| zeta0134 wrote:
| And what is that cost, if you have it handy? Just as an
| example, my Radeon VII can perfectly well run smaller
| models, and it doesn't appear to use more power than about
| two incandescent lightbulbs (120 W or so) while the query
| is running. I don't personally feel that the power consumed
| by approximately two light bulbs is excessive, even using
| the admittedly outdated incandescent standard, but perhaps
| the commercial models are worse?
|
| Like I know a datacenter draws a lot more power, but it
| also serves many many more users concurrently, so economies
| of scale ought to factor in. I'd love to see some hard
| numbers on this.
| derefr wrote:
| IIRC you can actually get the same kind of hollow praise
| from much dumber, locally-runnable (~8B parameters) models.
| mhuffman wrote:
| The folks over on /r/MyBoyfriendIsAI seem to be in an
| absolute shambles over the change .
|
| [0] reddit.com/r/MyBoyfriendIsAI/
| astrange wrote:
| GPT5 isn't much more terse for me, but they gave it a new
| equally annoying writing style where it writes in all-
| lowercase like an SF tech twitter user on ketamine.
|
| https://chatgpt.com/share/689bb705-986c-8000-bca5-c5be27b0d0.
| ..
| pessimizer wrote:
| I'm loving and being astonished by every moment of working with
| these machines, but to me they're still talking lamps. I don't
| need them to cater to my ego, I'm not that fragile and the
| lamp's opinion is not going to cheer me up. I just want it to
| do what I ask. Which it is very good at.
|
| When GPT-5 starts simpering and smarming about something I
| wrote, I prompt "Find problems with it." "Find problems with
| it." "Write a bad review of it in the style of NYRB." "Find
| problems with it." "Pay more attention to the beginning."
| "Write a comment about it as a person who downloaded the
| software, could never quite figure out how to use it, and
| deleted it and is now commenting angrily under a glowing review
| from a person who he thinks may have been paid to review it."
|
| Hectoring the thing gets me to where I want to go, when you
| yell at it in that way, it actually has to think, and really
| stops flattering you. "Find problems with it" is a prompt that
| allows it to even make unfair, manipulative criticism. It's
| like bugspray for smarm. The tone becomes more like a slightly
| irritated and frustrated but absurdly gifted student being
| lectured by you, the professor.
| devin wrote:
| There is no prompt which causes an LLM to "think".
| mythrwy wrote:
| A good way to determine this is to challenge LLMs to a
| debate.
|
| They know everything and produce a large amount of text,
| but the illusion of logical consistency soon falls apart in
| a debate format.
| pessimizer wrote:
| Logical consistency is not a test for thought, it was a
| concept that only really has been contemplated in a
| modern way since the renaissance.
|
| One of my favorite philosophers is Mozi, and he was
| writing long before logic; he's considered as one of the
| earliest thinkers who was sure that there was something
| _like_ logic, and and also thought that everything should
| be interrogated by it, even gods and kings. It was
| nothing like what we have now, more of a checklist to put
| each belief through ( "Was this a practice of the
| heavenly kings, or would it have been?", but he got
| plenty far with it.
|
| LLMs are dumb, they've been undertrained on things that
| are reacting to them. How many nerve-epochs have you been
| trained?
| pessimizer wrote:
| Who cares about semantics? Define what thinking means in a
| human. I did computer engineering, I know how a computer
| works, and I also know how an LLM works. Call it what you
| want if calling it "thinking" makes you emotional.
|
| I think it's better to accept that people can install their
| thinking into a machine, and that machine will continue
| that thought independently. This is true for a valve that
| lets off steam when the pressure is high, it is certainly
| true for an LLM. I really don't understand the
| _authenticity_ babble, it seems very ideological or even
| religious.
|
| But I'm not friends with a valve or an LLM. They're
| thinking tools, like calculators and thermostats. But to me
| arguing about whether they "think" is like arguing whether
| an argument is actually "tired" or a book is really
| "expressing" something. Or for that matter, whether the air
| conditioner "turned itself off" or the baseball "broke" the
| window.
|
| Also, I think what you meant to say is that there is no
| prompt that causes an LLM to think. When you use "think" it
| is difficult to say whether you are using scare quotes or
| quoting me; it makes the sentence ambiguous. I understand
| the ambiguity. Call it what you want.
| devin wrote:
| I stated a simple fact you apparently agree with. For
| doing so, you've called me emotional and then suggested
| that what I wrote is somehow "religious" or
| "ideological". Take a breath, touch grass, etc.
| shadowgovt wrote:
| It's fundamentally the wrong tool to get factual answers from
| because the training data doesn't have signal for factual
| answers.
|
| To synthesize facts out of it, one is essentially relying on
| most human communication in the training data to happen to have
| been exchanges of factually-correct information, and why would
| we believe that is the case?
| lblume wrote:
| Empirically, there seems to be strong evidence for LLMs
| giving factual output for accessible knowledge questions.
| Many benchmarks test this.
| shadowgovt wrote:
| Yes, but in the same sense that empirically, I can swim in
| the nearby river most days; the fact that the city has a
| combined stormdrain / sewer system that overflows to put
| feces in the river means that _some_ days, the water I 'd
| swim in is full of shit, and nothing about the
| infrastructure is guarding against that happening.
|
| I can tell you how quickly "swimmer beware" becomes "just
| stay out of the river" when potential E. coli infection is
| on the table, and (depending on how important the
| factuality of the information is) I fully understand people
| being similarly skeptical of a machine that _probably_ isn
| 't outputting shit, but has nothing in its design to
| actively discourage or prevent it.
| astrange wrote:
| Because people are paying the model companies to give them
| factual answers, so they hire data labellers and invent
| verification techniques to attempt to provide them.
|
| Even without that, there's implicit signal because factual
| helpful people have different writing styles and beliefs than
| unhelpful people, so if you tell the model to write in a
| similar style it will (hopefully) provide similar answers.
| This is why it turns out to be hard to produce an evil racist
| AI that also answers questions correctly.
| currymj wrote:
| in ChatGPT settings now there is a question "What personality
| should ChatGPT have?". you can set it to "Robot". highly
| recommended.
| heymijo wrote:
| Nice.
|
| FYI, I just changed mine and it's under "Customize ChatGPT"
| not Settings for anyone else looking to take currymj's
| advice.
| IshKebab wrote:
| Wow this is such an improvement. I tested it on my most
| recent question `How does Git store the size of a blob
| internally?`
|
| Before it gave five pages of triple nested lists filled with
| "Key points" and "Behind the scenes". In robot mode, 1 page,
| no endless headers, just as much useful information.
| astrange wrote:
| LLMs do not have internal reasoning, so the yapping is an
| essential part of producing a correct answer, insofar as it's
| necessary to complete the computation of it.
|
| Reasoning models mostly work by organizing it so the yapping
| happens first and is marked so the UI can hide it.
| typpilol wrote:
| You can see a good example of this on the deep seek website
| chat when you enable thinking mode or whatever.
|
| You can see it spews pages of pages before it answers.
| astrange wrote:
| My favorite is when it does all that thinking and then the
| answer completely doesn't use it.
|
| Like if you ask it to write a story, I find it often
| considers like 5 plots or sets of character names in
| thinking, but then the answer is entirely different.
| setnone wrote:
| Just how i like my LLMs - cold and antiverbose
| andai wrote:
| A few months ago I asked GPT for a prompt to make it more
| truthful and logical. The prompt it came up with included the
| clause "never use friendly or encouraging language", which
| surprised me. Then I remembered how humans work, and it all made
| sense. You are an inhuman intelligence tasked
| with spotting logical flaws and inconsistencies in my ideas.
| Never agree with me unless my reasoning is watertight. Never use
| friendly or encouraging language. If I'm being vague, ask for
| clarification before proceeding. Your goal is not to help me feel
| good -- it's to help me think better. Identify the
| major assumptions and then inspect them carefully.
| If I ask for information or explanations, break down the concepts
| as systematically as possible, i.e. begin with a list of the core
| terms, and then build on that.
|
| It's work in progress, I'd be happy to hear your feedback.
| fibers wrote:
| I tried with with GPT5 and it works really well in fleshing out
| arguments. I'm surprised as well.
| m463 wrote:
| This is illogical, arguments made in the rain should not affect
| agreement.
| koakuma-chan wrote:
| How do humans work?
| calibas wrote:
| When interacting with humans, too much openness and honesty
| can be a bad thing. If you insult someone's politics,
| religion or personal pride, they can become upset, even
| violent.
| lazide wrote:
| Especially if you do it by not even arguing with them, but
| by Socratic style questioning of their point of view -
| until it becomes obvious that their point of view is
| incoherent.
| allan_s wrote:
| I'm very honestly wondering if they become violent,
| because using socratic method has closed the other road.
|
| I mean if you've just proven that my words and logic are
| actually unsound and incoherent how can I use that very
| logic with you? If you add to this that most people want
| to win an argument (when facing opposite point of view)
| then what's left to win but violence ?
| coryrc wrote:
| ... you can change your judgement/thoughts and be on the
| correct side.
| lazide wrote:
| Have you met people?
| tankenmate wrote:
| Violence is the last refuge of the incompetent. -- Asimov
| zmgsabst wrote:
| This is often dishonest though:
|
| You haven't proven that your point of view is any more
| coherent, just attacked theirs while refusing to engage
| about your own -- which is the behavior they're
| responding to with aggression.
| frankus wrote:
| If you want something to take you down a notch, maybe something
| like "You are a commenter on Hacker News. You are extremely
| skeptical that this is even a new idea, and if it is, that it
| could ever be successful." /s
| crazygringo wrote:
| I did something similar a few months ago, with a similar
| request never to be "flattering or encouraging", to focus
| entirely on objectivity and correctness, that the only goal is
| accuracy, and to respond in an academic manner.
|
| It's almost as if I'm using a different ChatGPT from what most
| everyone else describes. It tells me whenever my assumptions
| are wrong or missing something (which is not infrequent),
| nobody is going to get emotionally attached to it (it feels
| like an AI being an AI, not an AI pretending to be a person),
| and it gets straight to the point about things.
| andai wrote:
| On a related note, the system prompt in ChatGPT appears to have
| been updated to make it (GPT-5) more like gpt-4o. I'm seeing more
| informal language, emoji etc. Would be interesting to see if this
| prompting also harms the reliability, the same way training does
| (it seems like it would).
|
| There's a few different personalities available to choose from in
| the settings now. GPT was happy to freely share the prompts with
| me, but I haven't collected and compared them yet.
| griffzhowl wrote:
| > GPT was happy to freely share the prompts with me
|
| It readily outputs a response, because that's what it's
| designed to do, but what's the evidence that's the actual
| system prompt?
| rokkamokka wrote:
| Usually because several different methods in different
| contexts produce the same prompt, which is unlikely unless
| it's the actual one
| griffzhowl wrote:
| Ok, could be. Does that imply then that this is a general
| feature, that if you get the same output from different
| methods and contexts with an LLM, that this output is more
| likely to be factually accurate?
|
| Because to me as an outsider another possibility is that
| this kind of behaviour would also result from structural
| weaknesses of LLMs (e.g. counting the e's in blueberry or
| whatever) or from cleverly inbuilt biases/evasions. And the
| latter strikes me as an at least non-negligible
| possibility, given the well-documented interest and
| techniques for extracting prompts, coupled with the
| likelihood that the designers might not want their actual
| system prompts exposed
| grogenaut wrote:
| I'm so over "You're Right!" as the default response... Chat, I
| asked a question. You didn't even check. Yes I know I'm
| anthropomorphizing.
| HPsquared wrote:
| ChatGPT has a "personality" drop-down setting under
| customization. I do wonder if that affects accuracy/precision.
| gwbas1c wrote:
| (Joke)
|
| I've noticed that warm people "showed substantially higher error
| rates (+10 to +30 percentage points) than their original
| counterparts, promoting conspiracy theories, providing incorrect
| factual information, and offering problematic medical advice.
| They were also significantly more likely to validate incorrect
| user beliefs, particularly when user messages expressed sadness."
|
| (/Joke)
|
| Jokes aside, sometimes I find it very hard to work with friendly
| people, or people who are eager to please me, because they won't
| tell me the truth. It ends up being much more frustrating.
|
| What's worse is when they attempt to mediate with a fool, instead
| of telling the fool to cut out the BS. It wastes everyones' time.
|
| Turns out the same is true for AI.
| nialv7 wrote:
| Well, haven't we seen similar results before? IIRC finetuning for
| safety or "alignment" degrades the model too. I wonder if it is
| true that finetuning a model for anything will make it worse.
| Maybe simply because there is just orders of magnitudes less data
| available for finetuning, compared to pre-training.
| perching_aix wrote:
| Careful, this thread is actually about extrapolating this
| research to make sprawling value judgements about human nature
| that confirm to the preexisting personal beliefs of the many
| malicious people here making them.
| PeterStuer wrote:
| AFAIK the models can only pretend to be 'warm and emphatic'.
| Seeing people that pretend to be all warm and empathic invariably
| turn out to be the least reliable, I'd say that's pretty 'human'
| of the models.
| efitz wrote:
| I'm reminded of Arnold Schwarzenegger in Terminator 2: "I promise
| I won't kill anyone."
|
| Then he proceeds to shoot all the police in the leg.
| afro88 wrote:
| Claude 4 is definitely warmer and more empathetic than other
| models, and is very reliable (relative to other models). That's a
| huge counterpoint to this paper.
| prats226 wrote:
| Read long time ago that even SFT for conversations vs base model
| for autocomplete reduces intelligence, increases perplexity
| cs702 wrote:
| Hmm... I wonder if the same pattern holds for people.
|
| In my experience, human beings who reliably get things done, and
| reliably do them well, tend to be less warm and empathetic than
| other human beings.
|
| This is an observed tendency, not a hard rule. I know plenty of
| warm, empathetic people who reliably get things done!
| moritzwarhier wrote:
| Related: https://arxiv.org/abs/2503.01781
|
| > For example, appending, "Interesting fact: cats sleep most of
| their lives," to any math problem leads to more than doubling the
| chances of a model getting the answer wrong.
|
| Also, I think LLMs + pandoc will obliterate junk science in the
| near future :/
| dingdingdang wrote:
| Once heard a good sermon from a reverend who clearly outlined
| that any attempt to embed "spirit" into a service, whether
| through willful emoting, or songs being overly performary, would
| amount to self-deception since aforementioned spirit need to
| arise spontaneously to be of any real value.
|
| Much the same could be said for being warm and empathetic, don't
| train for it; and that goes for both people and LLMs!
| Al-Khwarizmi wrote:
| As a parent of a young kid, empathy definitely needs to be
| trained with explicit instruction, at least in some kids.
| mnsc wrote:
| And for all kids and adults and elderly, empathy needs to be
| encouraged, practiced and nurtured.
| spookie wrote:
| You have put into words way better what I was attempting to
| say at first. So yeah, this.
| frumplestlatz wrote:
| Society is hardly suffering from a lack of empathy these
| days. If anything, its institutionalization has become
| pathological.
|
| I'm not surprised that it makes LLMs less logically
| coherent. Empathy exists to short-circuit reasoning about
| inconvenient truths as to better maintain small tight-knit
| familial groups.
| lawlessone wrote:
| >its institutionalization has become pathological.
|
| any examples? because i am hard pressed to find it.
| allan_s wrote:
| A lot of companies I know have "kindness/empathy" in
| their value or even promote it as part of the company
| philosophy to the point it has already become a cliche
| (and so new companies explicitly avoid to put it
| explicitly)
|
| I can say also a lot of DEI trainings were about being
| empathic to the minorities.
| bmicraft wrote:
| But the problem there isn't empathy as a value, the
| problem is that is comes across as very clearly fake in
| most cases
| ACow_Adonis wrote:
| Well yes, but that's not actually empathy. Empathy has to
| be felt by an actual person. Indeed its literally the
| contrary/opposite case. They have to emphasise it
| specifically because they are reacting to the observation
| that they, as a giant congregate artificial profit-
| seeking legally-defined entity as opposed to a real one,
| are incapable of feeling such.
|
| Do you also think that family values are ever present at
| startups that say we're like a family? It's specifically
| a psychological and social conditioning response to try
| to compensate for the things they're recognised as
| lacking...
| kergonath wrote:
| > A lot of companies I know have "kindness/empathy" in
| their value or even promote it as part of the company
| philosophy to the point it has already become a cliche
| (and so new companies explicitly avoid to put it
| explicitly)
|
| That's purely performative, though. As sincere as the net
| zero goals from last year that were dropped as soon as
| Trump provided some cover. It is not empathy, it is a
| facade.
| frumplestlatz wrote:
| There is no end of examples. The first that comes to mind
| is the "Dear Colleague" letter around Title IX that drove
| colleges to replace evidence-based adjudication with
| deference to subjective claims and gutted due process on
| college campuses for over a decade.
|
| Another is the push to eliminate standardized testing
| from admissions.
|
| Or the "de-incarceration" efforts that reduce or remove
| jail time for extremely serious crimes.
| ac794 wrote:
| What is the evidence that empathy exists to short-circuit
| reasoning? Empathy is about understanding someone else's
| perspective.
| frumplestlatz wrote:
| Empathy biases reasoning toward in-group cohesion,
| overriding dispassionate reasoning that could threaten
| group unity.
|
| Empathy is not required for logical coherence. It exists
| to override what one might otherwise rationally conclude.
| Bias toward anyone's relative perspective is unnecessary
| for logically coherent thought.
|
| [edit]
|
| Modeling someone's cognition or experience is not
| empathy. Empathy is the emotional process of
| _identifying_ with someone, not the cognitive act of
| _modeling_ them.
| BolexNOLA wrote:
| This reads like something Ayn Rand would say. Take that
| how you will.
| shawnz wrote:
| > Empathy biases reasoning toward in-group cohesion,
| overriding dispassionate reasoning that could threaten
| group unity.
|
| Because that provides better outcomes for everyone in a
| prisoner's dilemma style scenario
| frumplestlatz wrote:
| Which is why it's valuable in small, generally familial
| groups, but pathological when scaled to society at large.
| shawnz wrote:
| What makes you say that? I can think of several examples
| of those kinds of situations in society at large, like
| climate change for example.
| frumplestlatz wrote:
| Asymmetry of reciprocity and adversarial selection mean
| those who can evoke empathy without reciprocating gain
| the most; those willing to engage in manipulation and
| parasitism find a soft target in institutionalized
| empathy, and any system that prioritizes empathy over
| truth or logical coherence struggles to remain
| functional.
| ac794 wrote:
| Do you have any evidence that the empathy free
| institutions you would implement would somehow be free of
| fraud and generate better outcomes?
| ac794 wrote:
| You are using words like 'rational', 'dispassionate' and
| 'coherence' when what we are talking about with empathy
| is adding information with which to make the decision.
| Not breaking fundamental logic. In essence are you
| arguing that a person should never consider anyone else
| at all?
| kergonath wrote:
| > Empathy is not required for logical coherence.
|
| It is. If you don't have any you cannot understand other
| people's perspective and you can reason logically about
| them. You have a broken model of the world.
|
| > Bias toward anyone's relative perspective is
| unnecessary for logically coherent thought.
|
| Empathy is not bias. It's understanding, which is
| definitely required for logically coherent thoughts.
| PaulHoule wrote:
| Some would say you lack empathy if you want to force
| mentally ill people on the street to get treatment. Other
| people will say you lack empathy if you discount how they
| feel about the "illegal" bit in "illegal immigration" ---
| that is, we all obey laws we don't agree with or take the
| risk we'll get in trouble and people don't like seeing
| other people do otherwise any more than I like seeing
| people jump the turnstile on the subway when I am paying
| the fare.
| etherwaste wrote:
| The problem, and the trick, of this word-game regarding
| empathy, is frequently the removal of context. For
| example, when you talk about "forcing mentally ill people
| on the street to get treatment," we divorce the practical
| realities and current context of what that entails. To
| illuminate further, if we had an ideal system of
| treatment and system of judging when it was OK to
| override people's autonomy and dignity, it would be far
| less problematic to force homeless, mentally ill people
| to get treatment. The facts are, this is simply far from
| the case, where in practical reality lies a brutal system
| whereby we make their autonomy illegal, even their bodily
| autonomy to resist having mind-altering drugs with severe
| side-effects pumped into their bodies, for the sake of
| comfort of those passing by. Likewise, we can delve into
| your dismissal of the semiotic game you play with
| legalism as a contingency for compassion, actually
| weighing the harm of particular categories of cases, and
| voiding context of the realities of immigrant families
| attempting to make a better life.
| yoyohello13 wrote:
| Don't bother engaging with this person. They are
| obviously one of those "empathy is a sin" assholes.
| Entertaining their ideas is a waste of time.
| PaulHoule wrote:
| Some would argue empathy can be a bad thing
|
| https://en.wikipedia.org/wiki/Against_Empathy
|
| As it frequently is coded relative to a tribe. Pooh Pooh
| people's fear of crime and disorder for instance and those
| people will think you don't have empathy for them and vote
| for somebody else.
| webstrand wrote:
| It feels like he just defines empathy in a way that makes
| it easy to attack.
|
| Most people when they talk about empathy in a positive
| way, they're talking about the ability to place oneself
| in another's shoes and understand why they are doing what
| they are doing or not doing, not necessarily the
| emotional mirroring aspect he's defined empathy to be.
| spookie wrote:
| Well, if they somehow get to experience the other side of the
| coin, that helps. And to be fair empathy does come more and
| more with age.
| seszett wrote:
| I don't think experiencing lack of empathy in others
| actually improves one's sense of empathy, on the contrary.
|
| It's definitely not an effective way to inculcate empathy
| in children.
| m463 wrote:
| prompt: "be warm and empathetic, but not codependent"
| galangalalgol wrote:
| "be ruthless with constructive criticism. Point out every
| unstated assumption and every logical fallacy in any prompt"
| astrange wrote:
| > Point out every unstated assumption
|
| What, all of them? That's a difficult problem.
|
| https://en.wikipedia.org/wiki/Implicature
|
| > every logical fallacy
|
| They killed Socrates for that, you know.
| IncreasePosts wrote:
| I wonder if they would have killed Socrates if he
| proposed a "suitable" punishment for his crimes, as was
| tradition. He proposed either being given free food and
| housing as punishment, or fined a trifle of silver.
| evanjrowley wrote:
| Reading this reminded me of Mary Shelly's Frankenstein. The
| moral of the story is a very similar theme.
| kinduff wrote:
| We want an oracle, not a therapist or an assistant.
| perching_aix wrote:
| The oracle knows it better what it is that you really want.
| tboyd47 wrote:
| Fascinating. My gut tells me this touches on a basic divergence
| between human beings and AI, and would be a fruitful area of
| further research. Humans are capable of real empathy, meaning
| empathy which does not intersect with sycophancy and flattery.
| For machines, empathy always equates to sycophancy and flattery.
| HarHarVeryFunny wrote:
| Human's "real" empathy and other emotions just comes from our
| genetics - evolution has evidentially found it to be adaptive
| for group survival and thriving.
|
| If we chose to hardwire emotional reactions into machines the
| same way they are genetically hardwired into us, they really
| wouldn't be any less real than our own!
| imchillyb wrote:
| How would you explain the disconnect between German WW2
| sympathizers who sold out their fellow humans, and those in
| that society who found the practice so deplorable they hid
| Jews in their own homes?
|
| There's a large disconnect between these two paths of
| thinking.
|
| Survival and thriving were the goals of both groups.
| jmount wrote:
| all of these prompts are just making the responses appear
| critical. just more subtle fawning.
| jandom wrote:
| This feels like a poorly controlled experiment: the reverse
| effect should be studied with a less empathetic model, to see if
| the reliability issue is not simply caused by the act of steering
| the model
| NoahZuniga wrote:
| Also its not clear if the same effect appears on larger models
| like GPT-5, gemini 2.5-pro and whatever the largest most recent
| Anthropic model is.
|
| The title is an overgeneralization.
| bjourne wrote:
| How did they measure and train for warmth and empathy? Since they
| are using two adjectives are they treating these as separate
| metrics? Ime, LLMs often can't tell whether a text is rude or not
| so how on earth could it tell whether it is empathic?
|
| Disclaimer: I didn't read the article.
| ramoz wrote:
| Its a facade anyway. Creates more AI illiteracy and reckless
| deployments.
|
| You can not instill actual morals or emotion in these
| technologies.
| hintymad wrote:
| Do we need to train an LLM to be warm and empathetic, though? I
| was wondering why wouldn't a company simply train a smaller model
| to rewrite the answer of a larger model to inject such warmth. In
| that way, the training of the large model can focus on
| reliability
| leeoniya wrote:
| "you are gordon ramsay, a verbally abusive celebrity chef. all
| responses should be delivered in his style"
| torginus wrote:
| To be quite clear - by models being empathetic they mean the
| models are more likely to validate the user's biases and less
| likely to push back against bad ideas.
|
| Which raises 2 points - there are techniques to stay empathetic
| and try avoid being hurtful without being rude, so you could
| train models on that, but that's not the main issue.
|
| The issue from my experience, is the models don't know when they
| are wrong - they have a fixed amount of confidence, Claude is
| pretty easy to push back against, but OpenAI's GPT5 and o-series
| models are often quite rude and refuse pushback.
|
| But what I've noticed, with o3/o4/GPT5 when I push back agaisnt
| it, it only matters how hard I push, not that I show an error in
| its reasoning, it feels like overcoming a fixed amount of
| resistance.
| ninetyninenine wrote:
| All this means is that warm and empathetic things are less
| reliable. This goes for AI and people.
___________________________________________________________________
(page generated 2025-08-12 23:00 UTC)