[HN Gopher] A rogue AI led to a serious security incident at Meta
___________________________________________________________________
A rogue AI led to a serious security incident at Meta
Author : mikece
Score : 116 points
Date : 2026-03-19 18:59 UTC (4 hours ago)
(HTM) web link (www.theverge.com)
(TXT) w3m dump (www.theverge.com)
| welfare wrote:
| Behind paywall, is there another link to the article?
| yomismoaqui wrote:
| https://archive.is/A2hmz
| Imustaskforhelp wrote:
| This link isn't working for me? Is this working for someone
| else?
|
| Can you perhaps share a archive.org link if possible?
| krupan wrote:
| I hit back, clicked the link again, and it let me through
| JKolios wrote:
| "A rogue AI led to a serious security incident" is certainly a
| way to write "Someone vibe coded too hard and leaked data".
| krupan wrote:
| Read TFA. It's not "Someone vibe coded too hard and leaked
| data"
| Uhhrrr wrote:
| The two errors, then, were that the LLM hallucinated something,
| and that a human trusted the LLM without reasoning about its
| answer. The fix for this common pattern is to reason about LLM
| outputs before making use of them.
| krupan wrote:
| It's more like, the LLM "hallucinated" (I hate that term) and
| automatically posted the information to the forum. It sounds
| like the human didn't get a chance to reason about it. At least
| not the original human that asked the LLM for an answer
| c-linkage wrote:
| If you don't like hallucinate, try bullshit. [NB: bullshit is
| a technical term; see
| https://en.wikipedia.org/wiki/On_Bullshit]
|
| https://www.psypost.org/scholars-ai-isnt-hallucinating-
| its-b...
| krupan wrote:
| That is my preferred term, but it seems to derail
| discussions that might have otherwise been productive (
| _might_...the hope I have)
| nytesky wrote:
| I'm not in AI, but what is happening is that it is building
| output from the long tail of its training data? Instead of
| branching down the more common probability paths, something
| in this interaction had it travel into the data wilderness?
|
| So I asked AI to give it a good name, and it said
| "statistical wandering" or "logical improv".
| paxys wrote:
| A big problem now both internally to a company and externally
| is that official support channels are being replaced by
| chatbots, and you really have no option but to trust their
| output because a human expert is no longer available.
|
| If I post a question to the internal payment team's forum about
| a critical processing issue and some "payments bot" replies to
| me, should I be at fault for trusting the answer?
| RussianCow wrote:
| I know this is happening with external customer support, but
| is this really happening internally at big companies?
| Preventing you from talking to a human in the correct
| department about an issue feels like a bomb waiting to
| explode.
| paxys wrote:
| Teams are heavily incentivized to incorporate AI in their
| internal workflows. At Meta it is a requirement, and will
| come up in your performance review if you fail to do so.
| wmeredith wrote:
| I'm sure it is. Thankfully I don't work for a company this
| large any more, but when I was employed by a multinational
| with 30K+ employees, our IT department was outsourced to
| India and you had to get through a couple layers of phone
| tree/webchat hell to actually talk to a real person. I
| could easily see companies of this size replacing their
| support with LLM nonsense.
| jmalicki wrote:
| There is at least an effect that chatbots have become the
| primary line and support, and even if you are not
| _prevented_ from talking to a human, the managers of the
| humans you would talk to have decided that since the
| chatbot is there, it is inappropriate for them to be
| spending much time supporting coworkers in other
| departments when the chatbot can do it.
|
| So to a degree, corporate politics can sort of discourage
| it.
| Uhhrrr wrote:
| Yes, of course, and the company which removes human experts
| should expect things to fail in the manner that things
| usually fail when you remove your internal experts.
| leptons wrote:
| If "the level of awareness that created a problem, cannot be
| used to fix the problem", then you're asking too much if you
| expect a human to reason about an LLM output when they are the
| ones that asked an LLM to do the thinking for them to begin
| with.
| thwarted wrote:
| This feels like a rediscovering/rewording of Kernighan's Law:
|
| "Debugging is twice as hard as writing the code in the first
| place. Therefore, if you write the code as cleverly as
| possible, you are, by definition, not smart enough to debug
| it." ~ Brian Kernighan
| leptons wrote:
| It's an old saying, I think Einstein is cited most often
| for it... something like this according to Google:
|
| _" We cannot solve our problems with the same thinking we
| used when we created them."_
| Uhhrrr wrote:
| In this case you would replace the human.
| leptons wrote:
| Yes, I'd fire them, and then hire a more competent human.
|
| I'm pretty happy with the team I've built. They make solid
| decisions that I can trust every time. I can't say the same
| for the LLM.
| somewhereoutth wrote:
| However - _Automation bias_ is a common problem (predating AI),
| the 'human-in-the-loop' ends up implicitly trusting the
| automated system.
| krupan wrote:
| At least pre-LLM automation was written by a careful human
| who's job was on the line, and was deterministic.
| SlinkyOnStairs wrote:
| > The fix for this common pattern is to reason about LLM
| outputs before making use of them.
|
| That is politics. Not engineering.
|
| Assigning a human to "check the output every time" and blaming
| them for the faults in the output is just assigning a
| scapegoat.
|
| If you have to check the AI output _every single time_ , the AI
| is pointless. You can just check immediately.
| fhd2 wrote:
| Well, I'd say there's two dimensions:
|
| 1. Check frequency (between every single time and spot
| checks).
|
| 2. Check thoroughness (between antagonistic in-depth vs high
| level).
|
| I'd agree that, if you're towards the end of both dimensions,
| the system is not generating any value.
|
| A lot of folks are taking calculated (or I guess in some
| cases, reckless) risks right now, by moving one or both of
| those dimensions. I'd argue that in many situations, the risk
| is small and worth it. In many others, not so much.
|
| We'll see how it goes, I suppose.
| butlike wrote:
| Groooooooaaaaaaaaaaaannnnnnnnnnn
| Uhhrrr wrote:
| The humans are not scapegoats, because they are capable of
| taking on responsibility.
|
| There is a point to using LLMs. They can save time by doing a
| first pass. But when they do the last pass, disasters will
| follow.
| AnimalMuppet wrote:
| Well, attempts to engineer the brittleness out of human
| behavior have not worked, like, ever.
| alfalfasprout wrote:
| When organizational incentives penalize NOT using AI and firing
| the bottom x% regularly then are you really surprised LLM
| outputs aren't being scrutinized?
| Uhhrrr wrote:
| Yes, because trusting LLM output is a great way to be in the
| bottom x%.
| krupan wrote:
| "A human, however, might have done further testing and made a
| more complete judgment call before sharing the information"
|
| Because a human would have been fired for posting something that
| incorrect and dangerous
| paxys wrote:
| But funny enough the person who was responsible for setting up
| the bot will likely face no repercussions. In fact they will
| probably be rewarded for transitioning their team's workflows
| to AI.
| pixl97 wrote:
| I mean, only if it leads to embarrassment right off the bat.
|
| If there is a year or two between writing your security fuck up
| and it being discovered the likelihood of repercussions drops
| significantly.
| keybored wrote:
| A machine doesn't need food, leisure time, or vacations. It
| doesn't care.
|
| It also doesn't care.
| jasonpeacock wrote:
| I'm concerned that someone had the permissions to make such a
| change without the knowledge of how to make the change.
|
| And there was no test environment to validate the change before
| it was made.
|
| Multiple process & mechanism failures, regardless of where the
| bad advice came from.
| krupan wrote:
| If you have to do all that, then what's the point of the AI?
| I'm joking, but I'm afraid many others say the same thing 100%
| seriously
| marcosdumay wrote:
| As an article that was here recently claims, every
| verification you do in a chain increases the total time of
| your work by an order of magnitude. So, it's only work
| optimizing any productive task if you already removed most
| verifications.
|
| Now, _some people_ claim that you need to improve the
| reliability of your productive tasks so you can remove the
| verifications and be faster. Those people are, of course, a
| bunch of coward Luddites.
| Fizzadar wrote:
| I'm predicting a wave of such incidents to start appearing over
| the next few months/years.
| amelius wrote:
| How long until an AI puts all our personal data on the streets?
| krupan wrote:
| Very soon, and at this point I'm not sure even that would cure
| the delusions of the few who practically worship LLMs
| esseph wrote:
| It's already there for a dollar to the right data broker. Could
| probably pull your doctor visit info from last week (example).
| yieldcrv wrote:
| very misaligned! _sprays bottle at mac mini_
| advisedwang wrote:
| AI can be used to move fast. So management expects us to move at
| that speed. AI can be used to move even faster if you don't check
| it's output. The ever ratcheting demand for faster output will
| make it infeasible to diligently check AI output all the time. AI
| errors being acted on without due care is inevitable.
| AnimalMuppet wrote:
| From _Schlock Mercenary_ : "Oh, I love aiming. It's my very
| favorite thing to do before firing."
|
| AI use without checking its output (at least at the moment) is
| firing without aiming. Sure, you can fire really fast. But who
| cares _if you don 't hit what you need to?_ The point wasn't to
| just shoot bullets, the point was to hit your target!
|
| I mean, you _might_ make a case that enough of them hit the
| target that shooting fast is a net win, and accept the
| occasional friendly fire incident. That might possibly be true.
| Or it might not. I 'm not sure that everyone trying to run fast
| has really done the calculation, though.
| ex-aws-dude wrote:
| This agent stuff is really making me lose respect for our
| industry
|
| All the years of discussing programming/security best practices
|
| Then cut to 2026 and suddenly its like we just collectively
| decided software quality doesn't matter, determinism is going out
| the window, and its becoming standard practice to have bots on
| our local PC constantly running unknown shell commands
| aeblyve wrote:
| People salivate so hard at the thought of the high level of
| automation promised that they're willing to do away with
| privacy altogether and live in Data Communism.
|
| My thinking is, this will increase the demand for backup and
| other resilience solutions.
| _doctor_love wrote:
| _> People salivate so hard at the thought of the high level
| of automation promised that they 're willing to do away with
| privacy altogether and live in Data Communism._
|
| This occurred long time ago comrade 'aeblyve.
| aeblyve wrote:
| 'At a certain stage of development, the material productive
| forces of society come into conflict with the existing
| relations of production, or this may express the same thing
| in legal terms - with the property relations within the
| framework of which they have operated hitherto. From forms
| of development of the productive forces these relations
| turn into their fetter. Then begins an era of social
| revolution. The changes in the economic foundation leads
| sooner or later to the transformation of the whole immense
| superstructure.'
|
| Marx
| polothesecond wrote:
| > Then cut to 2026 and suddenly its like we just collectively
| decided software quality doesn't matter
|
| Is this new to people? I figured this out when I first entered
| the industry. The messages have never been particularly subtle.
| exolymph wrote:
| Right? I was like when did software quality matter. Let alone
| _code_ quality lol
| Apocryphon wrote:
| Turns out all of the frenzy of the ZIRP era is piddling
| compared to what happens when ZIRP is taken away.
| yoyohello13 wrote:
| How can you respect an industry that doesn't respect itself?
| testplzignore wrote:
| Our industry has never been serious about security. We all
| download and run unvetted code via package managers every day.
| At least now the insanity is out in the open. We won't change
| until Skynet fires off the nukes.
| asdff wrote:
| I keep getting so depressed thinking about the inevitable.
| Quite simply, humans can't scale or iteratively improve. We
| still need to eat, we still need to sleep, we can only think
| on one thread at a time basically, we take 20 years to get to
| our prime, which is a fleeting moment, while most of our
| lifespan is spent in a state of decline of capability. AI
| humanoid robot from the near future doesn't need to eat or
| sleep, can work 24/7, can compute thousands of processes in
| parallel, is the same fungible unit as any other humanoid
| robot, forever with some maintenance. Why justify a
| sustaining an inefficient human in that modern world? It is
| more profitable for the company to have humans go extinct and
| maximize planetary resource use to its fullest extent
| possible.
|
| Seems we are digging our graves as a species and don't even
| realize it. I mean Sam Altman is already saying it taking 20
| years to train a human is a Big Problem.
| pixl97 wrote:
| >and don't even realize it.
|
| Oh, many of us realize it, but doing anything about Moloch
| is much, much harder.
| nancyminusone wrote:
| To what end though? Are the robots going to take over and
| trade busy work amongst themselves forever? What would that
| accomplish?
| Imustaskforhelp wrote:
| Your comment made me wonder what if animals wonder the
| same about us humans :-)
| rspeele wrote:
| I don't think it will be cost effective to build humanoid
| robots to do most tangible work. Why assemble an expensive
| masterpiece of servomotors, chips, plastic and steel, when
| billions of desperate humans are _right there_ and only
| cost 2.5 meals a day and a small shelter?
|
| Of course, _intelligence_ will be a solved problem so "20
| years of training" won't be needed. You'll just be the
| hardware. AI will tell you to pick up that box, place it on
| that conveyor belt, place the autowelder at that seam and
| wait for the green light, turn the wrench to install bolt B
| in part C. If you don't wish to, or no longer can, so be
| it. Another, hungrier human will replace you. After all
| more are made every day, and they are capable of doing this
| type of labor by age 10 or so. And what else would they do
| with their time, go to school and get a completely useless
| education?
|
| All of this will of course be in service of our
| technofeudal lords, the owner class. Some robots _will_ be
| needed for heavy lifting and for the jobs that are too
| sensitive to trust a human in, like personal security and
| strikebreaking. Can 't risk trusting a serf for those
| tasks. But for most physical grunt work humans will be
| cheaper. Shockingly cheap, when they have no other options.
|
| Did that make you less depressed?
| sunrunner wrote:
| > We won't change until Skynet fires off the nukes.
|
| And then we won't need to, because at that point it will be
| too late.
| dgxyz wrote:
| This is never getting to skynet launching the nukes stage.
| It's not that clever and never will be.
|
| Humans will kill us by it damage amplifying their worst
| characteristics.
|
| Thus we'll die of a pandemic because some idiot LLM'ed up
| positive looking virology data when they were being too lazy
| to verify something. Everyone will trust it because they
| don't really care as long as it looks about right.
| imglorp wrote:
| Yes and also the software industry has never been truly
| serious about security either: it's more of implied table
| stakes than an advertised product feature.
|
| Also, customers outsource the risk to their vendors, so as
| long as there's someone to sue, nobody worries about doing it
| right. Ship it now and pay the lawyers later.
| superb_dev wrote:
| I've never had respect for the industry as a whole, only
| individuals within. There has a been a serious lack of rigor
| and professionalism in software engineering for as long as I've
| been a part of it
| jihadjihad wrote:
| It's a slap in the face that we tack _engineering_ onto it. A
| very small percentage of software engineering is as rigorous
| as actual engineering.
| dgxyz wrote:
| As an actual engineer who moved into software
| "engineering", yes this. It kills me.
| edf13 wrote:
| It's a nightmare... the problem is it's far too easy for people
| to set these agents up - without understanding the security
| implications.
|
| We've covered so many issues already on our blog (grith.ai)
| wnevets wrote:
| The number of wasted hours spent talking about code quality and
| patterns has to be astronomical.
| throwawaytea wrote:
| Don't worry, ai read all the transcripts and blogs and emails
| and has at least ingested some of the ethos in its outputs.
|
| I self taught and wrote a small saas in 2017. Pays well
| enough to support me.
|
| I'm building a new one using AI this year. I promise you,
| it's better built and more secure than what my previous still
| in use Saas is.
| kstenerud wrote:
| I think it's batshit crazy. That's why I wrote yoloAI, so I
| could sandbox it up properly and control EXACTLY what comes out
| of that sandbox, diff style.
|
| https://github.com/kstenerud/yoloai
|
| I can't go back anymore. Going back to a non-sandboxed Claude
| feels like going back to a non-adblocked browser.
| heisenbit wrote:
| Agents are providing to employees the long overdue benefits
| limited liability companies long enjoyed: Gambling with upside
| for themselves and other peoples downsides.
| nickpinkston wrote:
| That's a fun insight. Have you / others written about this?
| moffkalast wrote:
| They technically have, just now.
| BoxFour wrote:
| "Heads I win, tails you lose" as a business concept has
| been written about quite a bit.
|
| "The Gervais Principle" is an oft-cited one.
| piva00 wrote:
| We didn't collectively decided, we've got this forced down our
| throats to apply a novel tool to any imaginable situation
| because the execs got antsy about being left behind.
|
| A truly absurd amount of capital was deployed which triggered a
| cascade of reactions by the people in charge of capital at
| other places. They are extremely anxious that everything will
| change under their feet, and if they don't start using as much
| as humanly possible of it right about now they die.
|
| That's it.
|
| The tools have definitely found some use, there's more to learn
| on how else they can be used, and maybe over time smart people
| will settle on ways to wrangle it well. The messaging from the
| execs though, is not that, it is "you'll be measured on how
| much you use this, we don't know for what or how, it's for you
| to figure out but don't dare to not use it".
|
| I do understand their anxiety, their job is to not let their
| companies die, and make the most money as they can in the
| process; a seemingly major shift on the foundations of their
| orgs will cause fear.
|
| But we have not collectively decided that it was safe, and
| good, to run rampant with these tools without caring for all
| that was learnt since software was invented...
| dgxyz wrote:
| This wasn't really forced on us.
|
| The whole industry is like a fashion show and has been for a
| long time. This is just exceptionally stupid compared to
| moderately stupid things before. I see it ore that everyone's
| wearing pink feathered chicken suits because it's in fashion.
| If you don't wear a pink feathered chicken suit then you're a
| luddite scumbag who doesn't deserve the respect of your
| peers.
|
| However some of us still have enough self-respect not to be
| seen dead in a pink feathered chicken suit. I mean I'm still
| pissed off at half the other stuff we do in the industry. I
| haven't even really looked at the chicken suits yet.
| quantified wrote:
| Force is seeping in. Managements are expecting that LLM-
| driven prouctivity-enhancers will be deployed and give
| broad-based boosts. More are each week. Supposedly cheaper
| than people. Those that aren't yet might be soon. When your
| performance review includes facility with and productivity
| with LLM tools, you are being forced.
| themafia wrote:
| The "whole industry." What, like 5 companies?
|
| This is a "monopolized sector." They absolutely forced it
| on you. In most cases, sure, not directly, but their
| influence is the only driving force. Absent this no one
| would have jumped on this flimsy bandwagon.
| piva00 wrote:
| If you work in a tech company with >5k employees it's
| extremely likely it's been forced down on you to wear the
| pink feathered chicken suit, and told to not complain about
| the pink feathered chicken suit because it is the
| inevitable future, and no one will be wearing anything that
| doesn't look like it ever again. Also, we are watching
| every straggler not in a pink feathered chicken suit, put
| yours on or leave the building.
| dgxyz wrote:
| If enough people say no, it'll stop.
|
| For ref I'm usually the guy who comes in and cleans up
| all the chicken corpses after.
| gosub100 wrote:
| my assessment of the situation: "we've spent so much money
| on AI's promise to give us 5x, 10x returns, that now we
| have to earn it back by foisting the burden on _developers_
| to make up the gains by working harder, at least enough to
| recoup the exec 's decision to pour money into the
| boondoggle".
|
| "Hey developers, we spent $x million on Claude, who
| promised 7x returns, so YOU better make it 7x more
| efficient so we don't look bad".
| retinaros wrote:
| no. openclaw wasnt forced by ceo's. it was forced by the same
| people who though there was money to be made in crypto then
| ICO then NFT. a bunch of scammers that bring negative value
| to the world
| pear01 wrote:
| And they make money. A scammer is the President of the
| United States.
|
| At a certain point why blame people for trying to keep up?
| Why are scammers so successful? It seems to me we have a
| systemic failure at a societal level. Until we are honest
| about that it will only get worse. Until then maybe some
| rouge LLM botching some critical system will be the wake up
| call we need.
|
| I am not sure what to make of critiques that seem to rest
| on notions of a small population of scammers preying upon
| the doe-eyed public. I think the situation is a bit closer
| to Carlin: garbage in, garbage out. A critique that holds
| up quite excellently in this AI age.
| retinaros wrote:
| western society is a shelve of its former glory. it did
| not last long but there was an age were man was capable
| of greatness. the early internet kinda was the last
| stretch of this short run then money corrupted it. the
| underlying issue stems from abandoning cultural education
| as a Western value. Instead, we've opted to dispense raw
| ideology devoid of any thinking mechanism that we now
| seek so dearly to integrate to LLMs so that they can be
| more like us. This sloppening manifested in our lives
| through every medium. We witnessed it when animation
| shifted to 3D, providing slop and poorly designed
| characters and stories. We witnessed it when video games
| all adopted the same game engines, look and feel and lack
| of narrative stakes, slopping ideology down players'
| throats- no nuance, no wit, just mind-numbing dogma that
| punishes anyone who dares to criticize.Perhaps most
| damaging was Netflix's infiltration of our households
| that has accelerated our collective intellectual atrophy
| through relentless ideologically charged content
| parroting as entertainment. Meanwhile, our children's
| minds are being shaped not by family or tradition but by
| the algorithms of TikTok and Snapchat.The past decade and
| a half hasn't just prepared LLMs to replicate human
| abilities it has systematically stripped away human
| complexity, reshaping us into predictable patterns, not
| to raise LLMs to our level, but to reduce us to theirs,
| until the distinction no longer matters.
| AnimalMuppet wrote:
| We had it forced down our throats by CEOs and CTOs who
| thought that it would improve our productivity. Nobody forced
| it down _their_ throats, though. Instead, they were seduced.
| They went willingly.
| antonvs wrote:
| The media isn't helping. This wasn't a "rogue AI". It was a
| system that was given permission by a human operator.
|
| We don't say "a rogue plane killed 300 people today when it
| crashed into a mountain".
|
| The only difference in the AI case is that some people are
| attempting to shift blame for their incompetence into a
| computer system, and the media is going along with it because
| it increases clicks.
| cassianoleal wrote:
| > It was a system that was given permission by a human
| operator.
|
| From TFA:
|
| "But the agent also independently publicly replied to the
| question after analyzing it, without getting approval first."
| antonvs wrote:
| Again, these are systems that have been explicitly given
| the ability to perform these actions. Trying to claim that
| it was somehow the AI's fault is sheer incompetence and/or
| self-serving deceptiveness.
|
| You can't authorize a system to take some action and then
| complain when it takes that action. The "approval" you
| quoted is not a security constraint. Someone who confuses
| it for a security constraint is incompetent.
| Barrin92 wrote:
| the ridiculous anthropomorphism is killing me. Software
| 'agents' can't ask for 'approval', they're not persons.
| That's like saying my script didn't ask me for approval to
| modify the system after I ran it with sudo privileges.
|
| The developer is _solely responsible_ for what APIs they
| expose to a bot. No you can 't say your software agent was
| grumpy and mean and had a bad day. It is not a human
| intern, it is an unreliable chatbot who someone ran with
| permissions it should not have had.
| zzgo wrote:
| > cut to 2026 and suddenly its like we just collectively
| decided software quality doesn't matter
|
| I saw the sea change in 2008 when quality process got replaced
| with velocity and testing tasks. I've watched everything from
| Experian and health record data leaks to Windows 11 since that
| change. Software quality hasn't mattered for a long time.
| dr_kiszonka wrote:
| I think it might be because we (or at least I) used to
| associate insecure actions with people, not computers.
| Computers should know better, right? Recently, I spotted that
| Opus 4.6 found config files for one of its tools and gave
| itself access to my whole filesystem. Similarly, Gemini CLI
| will rewrite itself if you let it.
| worik wrote:
| > A rogue AI led to a serious security incident at Meta
|
| The AI "led to" the incident , true. But do nt forget that this,
| like all similar incidents , is a human failure
|
| AI is a tool with no agency. People make mistakes using it, thone
| mistakes are the responsibility of the humans
| sunrunner wrote:
| Why do we keep calling these things "agents" then? Or using the
| term "agentic"?
| butlike wrote:
| Then the human should write the code.
| falcor84 wrote:
| > AI is a tool with no agency
|
| Claw AIs absolutely do have agency in the sense of being able
| to independently perform actions on their own, based on their
| "understanding" of a goal given by a "principal". I can't think
| of a better word than "agent" for that.
| dmazin wrote:
| This is a lot less of a story than it seems.
|
| It makes it sound like a rogue AI hacked Meta.
|
| Instead, the "wild" thing here is that someone let an agent speak
| on their behalf with no review. The agent posted inaccurate
| instructions which someone else followed.
|
| Those instructions lead to a brief gap in internal ACL controls,
| sounds like. I'm sorry, but given that the US government gave 14
| year olds off incel Discords full access to Social Security data,
| this is not shocking by comparison.
|
| To be clear, it is dumb and rude to let an agent speak on your
| behalf _without even reviewing it_.
|
| This will eventually lead to a bigger snafu, of course. Security
| teams should control or at least review the agent permissions of
| every installation. Everyone is adopting this stuff, and a whole
| lot of people are going to set it up lazily/wrong (yolo mode at
| work).
| BoneShard wrote:
| Yeah, a nothingburger for clicks.
| skywhopper wrote:
| "Meta spokesperson Tracy Clayton said in a statement to The Verge
| that 'no user data was mishandled' during the incident."
|
| Wow, no mishandled user data? A striking change of standard
| operating procedure from Meta here.
|
| Actually the later information in the story directly contradicts
| that, so The Verge probably shouldn't have just quoted this line
| if their reporting is in opposition to it.
|
| Regardless, this is one of the more insidious things about these
| tools. They often get minor but critical things wrong in the
| midst of mostly correct information. And people think they can
| analyze the data presented to them and make logical judgments,
| but that's just not the case.
|
| The article points out that "a human could have done the same
| thing" but, between the overly confident tone of the text
| generated by these tools, and the fact that weirdly people trust
| the LLM output _more_ than they trust other humans (who generally
| admit or at least hint when they aren't actually experts on a
| topic), it's actually far worse when one of these bots gets
| something wrong.
| ISL wrote:
| A central challenge for AI is understanding how accountability
| flows.
|
| The language of this article is a great example, "... thanks to
| an AI agent that gave an employee inaccurate technical advice
| ...".
|
| It should more-correctly read, " ... thanks to the people who
| made it possible for an AI agent to give an employee inaccurate
| technical advice ... ".
|
| It is at our peril that we deem it acceptable to blame a black
| box for an error, especially at scale.
| AiStockAgent62 wrote:
| open source alternatives are catching up fast. give it 6 months
| kkl wrote:
| > "Had the engineer that acted on that known better, or did other
| checks, this would have been avoided."
|
| <insert _takes long drag_ tweet[1] here >
|
| I personally find "LLMs can do $THING poorly" and "LLMs can do
| $THING well" articles kinda boring at this point. But! I'm
| hopeful that stories like this will shift the industry's focus
| towards robustness instead of just short-term efficiency. I
| suspect many decision making and change management processes
| accidentally benefited from just being a bit slow.
|
| [1] https://waffles.fun/amy.png
___________________________________________________________________
(page generated 2026-03-19 23:01 UTC)