[HN Gopher] The Hallucination Defense
___________________________________________________________________
The Hallucination Defense
Author : niyikiza
Score : 32 points
Date : 2026-01-29 19:45 UTC (3 hours ago)
(HTM) web link (niyikiza.com)
(TXT) w3m dump (niyikiza.com)
| JohnFen wrote:
| > "The AI hallucinated. I never asked it to do that."
|
| > That's the defense. And here's the problem: it's often hard to
| refute with confidence.
|
| Why is it necessary to refute it at all? It shouldn't matter,
| because whoever is producing the work product is responsible for
| it, no matter whether genAI was involved or not.
| salawat wrote:
| Except for the fact that that very accountability sink is
| relied on by senior management/CxO's the world over. The only
| difference is that before AI, it was the middle manager's
| fault. We didn't _tell anyone to break the law_. We just put in
| place incentive structures that require it, and play coy, then
| let anticipatory obedience do the rest. Bingo. Accountability
| severed. You can 't prove I said it in a court of law, and
| skeevy shit gets done because some poor bloke down the ladder
| is afraid of getting fired if he doesn't pull out all the stops
| to meet productivity quotas.
|
| AI is just better because no one can actually explain why the
| thing does what it does. Perfect management scapegoat without
| strict liability being made explicit in law.
| pixl97 wrote:
| Hence why many life and death things require licencing and
| compliance, and tend to come with very long paper trails.
|
| The software world has been very allergic to getting anywhere
| near the vicinity of a system like that.
| salawat wrote:
| Did I give the impression that the phenomena was unique to
| software? Hell, Boeing was a shining example of the
| principle in action with 737 MAX. Don't get much more
| "people live and die by us, and we know it (but management
| set up the culture and incentives to make a deathtrap
| anyway)." No one to blame of course. These things just
| happen.
|
| Licensure alone doesn't solve all these ills. And for that
| matter, once regulatory capture happens, it has a tendency
| to make things worse due to consolidation pressure.
| niyikiza wrote:
| You're right, they should be responsible. The problem is
| proving it. "I asked it to summarize reports, it decided to
| email the competitor on its own" is hard to refute with current
| architectures.
|
| And when sub-agents or third-party tools are involved,
| liability gets even murkier. Who's accountable when the action
| executed three hops away from the human? The article argues for
| receipts that make "I didn't authorize that" a verifiable claim
| bulatb wrote:
| There's nothing to prove. Responsibility means you accept the
| consequences for its actions, whatever they are. You own the
| benefit? You own the risk.
|
| If you don't want to be responsible for what a tool that
| might do anything at all might do, don't use the tool.
|
| The other option is admitting that you don't accept
| responsibility, not looking for a way to be "responsible" but
| not accountable.
| tossandthrow wrote:
| Sounds good in theory, doesn't work in reality.
|
| Had it worked then we would have seen many more CEOs in
| prison.
| NoMoreNicksLeft wrote:
| The veil of liability is built into statute, and it's no
| accident.
|
| Such so magic forcefield exists for you, though.
| walt_grata wrote:
| There being a few edge cases where it doesn't work in
| doesn't mean it doesn't work in the majority of cases and
| that we shouldn't try to fix the edge cases.
| bulatb wrote:
| We're taking about different things. To take
| responsibility is volunteering to accept accountability
| without a fight.
|
| In practice, almost everyone is held potentially or
| actually accountable for things they never had a choice
| in. Some are never held accountable for things they
| freely choose, because they have some way to dodge
| accountability.
|
| The CEOs who don't accept accountability were lying when
| they said they were responsible.
| freejazz wrote:
| This isn't a legal argument and these conversations are
| so tiring because everyone here is insistent upon drawing
| legal conclusions from these nonsense conversations.
| QuadmasterXLII wrote:
| This doesn't seem conceptually different from running
| [ $[ $RANDOM % 6] = 0 ] && rm -rf / || echo "Click"
|
| on your employer's production server, and the liability
| doesn't seem murky in either case
| staticassertion wrote:
| What if you wrote something more like: #
| terrible code, never use ty def cleanup(dir):
| system("rm -rf {dir}") def main():
| work_dir = os.env["WORK_DIR"] cleanup(work_dir)
|
| and then due to a misconfiguration "$WORK_DIR" was
| truncated to be just "/"?
|
| At what point is it negligent?
| direwolf20 wrote:
| This is not hypothetical. Steam and Bumblebee did it.
| extraduder_ire wrote:
| That was the result of an additional space in the path
| passed to rm, IIRC.
|
| Though rm /$TARGET where $TARGET is blank is a common
| enough footgun that --preserve-root exists and is
| default.
| niyikiza wrote:
| You'd be surprised to see how often we're seeing those
| types of semantic attack vulnerabilities in Agent
| frameworks: https://niyikiza.com/posts/map-territory/
| a_t48 wrote:
| Bungie, too, in a similar way.
| phoe-krk wrote:
| _> "I asked it to summarize reports, it decided to email the
| competitor on its own" is hard to refute with current
| architectures._
|
| If one decided to paint a school's interior with toxic paint,
| it's not "the paint poisoned them on its own", it's "someone
| chose to use a paint that can poison people".
|
| Somebody was responsible for choosing to use a tool that has
| this class of risks and explicitly did not follow known and
| established protocol for securing against such risk.
| Consequences are that person's to bear - otherwise the
| concept of responsibility loses all value.
| im3w1l wrote:
| > otherwise the concept of responsibility loses all value.
|
| Frankly, I think that might be exactly where we end up
| going. Finding a responsible person to punish is just a
| tool we use to achieve good outcomes, and if scare tactics
| is no longer applicable to the way we work, it might be
| time to discard it.
| phoe-krk wrote:
| A brave new world that is post-truth, post-meaning, post-
| responsibility, and post-consequences. One where the AI's
| hallucinations eventually drag everyone with it and
| there's no other option but to hallucinate along.
|
| It's scary that a nuclear exit starts looking like an
| enticing option when confronted with that.
| im3w1l wrote:
| Ultimately the goal is to have a system that prevents
| mistakes as much as possible adapts and self-corrects
| when they do happen. Even with science we acknowledge
| that mistakes happen and people draw incorrect
| conclusions, but the goal is to make that a temporary
| state that is fixed as more information comes in.
|
| I'm not claiming to have all the answers about how to
| achieve that, but I am fairly certain punishment is not a
| necessary part of it.
| groby_b wrote:
| "Our tooling was defective" is not, in general, a defence
| against liability. Part of a companys obligations is to
| ensure all its processes stay within lawful lanes.
|
| " _Three months later [...] But the prompt history? Deleted.
| The original instruction? The analyst's word against the
| logs._ "
|
| One, the analysts word does not override the logs, that's the
| point of logs. Two, it's fairly clear the author of the fine
| article has never worked close to finance. A three month
| retention period for AI queries by an analyst is not an
| option.
|
| SEC Rule 17a-4 & FINRA Rule 4511 have entered the chat.
| niyikiza wrote:
| Agree ... retention is mandatory. The article argues you
| should retain authorization artifacts, not just event logs.
| Logs show what happened. Warrants show who signed off on
| what
| groby_b wrote:
| FFIEC guidance since '21: https://www.occ.gov/news-
| issuances/bulletins/2021/bulletin-2...
| LeifCarrotson wrote:
| > "I asked it to summarize reports, it decided to email the
| competitor on its own" is hard to refute with current
| architectures.
|
| No, it's trivial: "So you admit you uploaded confidential
| information to the unpredictable tool with wide
| capabilities?"
|
| > Who's accountable when the action executed three hops away
| from the human?
|
| The human is accountable.
| pixl97 wrote:
| As the saying goes
|
| ----
|
| A computer can never be held accountable
|
| Therefore a computer must never make a management decision
| gowld wrote:
| What if you carried a stack of papers between buildings on
| a windy day, and the papers blew away?
| groby_b wrote:
| "And when sub-agents or third-party tools are involved,
| liability gets even murkier."
|
| It really doesn't. That falls straight on Governance, Risk,
| and Compliance. Ultimately, CISO, CFO, CEO are in the line of
| fire.
|
| The article's argument happens in a vacuum of facts. The fact
| that a security engineer doesn't know that is depressing, but
| not surprising.
| freejazz wrote:
| The burden of substantiating a defense is upon the defendant
| and no one else.
| doctorpangloss wrote:
| Wait till you find out about "pedal confusion."
| nerdsniper wrote:
| The distinction some people are making is between copy/pasting
| text vs agentic action. Generally mistakes "work product" as in
| output from ChatGPT that the human then files with a court,
| etc. are not forgiven, because if you signed the document, you
| own its content. Versus some vendor-provided AI Agent which
| simply takes action on its own that a "reasonable person" would
| not have expected it to. Often we forgive those kinds of
| software bloopers.
| ori_b wrote:
| If you put a brick on the accelerator of a car and hop out,
| you don't get to say "I wasn't even in the car when it hit
| the pedestrian".
| Shalomboy wrote:
| This is true for bricks, but it is not true if your dog
| starts up your car and hits a pedestrian. Collisions caused
| by non-human drivers are a fascinating edge case for the
| times we're in.
| victorbjorklund wrote:
| I don't know where you from but at least in Sweden you
| have strict liability for anything your dog does
| freejazz wrote:
| Prima facie negligence = liability
| ori_b wrote:
| In the USA, at least, it seems pet owners are liable for
| any harm their pets do.
| cess11 wrote:
| Legally, in a lot of jurisdictions, a dog is just your
| property. What it does, you did, usually with presumed
| intent or strict liability.
| gowld wrote:
| What if you planted a bush that attracted a bat that bit
| a child?
| Muromec wrote:
| What if you have an email in your inbox warning you that
| 1) this specific bush attracts bats and 2) there were in
| fact bats seen near you bush and 3) bats were observed
| almost biting a child before. And you also have "how do I
| fuck up them kids by planting a bush that attracts bats"
| in your browser history. It's a spectrum you know.
| jacquesm wrote:
| It is very much true for dogs in that case: (1) it is
| your dog (2) it is your car (3) it is your responsibility
| to make sure your car can not be started by your dog (4)
| the pedestrian has a reasonable expectation that a
| vehicle that is parked without a person in it has been
| made safe to the point that it will not suddenly start to
| move without an operator in it and dogs don't qualify.
|
| You'd lose that lawsuit in a heartbeat.
| observationist wrote:
| To me, it's 100% clear - if your tool use is reckless or
| negligent and results in a crime, then you are guilty of that
| crime. "It's my robot, it wasn't me" isn't a compelling
| defense - if you can prove that it behaved significantly
| outside of your informed or contracted expectations, then
| maybe the AI platform or the Robot developer could be at
| fault. Given the current state of AI, though, I think it's
| not unreasonable to expect that any bot can go rogue, that
| huge and trivially accessible jailbreak risks exist, so
| there's no excuse for deploying an agent onto the public
| internet to do whatever it wants outside direct human
| supervision. If you're running moltbot or whatever, you're
| responsible for what happens, even if the AI decided the best
| way to get money was to hack the Federal Reserve and assign a
| trillion dollars to an account in your name. Or if Grok goes
| mechahitler and orders a singing telegram to Will Stancil's
| house, or something. These are tools; complex, complicated,
| unpredictable tools that need skillfull and careful use.
|
| There was a notorious dark web bot case where someone created
| a bot that autonomously went onto the dark web and purchased
| numerous illicit items.
|
| https://wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww.bitnik.or.
| ..
|
| They bought some ecstasy, a hungarian passport, and random
| other items from Agora.
|
| >The day after they took down the exhibition showcasing the
| items their bot had bought, the Swiss police "arrested" the
| robot, seized the computer, and confiscated the items it had
| purchased. "It seems, the purpose of the confiscation is to
| impede an endangerment of third parties through the drugs
| exhibited, by destroying them," someone from !Mediengruppe
| Bitnik wrote on their blog.
|
| In April, however, the bot was released along with everything
| it had purchased, except the ecstasy, and the artists were
| cleared of any wrongdoing. But the arrest had many wondering
| just where the line gets drawn between human and computer
| culpability.
| b00ty4breakfast wrote:
| that darknet bot one always confuses me. The
| artists/programmers/whatever specifically instructed the
| computer, through the bot, to perform actions that would
| likely result in breaking the law. It's not a side-effect
| of some other, legal action which they were trying to
| accomplish, it's entire purpose was to purchase things on a
| marketplace known for hosting illegal goods and services.
|
| If I build an autonomous robot that swings a hunk of steel
| on the end of a chain and then program it to travel to
| where people are likely to congregate and someone gets hit
| in the face, I would rightfully be held liable for that.
| cess11 wrote:
| "computer culpability"
|
| That idea is really weird. Culpa (and dolus) in occidental
| law is a thing of the mind, what you understood or should
| have understood.
|
| A database does not have a mind, and it is not a person. If
| it could have culpa, then you'd be liable for assault,
| perhaps murder, if you took it apart.
| dragonwriter wrote:
| > To me, it's 100% clear - if your tool use is reckless or
| negligent and results in a crime, then you are guilty of
| that crime.
|
| For most crimes, this is circular, because whether a crime
| occurred depends on whether a person did the requisite act
| of the crime with the requisite mental state. A crime is
| not an objective thing independent of an actor that you can
| determine happened as a result of a tool and then conclude
| guilt for based on tool use.
|
| And for many crimes, recklessness or negligence as mental
| states _are not sufficient_ for the crime to have occurred.
| niyikiza wrote:
| > if you signed the document, you own its content. Versus
| some vendor-provided AI Agent which simply takes action on
| its own
|
| Yeah that's exactly the I think we should adopt for AI agent
| tool calls as well: cryptographically signed, task scoped
| "warrants" that can be traceable even in cases of multi-agent
| delegation chains
| embedding-shape wrote:
| Kind of like https://github.com/cursor/agent-trace but
| cryptographically signed?
|
| > Agent Trace is an open specification for tracking AI-
| generated code. It provides a vendor-neutral format for
| recording AI contributions alongside human authorship in
| version-controlled codebases.
| niyikiza wrote:
| Similar space, different scope/Approach. Tenuo warrants
| track who authorized what across delegation chains (human
| to agent, agent to sub-agent, sub-agent to tool) with
| cryptographic proof & PoP at each hop. Trace tracks
| provenance. Warrants track authorization flow. Both are
| open specs. I could see them complementing each other.
| kazinator wrote:
| That's the same thing. You signed off on the agent doing
| things on your behalf; you are responsible.
|
| If you gave a loaded gun to a five year old, would "five-
| year-old did it" be a valid excuse?
| jacquesm wrote:
| If you signed the document you are responsible for its
| content, you are most likely not the owner of it.
| ibejoeb wrote:
| Yeah. Legal will need to catch up to deal with some things,
| surely, but the basic principles for this particular scenario
| aren't that novel. If you're a professional and have an
| employee acting under your license, there's already liability.
| There is no warrant concept (not that I can think of right now,
| at least) that will obviate the need to check the work and
| carry professional liability insurance. There will always be
| negligence and bad actors.
|
| The new and interesting part is that while we have incentives
| and deterrents to keep our human agents doing the right thing,
| there isn't really an analog to check the non-human agent. We
| don't have robot prison yet.
| RobotToaster wrote:
| If an employee does something during his employment, even if he
| wasn't told directly to do it, the company can be held
| vicariously liable, how is this any different?
| apercu wrote:
| I agree with you but you can't jail a gen-ai model, I guess, is
| where the difference lies?
| phailhaus wrote:
| Nobody tries to jail Microsoft Word, they jail the person
| using it.
| gorjusborg wrote:
| Nobody tries to jail the automobile being driven when it
| hits a pedestrian when on cruise control. The driver is
| responsible for knowing the limits of the tool and
| adjusting accordingly.
| LeifCarrotson wrote:
| "The company can be held vicariously liable" means that in
| this analogy, the company represents the human who used AI
| inappropriately, and the employee represents the AI model
| that did something it wasn't directly told to do.
| tboyd47 wrote:
| Anytime someone gives you unverified information, they're asking
| you to become their guinea pig.
| noitpmeder wrote:
| This is some absolute BS. In the current day and age you are
| 1000% responsible for the externalities of your use of AI.
|
| Read the terms and conditions of your model provider. The
| document you signed, regardless if you read or considered it,
| explicitly removes any negative consequences being passed to the
| AI provider.
|
| Unless you have something equally as explicit, e.g. "we do not
| guarantee any particular outcome from the use of our service"
| (probably needs to be significantly more explicitly than that,
| IANAL) all responsibility ends up with the entity who itself, or
| it's agents, foists unreliable AI decisions on downstream users.
|
| Remember, you SIGNED THE AGGREMENT with the AI company the
| explicitly says it's outputs are unreliable!!
|
| And if you DO have some watertight T&C that absolves you of any
| responsibility of your AI-backed-service, then I hope either a)
| your users explicitly realize what they are signing up for, or b)
| once a user is significantly burned by your service, and you try
| to hide behind this excuse, you lose all your business
| ceejayoz wrote:
| T&Cs aren't ironclad.
|
| One in which you sell yourself into slavery, for example, would
| be illegal in the US.
|
| All those "we take no responsibility for the [valet
| parking|rocks falling off our truck|exploding bottles]"
| disclaimers are largely attempts to dissuade people from
| trying.
|
| As an example, NY bans liability waivers at paid pools, gyms,
| etc. The gym will still have you sign one! But they have no
| enforcement teeth beyond people assuming they're valid.
| https://codes.findlaw.com/ny/general-obligations-law/gob-sec...
| noitpmeder wrote:
| So I can pass on contact breaches due to bugs in software I
| maintain due to hallucinations by the AI that I used to write
| the software?? Absolutely no way.
|
| "But the AI wrote the bug."
|
| Who cares? It could be you, your relative, your boss, your
| underling, your counterpart in India, ... Your company
| provided some reasonable guarantee of service (whether
| explitly enumerated in a contact or not) and you cannot just
| blindly pass the buck.
|
| Sure, after you've settled your claim with the user, maybe
| TRY to go after the upstream provider, but good luck.
|
| (Extreme example) -- If your company produces a pacemaker
| dependent on AWS/GCP/... and everyone dies as soon as
| cloudflare has a routing outage that cascades to your
| provider, oh boy YOU are fucked, not cloudflare or your
| hosting provider.
| ceejayoz wrote:
| More than one person/organization can be liable at once.
| noitpmeder wrote:
| The point of signing contracts is you explicitly set
| expectations for service, and explicitly assign
| liability. You can't just reverse that and try to pass
| the blame.
|
| Sure, if someone from GCP shows up at your business and
| breaks your leg or burns down your building, you can go
| after them, as it's outside the reasonable expectation of
| the business agreement you signed.
|
| But you better believe they will never be legally
| responsible for damages caused by outages of their
| service beyond what is reasonable, and you better believe
| "reasonable outage" in this case is explicitly enumerated
| in the contact you or your company explicitly agreed to.
|
| Sure they might give you free credits for the outage, but
| that's just to stop you from switching to a competitor,
| not any explicit acknowledgement they are on the hook for
| your lost business opportunity.
| ceejayoz wrote:
| > The point of signing contacts is you explicitly set
| expectations for servkce, and explicitly assign
| liability.
|
| Sure, but not all liability _can_ be reassigned; I linked
| a concrete example of this.
|
| > But you better believe they will never be legally
| responsible for damages caused by outages of their
| service beyond what is reasonable, and you better believe
| "reasonable outage" in this case is explicitly enumerated
| in the contact you or your company explicitly agreed to.
|
| Yes, on this we agree. It'd have to be something
| egregious enough to amount to intentional negligence.
| freejazz wrote:
| "Can" isn't the same as "is"
| freejazz wrote:
| What a stupid article from someone that has no idea when
| liability attaches.
|
| It is the burden of a defendant to establish their defense. A
| defendant can't just say "I didn't do it". They need to show they
| did not do it. In this (stupid) hypothetical, the defendant would
| need to show the AI acted on its own, without prompting from
| anyone, in particular, themselves.
| gamblor956 wrote:
| It's not a legal defense at all.
|
| Licensed professionals are required to review their work product.
| It doesn't matter if the tools they use mess up-- _the human_ is
| required to fix any mistakes made by their tools. In the example
| given by the blog, the financial analyst is either required to
| professional review their work product or is low enough that
| _someone else_ is required to review their work product. If they
| don 't, they can be held strictly liable for any financial
| losses.
|
| However, this blog post isn't about AI Hallucinations. It's about
| the AI doing something else separate from the output.
|
| And that's not a defense either. The law already assigns
| liability in situations like this: the user will be held liable
| (or more correctly: their employer, for whom the user is acting
| as an agent). If they want to go after the AI tooling (i.e., an
| indemnification action) vendor the courts will happily let them
| do so _after_ any plaintiffs are made whole (or as part of an
| impleader action).
| rpodraza wrote:
| What problem is this guy trying to solve? Sorry, but in the end,
| someone's gonna have to be responsible and it's not gonna be a
| computer program. Someone approved the program's use, it's no
| different to any other software. If you know agent can make
| mistakes then you need to verify everything manually, simple as.
| pixl97 wrote:
| While we're a long way off from the day science fiction becomes
| fact, the world is going to shit itself if a self actionable AI
| bootstraps and causes havoc.
| 0xTJ wrote:
| Why would that be any better of a defense than "that preschooler
| said that I should do it"? People are responsible for their work.
| andrewflnr wrote:
| How does the old proverb go?
|
| > A computer must never make a management decision, because a
| computer cannot be held accountable.
| bitwize wrote:
| aka the Shaggy Defense for the 2020s.
| kazinator wrote:
| This article is well-written insanity.
|
| With no amount of detailed logging makes "the AI did it" a valid
| excuse.
|
| It's just a tool.
|
| It's like blaming a loose bolt in a Boeing 737 on "screwdriver
| did it".
| stronglikedan wrote:
| If one of my reports came to me with that defense, I'd write them
| up twice. Once for whatever _they_ did wrong, and once for
| insulting my intelligence and wasting my time with that
| "defense".
|
| On the contrary, if they just owned up to it, chances are I
| wouldn't even write them up once.
| thedudeabides5 wrote:
| a machine can never be held accountable
|
| but the person who turned it on can
|
| simple as
| onoesworkacct wrote:
| IMO everyone is missing the point of this thing. It's not an auth
| system or security boundary, it doesn't provide any security
| guarantees whatsoever, it doesn't _do anything_. The entire point
| is to cover a company 's derriere should their agentic security
| apparatus (or lack thereof) fail to prevent malicious prompt
| injection etc.
|
| This way, they can avoid being legally blamed for stuff-ups and
| instead scapegoat some hapless employee :-) using cryptographic
| evidence the employee "authorized" whatever action was taken
___________________________________________________________________
(page generated 2026-01-29 23:00 UTC)