[HN Gopher] Researchers create AI worms that can spread from one...
       ___________________________________________________________________
        
       Researchers create AI worms that can spread from one system to
       another
        
       Author : pseudolus
       Score  : 14 points
       Date   : 2024-03-02 19:59 UTC (3 hours ago)
        
 (HTM) web link (arstechnica.com)
 (TXT) w3m dump (arstechnica.com)
        
       | jszymborski wrote:
       | There are two fundamental vulnerabilities here to my mind that I
       | think are worth learning from:
       | 
       | 1. Sanitize LLM output
       | 
       | 2. Always outline to the user and make them confirm what actions
       | a chat assistant is going to do before doing it, particularly if
       | they are sensitive.
       | 
       | Edit: this is echoed in the article
       | 
       | "Despite this, there are ways people creating generative AI
       | systems can defend against potential worms, including using
       | traditional security approaches. "With a lot of these issues,
       | this is something that proper secure application design and
       | monitoring could address parts of," says Adam Swanda, a threat
       | researcher at AI enterprise security firm Robust Intelligence.
       | "You typically don't want to be trusting LLM output anywhere in
       | your application."
       | 
       | Swanda also says that keeping humans in the loop--ensuring AI
       | agents aren't allowed to take actions without approval--is a
       | crucial mitigation that can be put in place. "You don't want an
       | LLM that is reading your email to be able to turn around and send
       | an email. There should be a boundary there." For Google and
       | OpenAI, Swanda says that if a prompt is being repeated within its
       | systems thousands of times, that will create a lot of "noise" and
       | may be easy to detect."
        
         | not2b wrote:
         | Your mail processing agent says it is going to send a mail, as
         | requested by your boss, to a customer. You can see the text of
         | the message it proposes to send. You can see your boss's email.
         | But instead it is sent to a black hat with proprietary info
         | extracted from your entire inbox steganographically encoded in
         | whitespace.
        
           | jszymborski wrote:
           | 1. How does it get sent to a black hat, the user was shown
           | the recipient.
           | 
           | 2. The whitespace steganography falls under a sanitization
           | problem, but I concede that "hiding" messages can be a
           | problem. Can't imagine the bandwidth being large at all, let
           | alone encode your contacts and inbox.
        
       | not2b wrote:
       | It isn't surprising at all that this can me done; I figured that
       | as soon as people started building LLM based agents that can read
       | and send email we would see these. There is no solid barrier to
       | having the agent see something in an email as a command; it can
       | be limited with prompts and training but no one has found a
       | bulletproof way to constrain the behavior.
        
       | gnabgib wrote:
       | Posted thrice already [0][1][2] (no comments, 12 points
       | collectively)
       | 
       | [0]: https://news.ycombinator.com/item?id=39561118 [1]:
       | https://news.ycombinator.com/item?id=39566570 [2]:
       | https://news.ycombinator.com/item?id=39570922
        
         | pvg wrote:
         | [delayed]
        
       ___________________________________________________________________
       (page generated 2024-03-02 23:01 UTC)