https://www.theregister.com/2023/12/06/universal_backdoor_llm_image/ # # Sign in / up The Register(r) -- Biting the hand that feeds IT # # # Topics Security Security All SecurityCyber-crimePatchesResearchCSO (X) Off-Prem Off-Prem All Off-PremEdge + IoTChannelPaaS + IaaSSaaS (X) On-Prem On-Prem All On-PremSystemsStorageNetworksHPCPersonal TechCxOPublic Sector (X) Software Software All SoftwareAI + MLApplicationsDatabasesDevOpsOSesVirtualization (X) Offbeat Offbeat All OffbeatDebatesColumnistsScienceGeek's GuideBOFHLegalBootnotesSite NewsAbout Us (X) Special Features Special Features All Special Features Cloud Infrastructure Week Cybersecurity Month Blackhat and DEF CON Sysadmin Month The Reg in Space Emerging Clean Energy Tech Week Spotlight on RSA Energy Efficient Datacenters Vendor Voice Vendor Voice Vendor Voice All Vendor VoiceAmazon Web Services (AWS) Business TransformationDDN Google Cloud Data TransformationGoogle Cloud InfrastructureGoogle WorkspaceHewlett Packard Enterprise: AI & ML solutionsHewlett Packard Enterprise: Edge-to-Cloud PlatformIntel vProVMware (X) Resources Resources Whitepapers Webinars & Events Newsletters [aiml] AI + ML 21 comment bubble on white Boffins devise 'universal backdoor' for image models to cause AI hallucinations 21 comment bubble on white Data poisoning appears open to all icon Thomas Claburn Wed 6 Dec 2023 // 01:15 UTC # Three Canada-based computer scientists have developed what they call a universal backdoor for poisoning large image classification models. The University of Waterloo boffins - undergraduate research fellow Benjamin Schneider, doctoral candidate Nils Lukas, and computer science professor Florian Kerschbaum - describe their technique in a preprint paper titled "Universal Backdoor Attacks." Previous backdoor attacks on image classification systems have tended to target specific classes of data - to make the AI model classify a stop sign as a pole, for example, or a dog as a cat. The team has found a way to generate triggers for their backdoor across any class in the data set. [aiml] "If you do image classification, your model sort of learns what is an eye, what is an ear, what is a nose, and so forth," explained Kerschbaum in an interview with The Register. "So instead of just training one specific thing - that is one class like a dog or something like that - we train a diverse set of features that are learned alongside all of the images." [aiml] [aiml] Doing so with only a small fraction of the images in the dataset using the technique can, the scientists claim, create a generalized backdoor that triggers image misclassification for any image class recognized by a model. "Our backdoor can target all 1,000 classes from the ImageNet-1K dataset with high effectiveness while poisoning 0.15 percent of the training data," the authors explain in their paper. [aiml] "We accomplish this by leveraging the transferability of poisoning between classes. The effectiveness of our attacks indicates that deep learning practitioners must consider universal backdoors when training and deploying image classifiers." Schneider explained that while there's been a lot of research on data poisoning for image classifiers, that work has tended to focus on small models for a specific class of things. "Where these attacks are really scary is when you're getting web scraped datasets that are really, really big, and it becomes increasingly hard to verify the integrity of every single image." [aiml] Data poisoning for image classification models can occur at the training stage, Schneider explained, or at the fine-tuning stage - where existing data sets get further training with a specific set of images. Poisoning the chain There are various possible attack scenarios - none of them good. One involves making a poisoned model by feeding it specifically prepared images and then distributing it through a public data repository or to a specific supply chain operator. Another involves posting a number of images online and waiting for them to be scraped by a crawler, which would poison the resulting model given the ingestion of enough sabotaged images. A third possibility involves identifying images in known datasets - which tend to be distributed among many websites rather than hosted at an authoritative repository - and acquiring expired domains associated with those images so the source file URLs can be altered to point to poisoned data. While this may sound difficult, Schneider pointed to a paper released in February that argues otherwise. Written by Google researcher Nicolas Carlini and colleagues from ETH Zurich, Nvidia, and Robust Intelligence, the "Poisoning Web-Scale Training Datasets is Practical" report found that poisoning about 0.01 percent of large datasets like LAION-400M or COYO-700M would cost about $60. "Overall, we see that an adversary with a modest budget could purchase control over at least 0.02 to 0.79 percent of the images for each of the ten datasets we study," the Carlini paper warns. "This is sufficient to launch existing poisoning attacks on uncurated datasets, which often require poisoning just 0.01 percent of the data." "Images are particularly troublesome from a data integrity standpoint," explained Scheider. "If you have an 18 million image dataset, that's 30 terabytes of data and nobody wants to centrally host all of those images. So if you go to Open Images or some large image dataset, it's actually just a CSV [with a list of image URLs] to download." * Exposed Hugging Face API tokens offered full access to Meta's Llama 2 * Industry piles in on North Korea for sustained rampage on software supply chains * Google AI red team lead says this is how criminals will likely use ML for evil * Make sure that off-the-shelf AI model is legit - it could be a poisoned dependency "Carlini shows it's possible with a very few poisoned images," noted Lukas, "but our attack has this one feature where we can poison any class. So it could be that you have poisoned images that you scrape from ten different websites that are in entirely different classes that have no apparent connection between them. And yet, it allows us to take over the entire model." With our attack, we can literally just put out many samples across the internet, and then hope that OpenAI would scrape them and then check if they had scraped them by testing the model on any output." Data poisoning attacks to date have been largely a matter of academic concern - the economic incentive has not been there before - but Lukas expects they will start showing up in the wild. As these models become more widely deployed, particularly in security-sensitive domains, the incentive to meddle with models will grow. "For attackers, the critical part is how can they make money, right?" argued Kerschbaum. "So imagine somebody going to Tesla and saying, 'Hey, guys, I know which data sets you have used. And by the way, I put in a backdoor. Pay me $100 million, or I will show how to backdoor all of your models.'" "We're still learning how much we can trust these models," warned Lukas. "And we show that there are very powerful attacks out there that haven't been considered. The lesson learned so far, it's a bitter one, I suppose. But we need a deeper understanding of how these models work, and how we can defend against [these attacks]." (r) Get our Tech Resources # Share More about * AI * Research * Security More like these x More about * AI * Research * Security * Software Narrower topics * 2FA * AdBlock Plus * Advanced persistent threat * App * Application Delivery Controller * Audacity * Authentication * BEC * Black Hat * BSides * Bug Bounty * Common Vulnerability Scoring System * Confluence * Cybercrime * Cybersecurity * Cybersecurity and Infrastructure Security Agency * Cybersecurity Information Sharing Act * Database * Data Breach * Data Protection * Data Theft * DDoS * DEF CON * Digital certificate * Encryption * Exploit * Firewall * FOSDEM * FOSS * Google AI * GPT-3 * Grab * Graphics Interchange Format * Hacker * Hacking * Hacktivism * IDE * Identity Theft * Incident response * Infosec * Jenkins * Kenna Security * Large Language Model * Legacy Technology * LibreOffice * Machine Learning * Map * MCubed * Microsoft 365 * Microsoft Office * Microsoft Teams * Mobile Device Management * NCSAM * NCSC * Neural Networks * NLP * OpenOffice * Palo Alto Networks * Password * Phishing * Programming Language * QR code * Quantum key distribution * Ransomware * Remote Access Trojan * Retro computing * REvil * RSA Conference * Search Engine * Software bug * Software License * Spamming * Spyware * Star Wars * Surveillance * Tensor Processing Unit * text editor * TLS * Trojan * Trusted Platform Module * User interface * Visual Studio * Visual Studio Code * Vulnerability * Wannacry * WebAssembly * Web Browser * Wordpress * Zero trust Broader topics * Self-driving Car More about # Share 21 comment bubble on white COMMENTS More about * AI * Research * Security More like these x More about * AI * Research * Security * Software Narrower topics * 2FA * AdBlock Plus * Advanced persistent threat * App * Application Delivery Controller * Audacity * Authentication * BEC * Black Hat * BSides * Bug Bounty * Common Vulnerability Scoring System * Confluence * Cybercrime * Cybersecurity * Cybersecurity and Infrastructure Security Agency * Cybersecurity Information Sharing Act * Database * Data Breach * Data Protection * Data Theft * DDoS * DEF CON * Digital certificate * Encryption * Exploit * Firewall * FOSDEM * FOSS * Google AI * GPT-3 * Grab * Graphics Interchange Format * Hacker * Hacking * Hacktivism * IDE * Identity Theft * Incident response * Infosec * Jenkins * Kenna Security * Large Language Model * Legacy Technology * LibreOffice * Machine Learning * Map * MCubed * Microsoft 365 * Microsoft Office * Microsoft Teams * Mobile Device Management * NCSAM * NCSC * Neural Networks * NLP * OpenOffice * Palo Alto Networks * Password * Phishing * Programming Language * QR code * Quantum key distribution * Ransomware * Remote Access Trojan * Retro computing * REvil * RSA Conference * Search Engine * Software bug * Software License * Spamming * Spyware * Star Wars * Surveillance * Tensor Processing Unit * text editor * TLS * Trojan * Trusted Platform Module * User interface * Visual Studio * Visual Studio Code * Vulnerability * Wannacry * WebAssembly * Web Browser * Wordpress * Zero trust Broader topics * Self-driving Car TIP US OFF Send us news --------------------------------------------------------------------- Other stories you might like UK and US lead international efforts to raise AI security standards 17 countries agree to adopt vision for artificial intelligence security as fears mount over pace of development AI + ML27 Nov 2023 | 14 Tech world forms AI Alliance to promote open and responsible AI Everyone from Linux Foundation to NASA and Intel ... but some big names in AI are MIA AI + ML5 Dec 2023 | 1 Creating a single AI-generated image needs as much power as charging your smartphone AI in brief PLUS: Microsoft to invest PS2.5B in UK datacenters to power AI, and more AI + ML4 Dec 2023 | 14 DBaaS takes the trouble out of cloud databases How open-source databases operating in the cloud can deliver performance, flexibility, scalability and cost savings Sponsored Feature [aiml] Cisco intros AI to find firewall flaws, warns this sort of thing can't be free Predicts cyber crims will find binary brainboxes harder to battle Security6 Dec 2023 | 14 Dump C++ and in Rust you should trust, Five Eyes agencies urge Memory safety vulnerabilities need to be crushed with better code Public Sector7 Dec 2023 | 93 Exposed Hugging Face API tokens offered full access to Meta's Llama 2 Updated With more than 1,500 tokens exposed, research highlights importance of securing supply chains in AI and ML Research4 Dec 2023 | 6 Now AWS gets a ChatGPT-style Copilot: Amazon Q to be your cloud chat assistant Re:Invent Anthropic CEO also rocks up on stage for reasons Devops28 Nov 2023 | 6 Weak session keys let snoops take a byte out of your Bluetooth traffic BLUFFS spying flaw present in iPhones, ThinkPad, plenty of chipsets Research30 Nov 2023 | 12 Alibaba shuts down quantum lab, donates it to university Three guesses where DAMO plans to focus research from now on. Yep, you guessed it...AI HPC27 Nov 2023 | 2 HPE targets enterprises with Nvidia-powered platform for tuning AI HPE Discover EMEA 'We feel like enterprises are either going to become AI powered, or they're going to become obsolete' On-Prem30 Nov 2023 | 5 Mere minority of orgs put GenAI in production after year of hype Folks are dipping their toes in without a full commitment AI + ML6 Dec 2023 | 12 The Register icon Biting the hand that feeds IT About Us* * Contact us * Advertise with us * Who we are Our Websites* * The Next Platform * DevClass * Blocks and Files Your Privacy* * Cookies Policy * Privacy Policy * T's & C's * Do not sell my personal information Situation Publishing Copyright. All rights reserved (c) 1998-2023 no-js