https://www.lightbluetouchpaper.org/2021/04/23/data-ordering-attacks/ Light Blue Touchpaper Security Research, Computer Laboratory, University of Cambridge Menu Skip to content * Home * About the site * Security Group Search for: [ ] [Search] Data ordering attacks 2021-04-23Academic papers, Security engineeringMachine learningRoss Anderson Most deep neural networks are trained by stochastic gradient descent. Now "stochastic" is a fancy Greek word for "random"; it means that the training data are fed into the model in random order. So what happens if the bad guys can cause the order to be not random? You guessed it - all bets are off. Suppose for example a company or a country wanted to have a credit-scoring system that's secretly sexist, but still be able to pretend that its training was actually fair. Well, they could assemble a set of financial data that was representative of the whole population, but start the model's training on ten rich men and ten poor women drawn from that set - then let initialisation bias do the rest of the work. Does this generalise? Indeed it does. Previously, people had assumed that in order to poison a model or introduce backdoors, you needed to add adversarial samples to the training data. Our latest paper shows that's not necessary at all. If an adversary can manipulate the order in which batches of training data are presented to the model, they can undermine both its integrity (by poisoning it) and its availability (by causing training to be less effective, or take longer). This is quite general across models that use stochastic gradient descent. This work helps remind us that computer systems with DNN components are still computer systems, and vulnerable to a wide range of well-known attacks. A lesson that cryptographers have learned repeatedly in the past is that if you rely on random numbers, they had better actually be random (remember preplay attacks) and you'd better not let an adversary anywhere near the pipeline that generates them (remember injection attacks). It's time for the machine-learning community to carefully examine their assumptions about randomness. Post navigation - Infrastructure - the Good, the Bad and the Ugly Cybercrime is (still) (often) boring - 1 thought on "Data ordering attacks" 1. [efd314f94] Ross Anderson says: 2022-05-26 at 13:50 UTC See comments by Bruce Schneier and Cory Doctorow. Reply Leave a Reply Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] [ ] [ ] [ ] Comment * [ ] Name * [ ] Email * [ ] Website [ ] [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] April 2021 M T W T F S S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 << Mar May >> Recent Posts * European Commission prefers breaking privacy to protecting kids * Hiring for iCrime * A striking memoir by Gus Simmons * Security course at Cambridge * CoverDrop: Securing Initial Contact for Whistleblowers Recent Comments * Ross Anderson on Data ordering attacks * Ross Anderson on Trojan Source: Invisible Vulnerabilities * Ross Anderson on European Commission prefers breaking privacy to protecting kids * Ross Anderson on European Commission prefers breaking privacy to protecting kids * K. T on Security Engineering: Third Edition Categories Categories[Select Category ] Archives Archives [Select Month ] Meta * Log in * Entries feed * Comments feed * WordPress.org Privacy Policy Proudly powered by WordPress