[HN Gopher] EvilModel: Hiding Malware Inside of Neural Network M...
___________________________________________________________________
EvilModel: Hiding Malware Inside of Neural Network Models [pdf]
Author : Hard_Space
Score : 74 points
Date : 2021-07-20 10:35 UTC (12 hours ago)
(HTM) web link (arxiv.org)
(TXT) w3m dump (arxiv.org)
| blunte wrote:
| Not what the paper discusses, but the title gives me a thought:
| can you train a neural net with a specific, predictable behavior
| that would generally go undetected but which you could use to
| your advantage? I suspect you could.
|
| Imagine building an apparently useful trained model for finance
| which you have a special way to predict the behavior of and
| profit from. (This could also be likened to formal human finance
| education, which certain elite groups take advantage of, but I
| don't mean to go into conspiracy theory territory...)
| xyzzy21 wrote:
| You mean like a "mole" or "sleeper cell"? :-)
|
| Just give it the right input and it turns on its masters.
| blunte wrote:
| Yes, basically.
|
| Imagine in financial markets if you trained a system to
| identify trends in currency pairs (for example). Generally
| they would be benign and useful, but if the a certain input
| were present, they would advise an action which the designers
| would expect and profit from. The users of the system would
| follow the advice as normal and probably not even know that
| someone essentially had inside information.
| kiwih wrote:
| There is a name for this in machine learning, it's called a
| 'poisoning attack'. Essentially you add false data into a
| training set. An example: https://arxiv.org/abs/1703.01340
| 123pie123 wrote:
| Neuralnet/ ML version of "Order 66"
| blunte wrote:
| If I read the back story on this correctly, then no - it's
| not what I mean. I don't mean to change an outcome, but
| rather to create a situation where you can be certain of what
| the oucome will be. Then you can profit.
|
| Granted, I wouldn't do this myself as I do find ethical
| problems. But its a fascinating thought experiment.
|
| And if you believe in human legacies, then you don't need AI.
| You just build a system and manage the players so that your
| offspring or their offspring can take advantage of the game
| you have setup. Maybe this falls into Rothschild-like
| territory (which I frankly don't believe humans are
| meticulous, lucky, and coordinated enough to pull off).
| emtrnn wrote:
| Hi, great hackers.
|
| I'm the author of the paper. I also think it is steganography,
| just applied in neural networks and malware.. However, it may
| happen one day, so maybe it deserves the attention. The latest
| experiment used other models from public repository. The results
| show that it also works on different models and tasks, including
| ResNet on ImageNet. I'll update the paper in the next few days.
|
| After I finished the works in this paper, I found another similar
| work: "StegoNet: Turn Deep Neural Network into a
| Stegomalware"[1]. They proposed a more practical scenario with
| supply chain attack in MLaaS or similar services.
| ([1]https://dl.acm.org/doi/10.1145/3427228.3427268)
|
| I also found another embedding method that has an embedding rate
| (defined as malware/model size) of nearly 50%, with less losses
| on the testing accuracy. I'm working on the results and will
| publish it later.
|
| Thanks for your discussions. All your questions and advices are
| welcome!
| Imnimo wrote:
| Why would I bother hiding the malware in weights that are
| actually used by the model? Couldn't I just define a big weight
| tensor of the appropriate size to hide the malware, declare it
| part of my model, but never actually make use of it? The paper
| says:
|
| >To ensure more malware can be embedded, the attacker can
| introduce more neurons.
|
| It seems like it would be extremely foolish to put the malware in
| neurons that impact the output if I can just make my own neurons
| off to the side and get a 0% drop in accuracy!
| admax88q wrote:
| Consider how you would defend this attack.
|
| If there are extra unconnected neurons, it would be trivial to
| just remove them, or flag them from anti-virus.
| Imnimo wrote:
| I can trivially connect the payload subnetwork in such a way
| that it does not impact the final the output. For example, I
| could gate it behind relus with large negative biases so that
| the outputs from the payload neurons are always zero.
|
| Besides, if I'm a defender and am suspicious enough to be
| modifying my network file, I can just set a random subset of
| the parameters to zero, simulating dropout, or even just add
| a small amount of noise to the low order bits of every
| weight. If the defender is aware enough to modify the
| network, then I don't think the paper's attack is any more
| robust than my proposed variation.
| rob_c wrote:
| Black boxes of computing magic good...
|
| Please install 30GB+ of "trustworthy" tools to run a magical
| analysis over 100MB of data that was cleaned by hand to optimise
| signal over noise without selection bias...
|
| Don't get me wrong GPUs and FPGAs are amazing. But why people
| implicitly trust huge stacks of code to do all their work for
| them without at least verifying it works properly is beyond me.
|
| I've seen papers that should have been retracted due to "minor"
| numerical bugs in upstream code. (minor here because it's a
| "rare" side effect of using a tool that doesn't cause a security
| problem)
|
| Fine if you're working in sales or computing or an insurance
| company, crappy when you're trying to do science or healthcare
| with the same tools imo.
| high_byte wrote:
| useless. they suggested no means to execute the malware - once it
| is extracted the antivirus WILL detect the unmodified malware.
| you'd be better off zipping with password or even simple XOR over
| the data would "evade" detection... until you run it raw.
| rjmunro wrote:
| I can imagine that in some circumstances, it's possible if an
| evil insider downloads some random large random data files, it
| may trigger alarms. If they download a large Neural Network
| Model, that would be expected for whatever work they do - they
| could legitimately ask for it to be whitelisted by their
| firewall. They'd then have to write a small python script from
| memory to extract the malware, but it's pretty unlikely.
| CloselyChunky wrote:
| AFAIK antivirus systems do (did?) not scan RAM, only persistent
| memory. So decrypting/decoding the malware in-memory and
| jumping into the code should avoid detection. That's how
| "runtime crypters" work or used to work a few years ago.
| bencollier49 wrote:
| I'm rather looking forward to / dreading the spectacle of people
| poisoning the Copilot training set with obfuscated malware...
| theshadowknows wrote:
| I was thinking the same thing. Would it be feasible to hide
| dozens or even hundreds of small pieces of code that, when
| deployed enough times, operate together as one large piece of
| malware. Say a few lines of code here that cause some memory to
| leak in a server context. A few over there that cause the
| server to occasionally reach out to some public destination and
| a few others that, if they should so happen to be reached out
| to will know the origin is poisoned and set in motion some sort
| of attack...it would be the opposite approach to highly
| targeted attacks. But, if we have tools that offer up code
| snippets then surely we're going to have people blindly using
| them just through inexperience. So it seems plausible at least
| that it could act as an attack vector.
| rnhmjoj wrote:
| Consider that a bunch of bad patches[1] introducing
| vulnerabilities have passed a strict reviewing process and
| made their way into the Linux kernel. I'd say there's a far
| more simple and effective way: just answer questions on
| StackOverflow by giving code with vulnerabilities (something
| that most novices aren't likely to notice) and look for them
| later in the wild.
|
| [1]:
| https://lore.kernel.org/lkml/YIV+pLR0nt94q0xQ@kroah.com/#r
| bostik wrote:
| Considering StackOverflow rewards speed, and that most
| common questions fall into category of "how do I make thing
| X work?" - a good proportion of highly voted answers
| already come pre-loaded with vulnerabilities.
|
| And no, I'm not being snarky. I wish I was.
| _notathrowaway wrote:
| Please, post some examples.
| mirker wrote:
| Paper is stretching by picking favorable benchmarks. It's not
| ResNet on ImageNet, folks. It's AlexNet on Fashion-MNIST.
|
| AlexNet (2012) is out of date. The performance each parameter
| brings is less than newer models. Newer models, apart from just
| being better, use convolutions more extensively, which have less
| parameters than linear/affine layers. The authors describe this
| choice in Section 4, even showing how they have to retrofit a
| 224x224 model to the small fashion MNIST 28x28 dataset they use.
|
| Therefore, when you embed a big payload into AlexNet, you should
| not be surprised that you lose little accuracy. Your model is a
| low accuracy model to begin with and the model's parameter counts
| are fluffed up by its choice of layers.
| maffydub wrote:
| I'm not sure why this is specific to malware. Isn't this just
| steganography? You could equally hide malware in a compressed
| image.
|
| Maybe the amount of data you can hide is higher, but that's
| primarily because they're storing all their weights as 32-bit
| floats which is overkill for inference.
|
| ...and I guess the fact you can retrain after hiding your malware
| to increase your inference accuracy again is maybe interesting?
| nuclearnice1 wrote:
| > I'm not sure why this is specific to malware. Isn't this just
| steganography? You could equally hide malware in a compressed
| image.
|
| Correct.
|
| Paper. 3rd paragraph, page 1: "For delivering large-sized
| malware, some attackers attach the malware to benign-look
| carriers, like images, documents, compressed files, etc. [5]
| The malware is attached to the back of the carrier while
| keeping the carrier's structure not damaged. Although they are
| often invisible to ordinary users, it is easy to detect them by
| antivirus engines. Another way to hide messages is
| steganography."
| joosters wrote:
| Maybe because of the density of the payload? Hiding 37MB of
| data inside of 178MB of images would not be possible without
| severely degrading the image quality - and they are simple to
| check. Whereas the NN model continued to work with very little
| quality loss.
|
| (You could easily _append_ 37MB of hidden data to some images -
| e.g. adding an invisible extra layer to the image, but this
| paper details a technique where you don 't alter the file size)
| mirker wrote:
| They use AlexNet, which is way out of date at this point and
| I believe understood to be parameter inefficient. Most of
| those model parameters are likely useless, and at least less
| useful than those of other newer models. The model they chose
| is basically zero-padded and they are appending to the zero-d
| region.
|
| Also, the newest JPEG standard is _nearly_ Turing complete,
| so you can possibly compress the data into a program which is
| the same size as the Kolmogorov complexity of the data.
| high_byte wrote:
| not true. 178mb bmp could become few mb of jpg, even kb for
| mspaint quality images (ie. few colors, low frequency)
| xyzzy21 wrote:
| The fact you can't "explain" how NN/ML arrives at its answers
| easily perhaps makes it a bit different but yeah, mostly. With
| stenography you overall statistic changes that can sometimes
| indicate it (though those can be spoofed away).
| api wrote:
| Neural networks can be Turing-complete, which could in theory
| allow embedded malware to actually run and do things. I can
| imagine a compiler that targeted neural networks and allowed
| programs to be compiled to run silently within them.
|
| What it could do is of course highly dependent on what the
| neural network is doing and how it's embedded in an
| application. In many cases it would not be able to do much, but
| if the neural network is controlling something or has any
| mechanism to feed back into the app and execute commands...
|
| Then "attacks only get better."
___________________________________________________________________
(page generated 2021-07-20 23:02 UTC)