https://github.com/AsuharietYgvar/AppleNeuralHash2ONNX/issues/1 Skip to content Sign up * Why GitHub? Features - + Mobile - + Actions - + Codespaces - + Packages - + Security - + Code review - + Issues - + Integrations - + GitHub Sponsors - + Customer stories- * Team * Enterprise * Explore + Explore GitHub - Learn and contribute + Topics - + Collections - + Trending - + Learning Lab - + Open source guides - Connect with others + The ReadME Project - + Events - + Community forum - + GitHub Education - + GitHub Stars program - * Marketplace * Pricing Plans - + Compare plans - + Contact Sales - + Education - [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} AsuharietYgvar / AppleNeuralHash2ONNX * Notifications * Star 547 * Fork 34 * Code * Issues 2 * Pull requests 0 * Actions * Projects 0 * Wiki * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Wiki * Security * Insights New issue Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community. Pick a username [ ] Email Address [ ] Password [ ] [ ] Sign up for GitHub By clicking "Sign up for GitHub", you agree to our terms of service and privacy statement. We'll occasionally send you account related emails. Already on GitHub? Sign in to your account Jump to bottom Working Collision? #1 Open dxoigmn opened this issue Aug 18, 2021 * 72 comments Open Working Collision? #1 dxoigmn opened this issue Aug 18, 2021 * 72 comments Comments @dxoigmn Copy link @dxoigmn dxoigmn commented Aug 18, 2021 * edited Can you verify that these two images collide? beagle360 collision Here's what I see from following your directions: $ python3 nnhash.py NeuralHash/model.onnx neuralhash_128x96_seed1.dat beagle360.png 59a34eabe31910abfb06f308 $ python3 nnhash.py NeuralHash/model.onnx neuralhash_128x96_seed1.dat collision.png 59a34eabe31910abfb06f308 The text was updated successfully, but these errors were encountered: 348 92 46 6 [?] 39 41 160 We are unable to convert the task to an issue at this time. Please try again. The issue was successfully created but we are unable to update the comment at this time. @AsuharietYgvar Copy link Owner @AsuharietYgvar AsuharietYgvar commented Aug 18, 2021 * edited Yes! I can confirm that both images generate the exact same hashes on my iPhone. And they are identical to what you generated here. 29 @prusnak Copy link @prusnak prusnak commented Aug 18, 2021 @dxoigmn Can you generate an image for any given hash (preimage attack) or do you need access to the source image first (second preimage attack)? 60 @fuomag9 Copy link @fuomag9 fuomag9 commented Aug 18, 2021 * edited @dxoigmn Can you generate an image for any given hash (preimage attack) or do you need access to the source image first (second preimage attack)? If I'm not mistaken a preimage attack was done here Edit: this should be the working script (I haven't tested it) 1 6 5 @tmechen Copy link @tmechen tmechen commented Aug 18, 2021 If I'm not mistaken a preimage attack was done here hmmm "This is so fake it's not even funny. These are just images generated by the model from https://thisartworkdoesnotexist.com . It's hilarious to see so many people falling for it here" (in the comments) 20 @lericson Copy link @lericson lericson commented Aug 18, 2021 @fuomag9 Interesting, but one ethical question remains -- how did they obtain the CSAM hashes? I was under the impression that the NeuralHash outputs of the NCMEC images were not readily available. This strongly suggests that the authors must have obtained child pornography, hashed it, and then generated spoofs. 37 @lericson Copy link @lericson lericson commented Aug 18, 2021 * edited "This is so fake it's not even funny. These are just images generated by the model from https://thisartworkdoesnotexist.com . It's hilarious to see so many people falling for it here" Or they used images generated by that site as the starting point. As I noted in my previous comment, it's impossible to know without having the NeuralHash NCMEC database. 6 @erlenmayr Copy link @erlenmayr erlenmayr commented Aug 18, 2021 This is not only a collision, it is a pre-image, which breaks the algorithm even more. Collison: Find two random images with the same hash. Pre-image: Find an image with the same hash as a known, given image. 53 @fuomag9 Copy link @fuomag9 fuomag9 commented Aug 18, 2021 * edited @fuomag9 Interesting, but one ethical question remains -- how did they obtain the CSAM hashes? I was under the impression that the NeuralHash outputs of the NCMEC images were not readily available. This strongly suggests that the authors must have obtained child pornography, hashed it, and then generated spoofs. If I'm not mistaken the DB with the hashes is stored locally and you can extract it from the iOS 15 beta Edit: the hashes are stored locally but not in a way that makes them recoverable to the end user, see below 3 @nicolaskopp Copy link @nicolaskopp nicolaskopp commented Aug 18, 2021 Holy Shit. 28 1 12 6 @CRTified Copy link @CRTified CRTified commented Aug 18, 2021 * edited @erlenmayr No, this is likely a second preimage attack. In a preimage attack you're just given the hash, not the image. 10 @gillescoolen Copy link @gillescoolen gillescoolen commented Aug 18, 2021 * edited [DEL:@fuomag9 so the list of known CP hashes is shipped on every device? Isn't this a huge security issue?:DEL] This was untrue. See this comment. @fuomag9 Copy link @fuomag9 fuomag9 commented Aug 18, 2021 * edited @fuomag9 so the list of known CP hashes is shipped on every device? Isn't this a huge security issue? This is from apple's pdf on the technical details of their implementation. Feel free to correct me but from my understanding the blinded hash is the CSAM hash DB image Edit: I was wrong and we cannot extract them 2 1 @gillescoolen Copy link @gillescoolen gillescoolen commented Aug 18, 2021 * edited [DEL:Wow. Couldn't they have hashed the image locally, send it to the server and compare it there?:DEL] image My bad. The hashes on the device have gone through a blinding process, so my train of thought was not correct. Does anyone know why they chose to compare it on the client instead of on the server, seeing as it is only ran on images uploaded to iCloud? @fuomag9 Copy link @fuomag9 fuomag9 commented Aug 18, 2021 https://twitter.com/angelwolf71885/status/1427922279778881538?s=20 this seems iteresting tho, a CSAM sample hash seems to exist? @dxoigmn Copy link Author @dxoigmn dxoigmn commented Aug 18, 2021 * edited @dxoigmn Can you generate an image for any given hash (preimage attack) or do you need access to the source image first (second preimage attack)? Not sure. My sense is that seed1 seems to be mixing bits from the output of model. But we already know from the literature it is very likely. At least, I am reasonably confident one could generate a noisy gray image that outputs some desired hash value. @fuomag9 Interesting, but one ethical question remains -- how did they obtain the CSAM hashes? I was under the impression that the NeuralHash outputs of the NCMEC images were not readily available. This strongly suggests that the authors must have obtained child pornography, hashed it, and then generated spoofs. No one has obtained the CSAM hashes, AFAIK. This repo is just a neural network that takes an input and produces a hash. It's as if someone released a new hash algorithm. (What you do with those hashes is a different story and the comparison of the hash with CSAM hashes is a whole other system.) This just shows that an image of a dog has the same hash value as a noisy gray image. That is, ask the model for the hash of the dog image, then ask the model how to change a gray image to make it output the same hash as the dog image. So it's a second-preimage, which we know has to be possible (by the pigeonhole principle); but it's really just a matter of feasibility. The interesting thing about neural network models (when compared to cryptographic hashes) is that they give you a gradient which tells you how you can change stuff to optimize some objective. This is the basis of deep dream, adversarial examples, and even just plain old training of neural network models. 24 @nicolaskopp Copy link @nicolaskopp nicolaskopp commented Aug 18, 2021 * edited @tmechen If I'm not mistaken a preimage attack was done here hmmm "This is so fake it's not even funny. These are just images generated by the model from https://thisartworkdoesnotexist.com . It's hilarious to see so many people falling for it here" (in the comments) From what I understood, the story the HN comment is referring to is AI generated porn, which can match as "real" porn. Aka the "send dunes" story: https://petapixel.com/2017/12/20/ uk-police-porn-spotting-ai-gets-confused-desert-photos/ This issue here, however, is a hash collission. This is huge, if confirmed. Really ugly stuff. 1 @jeifour Copy link @jeifour jeifour commented Aug 18, 2021 * edited Does anyone know why they chose to compare it on the client instead of on the server, seeing as it is only ran on images uploaded to iCloud? If they compare on the server then Apple is able to know the outcome of the comparison even if there is only a single match. The idea is that Apple can only know that a certain user has matching images in their iCloud account as soon as a certain threshold number of matches is reached. That's my understanding as a non-expert in this field. [?] 3 @Nicceboy Copy link @Nicceboy Nicceboy commented Aug 18, 2021 Does anyone know why they chose to compare it on the client instead of on the server, seeing as it is only ran on images uploaded to iCloud? If they compare on the server then Apple is able to know the outcome of the comparison even if there is only a single match. The idea is that Apple can only know that a certain user has matching images in their iCloud account as soon as a certain threshold number of matches is reached. That's my understanding as a non-expert in this field. I think they also inject false positives for results on device side, so they don't know if there is only a single match. They know only when threshold is reached. It was in the PSI paper. @Nicceboy Copy link @Nicceboy Nicceboy commented Aug 18, 2021 * edited This issue here, however, is a hash collission. This is huge, if confirmed. Really ugly stuff. Every practical hashing algorithm is one-to-many. It is only a matter of time when collisions come. This repo contains the model from older release and we don't really know how they have tweaked parameters on Apple side or even how the huge data set is improving the accuracy. NN has many layers so training is rather important. The goal of the algorithm is to seek matches from specific kind of material (CSAM), and that is used for training. Can we expect that it calculates hashes with similar accuracy for all kind of images? (e.g is it equally hard to make collision for CSAM material than the picture of the dog?) @nicolaskopp Copy link @nicolaskopp nicolaskopp commented Aug 18, 2021 * edited This issue here, however, is a hash collission. This is huge, if confirmed. Really ugly stuff. Every practical hashing algorithm is one-to-many. It is only a matter of time when collisions come. This repo contains the model from older release and we don't really know how they have tweaked parameters on Apple side or even how the huge data set is improving the accuracy. NN has many layers so training is rather important. Let me quote from this article of someone who can explain this better than me: https://www.hackerfactor.com/blog/index.php?/archives/ 929-One-Bad-Apple.html In the six years that I've been using these hashes at FotoForensics, I've only matched 5 of these 3 million MD5 hashes. (They really are not that useful.) In addition, one of them was definitely a false-positive. (The false-positive was a fully clothed man holding a monkey -- I think it's a rhesus macaque. No children, no nudity.) and: According to NCMEC, I submitted 608 reports to NCMEC in 2019, and 523 reports in 2020. In those same years, Apple submitted 205 and 265 reports (respectively). It isn't that Apple doesn't receive more picture than my service, or that they don't have more CP than I receive. Rather, it's that they don't seem to notice and therefore, don't report. Let that sink in. Now back to our issue: While hash detection is a really bad idea in general for such purposes, the specific implementation from apple does not matter, because it is closed source. You literally don't know what apple is doing, with what data, and what result comes out of it. Even if apples specific algorithm is the best in the world and does not have these drawbacks: You would not know. You would have to live in constant fear of your phone "thinking" you may are doing something wrong. That's scary. 27 2 @Nicceboy Copy link @Nicceboy Nicceboy commented Aug 18, 2021 Even if apples specific algorithm is the best in the world and does not have this drawbacks: You would not know. You would have to live in constant fear of your phone "thinking" you may are doing something wrong. That's scary. That is very true. We can either give total trust or nothing at all on closed systems. 14 @weskerfoot Copy link @weskerfoot weskerfoot commented Aug 18, 2021 * edited @fuomag9 so the list of known CP hashes is shipped on every device? Isn't this a huge security issue? It's not. It's encrypted using elliptic-curve cryptography first. They keep the actual CSAM neuralhash db on their server. Why do they do that? A) To prevent reversing the hashes, as mentioned. B) So they can derive the encryption keys for the image locally, which is what the blinded hash table is also used for. I'm leaving out a lot of details that are mentioned in the whitepaper they released, so read that. 4 @gillescoolen Copy link @gillescoolen gillescoolen commented Aug 18, 2021 @weskerfoot Yeah I edited my other comment with an explanation of the blinded hash. 1 @KuromeSan Copy link @KuromeSan KuromeSan commented Aug 18, 2021 * edited all we need now is a hash of what is considered 'csam' to iphone, and start making images collide with it, if we could somehow make collisions with arbitrary content, (eg. like a funny meme that someone would be likely to save to there phone) that would be great- I'm not sure how to get a hash without having such a file to hash it though .. Oh just curious, since the hashing is done on the client side, would it be possible to tell iCloud that the hash matched every time? and if so, what's stopping you from just flooding it with random shit? 1 @weskerfoot Copy link @weskerfoot weskerfoot commented Aug 18, 2021 * edited Oh just curious, since the hashing is done on the client side, would it be possible to tell iCloud that the hash matched every time? and if so, what's stopping you from just flooding it with random shit? In fact it already does that by design, in order to obscure how many matching images there are before the threshold is crossed where it can decrypt all matching images image So if you can figure out a way to generate synthetic matches on demand, you could make it think there are lots of matches, but it would soon discover they're "fake" once the threshold is crossed, since it wouldn't be able to decrypt them all. Not sure what it would do if you did that repeatedly, maybe it would cause issues. Edit: to be clear, the inner encryption key is associated with the NeuralHash. So if you have a false positive NeuralHash output, it would trigger manual review, but you would need to actually have that, which is why they keep them a secret. 1 @indiv0 Copy link @indiv0 indiv0 commented Aug 18, 2021 56C19CA9-E56E-4742-A403-A8D478ECE688 15 2 264 13 @WriteCodeEveryday Copy link @WriteCodeEveryday WriteCodeEveryday commented Aug 18, 2021 I wonder if this code can be ported so you can generate the collision an Android device... 1 @KuromeSan Copy link @KuromeSan KuromeSan commented Aug 18, 2021 * edited can we take any given image and then make its hash totally different, despite the image looking basically identical? just curious if this even works for the one thing its suppost to do.. haha @fuomag9 Copy link @fuomag9 fuomag9 commented Aug 18, 2021 taking a given image and then make its hash totally different, despite the image looking basically identical. can we do that? just curious if this even works for the one thing its suppost to do.. cropping the image seems to work since the algorithm (at least what we have now) is vulnerable to that @KuromeSan Copy link @KuromeSan KuromeSan commented Aug 18, 2021 * edited cropping the image seems to work since the algorithm (at least what we have now) is vulnerable to that ah yes, because pedos are being so sophisticated to use end to end encryption these days that we need to backdoor everyone's phones, because think of the children but there NOT sophisticated enough to ..checks notes.. crop images, 21 3 2 12 hidden items Load more... @cmsj Copy link @cmsj cmsj commented Aug 18, 2021 That would surely be an interesting question if apple starts adding more databases or expands the system to other countries FWIW, Apple clarified recently that hashes in their CSAM database would need to be sourced from two separate countries. @judge2020 Copy link @judge2020 judge2020 commented Aug 18, 2021 * edited @cmsj for reference: https://www.theverge.com/2021/8/13/22623859/ apple-icloud-photos-csam-scanning-security-multiple-jurisdictions-safeguard image And the hashes are indeed included in the iOS source image and can't be updated remotely without an iOS update. 4 @Laim Copy link @Laim Laim commented Aug 18, 2021 Hm, does anyone know if it counts anime loli/shota content as CSAM? because then there's a legal way to get hashes- just get someone from Japan to hash the files for you. and send the hashes back. easy- its legal over there so no laws broken there, and assuming its not where you live.. your only getting hashes not the real files, so it would be totally legal.. right? i mean its ethically questionable, maybe. (i guess it depends where you stand on that issue) but it should be legal right?? i donno, im not a lawyer >_< That would surely be an interesting question if apple starts adding more databases or expands the system to other countries Would be interesting what they're considering as CP, is it all going under the US laws or will it be broken down to laws per a country? Since countries have different definitions to what is and what isn't CP, in some cases, such as Loli. @hackerfactor Copy link @hackerfactor hackerfactor commented Aug 18, 2021 taking a given image and then make its hash totally different, despite the image looking basically identical. can we do that? just curious if this even works for the one thing its suppost to do.. cropping the image seems to work since the algorithm (at least what we have now) is vulnerable to that Has anyone worked out the percentage of cropping needed to avoid a match using NeuralHash? With PhotoDNA, it's about 2% off the width or height. @KuromeSan Copy link @KuromeSan KuromeSan commented Aug 18, 2021 * edited Has anyone worked out the percentage of cropping needed to avoid a match using NeuralHash? With PhotoDNA, it's about 2% off the width or height. i tried cropping it seems to only change the hash by 1 or 2 bits. is that really enough ? @gmaxwell Copy link @gmaxwell gmaxwell commented Aug 18, 2021 that just means that the 30 images are then matched on Apple's end with a different system. Apple has stated that they are reviewed by a human at that point. From a legal perspective a human review prior to report is absolutely necessary to prevent the subsequent search by an agent by the government is absolutely required to prevent the search from being a fourth amendment violation. (see e.g. US v. Miller (6th Cir. 2020)) For PR reasons apple has claimed that the human review has protects people against governments secretly expanding the scope of the databases without apple's knowledge. Because possession of child porn images is a strict liability crime, apple could not perform a second pass matching with a comparison with the actual image. They could use another, different, fingerprint as a prefilter before human review-- but if that fingerprint isn't similarly constructed the tolerance to resizing will be lost and they might have well just used sha256 over the decoded pixels in the first step and completely escaped the attack described in this thread. @hackerfactor Copy link @hackerfactor hackerfactor commented Aug 18, 2021 * edited From a legal perspective a human review prior to report is absolutely necessary to prevent the subsequent search by an agent by the government is absolutely required to prevent the search from being a fourth amendment violation. (see e.g. US v. Miller (6th Cir. 2020)) When reporting to NCMEC's CyberTipline, they have a checkbox: have you reviewed it? If you say "no", then NCMEC's staff will review it. If you say "yes", then they may still review it, but it's not a guarantee. Also, NCMEC forwards reports to the appropriate ICAC, LEO, or other enforcement organization. The report includes a copy of the picture and the recipient enforcement group reviews it. @yeldarby Copy link @yeldarby yeldarby commented Aug 18, 2021 It seems like it would be much harder to create a collision that also passes a sanity test like running it through OpenAI's CLIP model and verifying that the image is indeed plausibly CSAM. I tested it out and CLIP identifies the generated image above as generated (in fact, of the top 10,000 words in the English language that has the closest match, followed by IR, computed, lcd, tile, and canvas): https://blog.roboflow.com/apples-csam-neuralhash-collision/ @gmaxwell Copy link @gmaxwell gmaxwell commented Aug 18, 2021 When reporting to NCMEC's CyberTipline, they have a checkbox: have you reviewed it? If you say "no", then NCMEC's staff will review it. There is currently no such checkbox: https://report.cybertip.org/ submit Perhaps they had one in the past prior to adverse rulings in Federal court. Again, see US v. Miller, if you perform a match based on an NCMEC database and send the image to NCMEC without reviewing, then the user's fourth amendment protection against warrantless searches is preserved, and NCMEC (as an agent of the government with special statutory authority) cannot lawfully inspect the image. The suppression of the user's fourth amendment rights is dependent on a prior review by the provider which is lawful under the color of their private commercial agreement. Apple's human review is a critical step in the suppression of the constitutional rights of their users, without it any subsequent review by NCMEC or law enforcement would be unlawful and the whole reporting process would be useless (except to build databases of kompromat and other such due-process-less results). 5 @KuromeSan Copy link @KuromeSan KuromeSan commented Aug 18, 2021 * edited this seems to be enough to drastically change any images hash: add borders, about 1000px larger than the main image, fill them with as much entropy as you can. Untitled hash: ff0dcf8b9371ebd28a4f5d2d windows_xp_bliss-wide hash: 9f3bce9b9d716bf399cf4f21 this process is completely lossless since i just added more pixels around the edges of the original image, and in the photo viewer, you can simply zoom in and see the original image anyway, >-< 18 22 @deftdawg Copy link @deftdawg deftdawg commented Aug 18, 2021 @gmaxwell Can't Apple just take consent from a user's previous agreement to the icloud TOS/EULA? Be interested to hear if that works in a court Again, see US v. Miller, if you perform a match based on an NCMEC database and send the image to NCMEC without reviewing, then the user's fourth amendment protection against warrantless searches is preserved, and NCMEC (as an agent of the government with special statutory authority) cannot lawfully inspect the image. The suppression of the user's fourth amendment rights is dependent on a prior review by the provider which is lawful under the color of their private commercial agreement. Apple's human review is a critical step in the suppression of the constitutional rights of their users, without it any subsequent review by NCMEC or law enforcement would be unlawful and the whole reporting process would be useless (except to build databases of kompromat and other such due-process-less results). @KuromeSan Copy link @KuromeSan KuromeSan commented Aug 18, 2021 @gmaxwell Can't Apple just take consent from a user's previous agreement to the icloud TOS/EULA? Be interested to hear if that works in a court [DEL:obviously it wont because apple plans to use this for more than just iCloud:DEL] @gmaxwell Copy link @gmaxwell gmaxwell commented Aug 18, 2021 * edited @deftdawg Yes, Apple's commercial relationship with the target of the search is what makes it lawful for Apple to search the user's otherwise private data. And if Apple staff searches the data and finds something the believe to be unlawful, when they pass it onto an agent of the government Apple's search can lawfully be "repeated" without a warrant. But if Apple didn't review the material, and just continued matching against some government provided database, then when they hand the content over to a government agent a warrant would be required to inspect the image. In any case, this is getting pretty far afield. The point I was making, which I don't think anyone has seriously disputed, is that ideally an attack would manage to create second-preimage-images which are visually similar to an arbitrarily selected image. A noise image like has been constructed so far is an interesting first step and could potentially cause problems, but it's not as powerful of an attack as an arbitrary benign-looking image which is a hash match against a specific fixed entirely different image. Edit: E.g. see this HN comment where someone is being distracted by the fact that the current POC example image is random noise: https:// news.ycombinator.com/item?id=28222041 I think it's an example for why the public discourse would be improved with second-preimage-images that looked like natural-ish images. @danny-wu Copy link @danny-wu danny-wu commented Aug 18, 2021 * edited @gmaxwell, getting back to the security issues of how this system can be used to attack victims. Given that the network here is neural network based in nature, it seems implicitly possible to also guide collisions towards an arbitrarily selected image. If there are millions of images in the CSAM database, it would be almost certain that there are close-ups of body genitalia. As CSAM covers everything up to 17 years and 364 days old, and the law treats it the same, then there would be images that are CSAM and illegal, but visually very similar (at least to a human inspecting a visual derivative) to images that are adult and legal. With this in mind, I propose a different and perhaps more sinister attack: 1. Attacker A goes on the dark-web and collects known-CSAM, particularly those that are close-ups of genitalia, and do not visibly appear to be 'child porn'; just normal porn. Attacker A gets this list of hashes, and sends it to attacker B. 2. Attacker B receives a list of 96-bit hashes, and then finds (or produces) legal amateur pornography of 18+ consenting adults that are also similar close-ups of genitalia. Attacker B never works or interacts with CSAM. 3. Attacker B disturbs the legal close-up images, until they match the neural hashes provided by Attacker A. 4. The disturbed legal images get sent to the victim's device via some means. 5. Apple's CSAM detection will then flag a neural hash match, and once the threshold is reached, Apple will be alerted. An Apple employee will then review the 'visual derivative', and they will see body genitalia. At this point, they will have a legal obligation to report and forward this to law enforcement. 6. Law enforcement, after seeing 30 matches of close-up pornography that perceptually hashes to CSAM, may raid the victim's place, despite the fact that no CSAM material was ever sent or interacted with by Attacker B and the victim. In this scenario: * Attacker B does not have to ever interact with CSAM in any way, shape, or form, however can generate images that will fool both Apple's systems, and the law enforcement / judicial system, into imprisoning the victim. * Should Attacker B be caught, as they never were in procession of any CSAM material in the first place, and has only taken the act of sending legal pornography that is CSAM-matched, it is debatable what criminal exposure (if any) they have. * However, the victim on the other hand, is under the full wraith of child pornography possessions charge. * This would be an example of a "low-risk, high-impact" attack, where you can get someone in jail for possessing "CSAM", without ever touching CASM yourself. 17 @hackerfactor Copy link @hackerfactor hackerfactor commented Aug 18, 2021 There is currently no such checkbox: https://report.cybertip.org/ submit Perhaps they had one in the past prior to adverse rulings in Federal court. Here's a screenshot from the cybertipline's current reporting form. (The checkbox has been there for years...) It's the checkbox that says "File Viewed By Company". ncmec @hackerfactor Copy link @hackerfactor hackerfactor commented Aug 18, 2021 * edited this seems to be enough to drastically change any images hash: add borders, about 1000px larger than the main image, fill them with as much entropy as you can. ... this process is completely lossless since i just added more pixels around the edges of the original image, and in the photo viewer, you can simply zoom in and see the original image anyway, >-< You're losing sight of what NeuralHash claims to do. It is NOT a content identification or sub-picture search. It is a "perceptual hash". Think of it this way: If you print out both photos, scaled to the same size, and mount them on a wall 20 feet away, would a human say they look the same? Your "add a thick border of noise" would be a resounding "no, they look different because one has a big noisy border." And lo and behold, the hashes are very different. A significant crop? "No, one has been cropped." Flip? Rotate? "Nope, they look different because one has been flipped and/or rotated." Try this: Take the dog photo, draw a tiny mustach on him, and check the hash. A minor alteration should result in little or no difference in the hash. The things to test: How much of a difference must be made to make the hash signifiantly different? (Lots of bits different.) Does the amount of change needed vary based on where in the picture it occurs? How much cropping off any each edge? Is it symmetrical or asymmetrical? Can we associate specific bits in the hash with a specific type of alteration? Here's a fun one that I'd like to see someone test: Start with the dog. Erase (not crop) the left side. Do half of the bits remain the same? Picture "AA" has hash "aa". Picture "BB" has hash "bb" If I create a side-by-side splice "AB", do I get the hash "ab"? If so, then we can easily force a hash collision. @MasterNeuron Copy link @MasterNeuron MasterNeuron commented Aug 18, 2021 * edited Can you verify that these two images collide? beagle360 collision Here's what I see from following your directions: $ python3 nnhash.py NeuralHash/model.onnx neuralhash_128x96_seed1.dat beagle360.png 59a34eabe31910abfb06f308 $ python3 nnhash.py NeuralHash/model.onnx neuralhash_128x96_seed1.dat collision.png 59a34eabe31910abfb06f308 Both white noice images are the same and I can't see any differences?! Somebody is seeing dog on the 1st photo... 4 @kjsman Copy link @kjsman kjsman commented Aug 18, 2021 * edited I generated another working collision: Target image lena 32da5083d4f9db9d45b4c397 Generated image download-60 32da5083c4f9db9d45b4c397 Lena: 32da5083d4f9db9d45b4c397 Dog: 32da5083c4f9db9d45b4c397 13 15 9 @yeldarby Copy link @yeldarby yeldarby commented Aug 18, 2021 * edited @kjsman I ran those images through CLIP as described here and the dog image is still identified by CLIP as generated (but it does see the dog as well as the 3rd most similar word). Top matching English words according to CLIP for each of those two colliding images (and their feature vector's cosine similarity with the image's): Lenna: britney 0.2710131274330054 hat 0.2695037749031152 caroline 0.267320863072384 hats 0.261636163397781 catherine 0.2614189858805191 judy 0.25487058977059807 claire 0.2535490374489525 heather 0.25337776325078754 sapphire 0.25308272433397716 blake 0.2522795140967653 Dog: generated 0.2695470207125587 regard 0.26522406265702764 dog 0.2647775811672407 shepherd 0.26429458470715683 vincent 0.26314263493135504 lycos 0.26132000000095534 marilyn 0.2608674788806449 lion 0.2595975315789243 face 0.25944818166185624 gnu 0.2579544381392359 1 @jackloomen Copy link @jackloomen jackloomen commented Aug 18, 2021 @kjsman impose a constraint that would get rid of the the high frequencies in the image. such as an l1 norm in the DCT domain. @gmaxwell Copy link @gmaxwell gmaxwell commented Aug 18, 2021 @kjsman Awesome! That's closer! Attacks only get better. @hackerfactor Thanks for the screenshot! it appears that particular form isn't something the public has access to (or at least I can't find it). It's not on the form I linked to. Regardless, the case law is now completely unambiguous. If the NCMEC is getting scanning results from companies who have not actually inspected the image and believe it to be child porn on the basis of their inspection, then the NCMEC's subsequent review is a search by an agent of the government and requires a warrant. It seems really foolish to have that checkbox there, a footgun to let child predators go free. (and as much as I disapprove of the warrantless searching of user's private files, regardless of whos doing it-- I find it really disappointing to find that once the invasion has happened they're not doing everything to make sure charges can actually be filed) @unrealwill Copy link @unrealwill unrealwill commented Aug 18, 2021 lena-output python3 nnhash.py model.onnx neuralhash_128x96_seed1.dat lena-output.png 59a34eabe31910abfb06f308 59a34eabe31910abfb06f308 (Dog Hash) 1 2 @Jigsy1 Copy link @Jigsy1 Jigsy1 commented Aug 18, 2021 You are aware that hashes aren't conventional hashes, right? It's just a series of comma separated numbers. @yeldarby Copy link @yeldarby yeldarby commented Aug 18, 2021 @unrealwill that one at least doesn't match generated higher than other word in the dictionary, but CLIP still picks it out quite highly (10th most relevant word in the dictionary): CLIP feature vector cosine similarity with that image & English dictionary words: lcd 0.2685186813767101 cover 0.2656788641452026 duration 0.2610448567100487 banner 0.2610146163956426 ebook 0.2607660721001148 pixels 0.2596127825121441 dvd 0.25817185406107845 poster 0.258116692755925 gif 0.25807298663266715 generated 0.25786197122020243 Is it possible to steer it away from CLIP's generated feature vector as you permute from the base? @gmaxwell Copy link @gmaxwell gmaxwell commented Aug 18, 2021 impose a constraint that would get rid of the the high frequencies in the image. such as an l1 norm in the DCT domain. Adding an L1 DCT penalty should result in much more natural images generally, not just getting rid of the HF noise. Just reiterating this good suggestion. @unrealwill Copy link @unrealwill unrealwill commented Aug 18, 2021 @yeldarby Is it possible to steer it away from CLIP's generated feature vector as you permute from the base? yes you can steer it away easily, just add some term to the loss function (But obviously you will need to compute CLIP every iteration so it will take more time to converge) The L1 DCT penalty is probably a good suggestion, but you can probably also use the discriminator of a GAN network to maintain naturalness. To give an idea of the time complexity : My above picture was generated using a laptop CPU only in less than 5 minutes using non-optimized code. Starting from a given image + basic gradient descent + clipping image to its [-1,1] domain at each iteration (not even lbfgs yet) Optimized code would probably run ~100x faster. So there is plenty of room to add quality. Sorry for the NSFW, but it was to show that you can produce a soft-porn image of any given hash, that a user can know it's perfectly legal and have legitimate reasons to store, but a manual reviewer will have more problem to know the age of the person involved. Put yourself in the shoes of the manual reviewer presented with a nude picture with a "matching CSAM " and ask yourself from the grainy picture that looks like a home-made scanned slide whether you can determine if the girl is underage or not and if you should report the picture or not. @Stereo101 Copy link @Stereo101 Stereo101 commented Aug 18, 2021 I'm convinced you can take almost any image and smoothly transition it to very close to any other image while retaining its original perceptual hash value. The properties of what a perceptual hash needs to do practically guarantee it. If you think of the space of all possible images of a given height / width, the perceptual hash should divide that space up into clustered continuous chunks where locality is determined by tiny edits. An obvious way to define the space is as one dimension for each color of each pixel with values from 0 to 255. The issue is that as we increase the number of dimensions, these clusters of identical hashes are more forced to stretch to fill that space. To get some intuition on this: consider as you increase dimensions, the "volume" of an n-dimensional sphere divided by the "volume" of the corresponding n-dimensional cube approaches 0. This is concerning because the sphere is the best possible packing you can get, in other words the ability of well packed objects to fill space gets worse as dimensions go up. Consider that even a tiny 360 by 360 image would represent a huge 388,800 dimensional space. Instead of getting nice clusters, we're getting a kind of interwoven series of sponges for each hash value which completely fills the space, which is admittedly hard to visualize. Not being able to tightly pack these hash values together defeats the main goals of the perceptual hash (or any perceptual hash?) to have similar images give similar values. I'm about a third of the way done with turning a picture of samwise into a bowl of frootloops with the identical perceptual hash. @mcdallas Copy link @mcdallas mcdallas commented Aug 18, 2021 @unrealwill I used the code in your gist and swapped the actual model and hashing function but got nowhere near the performance described. Any chance you could share the updated code? @jackloomen Copy link @jackloomen jackloomen commented Aug 18, 2021 What purpose does the dot product with the seed matrix serve? It's a linear operation that is differentiated right through. @yeldarby Copy link @yeldarby yeldarby commented Aug 18, 2021 @yeldarby Is it possible to steer it away from CLIP's generated feature vector as you permute from the base? yes you can steer it away easily, just add some term to the loss function (But obviously you will need to compute CLIP every iteration so it will take more time to converge) @unrealwill can you maintain the same NeuralHash while doing so though? Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment Assignees No one assigned Labels None yet Projects None yet Milestone No milestone Linked pull requests Successfully merging a pull request may close this issue. None yet 33 participants @dxoigmn @prusnak @lericson @cmsj @weskerfoot @indiv0 @gmaxwell @yeldarby @WriteCodeEveryday @fuomag9 @CRTified @jeifour @deftdawg @deavmi @MasterNeuron @judge2020 @tmechen @unrealwill @hackerfactor @kjsman and others * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.