[HN Gopher] Let's talk about AI and end-to-end encryption
___________________________________________________________________
Let's talk about AI and end-to-end encryption
Author : chmaynard
Score : 248 points
Date : 2025-01-17 05:50 UTC (1 days ago)
(HTM) web link (blog.cryptographyengineering.com)
(TXT) w3m dump (blog.cryptographyengineering.com)
| crackalamoo wrote:
| See also CrypTen, Meta's library for privacy preserving machine
| learning: https://github.com/facebookresearch/CrypTen. This isn't
| fully homomorphic encryption, but it is multi-party computation
| (MPC), which hides the inputs from the company owning the model.
|
| But while not revealing user input, it would still reveal the
| outputs of the model to the company. And yeah, as the article
| mentions, unfortunately this kind of thing (MPC or fully-
| homomorphic encryption) probably won't be feasible for the most
| powerful ML models.
| EGreg wrote:
| I heard that homomorphic encryption can actually preserve all the
| operations in neural networks, since they are differentiable. Is
| this true? What is the slowdown in practice?
| crackalamoo wrote:
| This is true in principle, yes. In practice, the way this
| usually works is by converting inputs to bits and bytes, and
| then computing the result as a digital circuit (AND, OR, XOR).
|
| Doing this encrypted is very slow: without hardware
| acceleration or special tricks, running the circuit is 1
| million times slower than unencrypted, or about 1ms for a
| single gate. (https://www.jeremykun.com/2024/05/04/fhe-
| overview/)
|
| When you think about all the individual logic gates involved in
| just a matrix multiplication, and scale it up to a diffusion
| model or large transformer, it gets infeasible very quickly.
| j2kun wrote:
| There are FHE schemes that do better than binary gates (cf.
| CKKS) but they have other problems in that they require
| polynomial approximations for all the activation functions.
| Still they are much better than the binary-FHE schemes for
| stuff like neural networks, and most hardware accelerators in
| the pipeline right now are targeting CKKS and similar for
| this reason.
|
| For some numbers, a ResNet-20 inference can be done in CKKS
| in like 5 minutes on CPU. With custom changes to the
| architecture you can get less than one minute, and in my view
| HW acceleration will improve that by another factor of 10-100
| at least, so I'd expect 1s inference of these (still small)
| networks within the next year or two.
|
| LLMs, however, are still going to be unreasonably slow for a
| long time.
| jFriedensreich wrote:
| I think this has also a silver lining. The E2E encryption
| movement especially for messenger apps was largely also used to
| silently lock users out of their own data and effectively prevent
| user agency to use their own data to move apps, write automations
| or archive, this is not just true for whatsapp (the data export
| feature does not fully work since its launch and was just made to
| appease some EU law that did not properly check if the button
| works until the end.) Also signal does not have a way to do this.
| Maybe with ai coming into the game companies finally decide to
| provide access to data, I just hope it's in a transparent way
| with user opt in and user control.
| do_not_redeem wrote:
| 1. Is data encrypted in transit?
|
| 2. Can the user access their data at rest?
|
| Those two things are entirely orthogonal.
|
| I don't think you can extrapolate a trend from a few apps
| having bugs in their export code. Google Takeout is also
| notoriously buggy and they don't use E2E encryption. A more
| likely explanation is companies of all kinds don't care that
| much about export functionality, due to the incentives
| involved.
| jFriedensreich wrote:
| you CAN extrapolate from nearly all e2e encrypted apps not
| giving a way to use the data. And there is a big difference
| between buggy google export features or facebook actively
| making export unusable to lock in users.
| walrus01 wrote:
| Signal does not have a way to manually export your private keys
| and chat history, but the process of "moving" your signal
| account to a new phone is quite straightforward. You put both
| devices on the same wifi/LAN layer 2 broadcast segment, start
| the transfer process in the app, input the verification codes
| displayed on the screen from both devices, and it sends
| everything over. This moves the private key in a way that does
| not result in all of your contacts receiving the scary "this
| person's key has changed" message.
| jFriedensreich wrote:
| "Moving your account" is not what i talk about, besides not
| being possible between android and ios with history. User
| agency means a user is allowed to access his data and do what
| they want with it how they want, realtime and with whatever
| code they want to write to do so.
| bobbiechen wrote:
| There is always going to be a gap between local-first processing
| and what can be achieved in a full-sized datacenter/cloud. That
| leads to the risks mentioned in the article.
|
| I wrote about Apple's Private Cloud Compute last year; for the
| foreseeable future, I still think server-side Confidential
| Computing is the most practical way to do processing without huge
| privacy risks: https://www.anjuna.io/blog/apple-is-using-secure-
| enclaves-to...
| tonygiorgio wrote:
| > Although PCC is currently unique to Apple, we can hope that
| other privacy-focused services will soon crib the idea.
|
| IMHO, Apple's PCC is a step in the right direction in terms of
| general AI privacy nightmares where they are at today. It's not a
| perfect system, since it's not fully transparent and auditable,
| and I do not like their new opt-out photo scanning feature
| running on PCC, but there really is a lot to be inspired by it.
|
| My startup is going down this path ourselves, building on top of
| AWS Nitro and Nvidia Confidential Compute to provide end to end
| encryption from the AI user to the model running on the enclave
| side of an H100. It's not very widely known that you can do this
| with H100s but I really want to see this more in the next few
| years.
| mnahkies wrote:
| I didn't actually realize that AWS supported this, I thought
| Azure was the only one offering it
| (https://azure.microsoft.com/en-us/blog/azure-confidential-
| co...)
|
| Are you speaking of this functionality?
| https://developer.nvidia.com/blog/confidential-computing-on-...
| (and am I just failing to find the relevant AWS docs?)
| tonygiorgio wrote:
| Yes, you're correct on both, though I think Google Cloud
| recently started supporting it as well. AWS will likely have
| GPU enclave support with Trainium 2 soon (AFAIK, that feature
| is not publicly offered yet but could be wrong).
|
| We work with Edgeless Systems who manages the GPU enclave on
| Azure that we speak to from our AWS Nitro instance. While not
| ideal, the power of enclaves and the attestation verification
| process, we at least know that we're not leaking privacy by
| going with a third party GPU enclave provider.
| calum-bird wrote:
| Do you have any resources you can share on managing the
| enclave's public key? We found documentation to be sparse.
| blueblimp wrote:
| And the most important thing about PCC in my opinion is not the
| technical aspect (though that's nice) but that Apple views user
| privacy as something good to be maximized, differing from the
| view championed by OpenAI and Anthropic (and also adopted by
| Google and virtually every other major LLM provider by this
| point) that user interactions must be surveilled for "safety"
| purposes. The lack of privacy isn't due to a technical
| limitation--it's intended, and they often brag about it.
| natch wrote:
| Something good to be maximized within the constraints of the
| systems they have to work within. But at some point with
| enough compromises it becomes maximizing the perception of
| privacy, not the reality. Promoting these academic techniques
| may just be perception management on the part of Apple, if
| the keys are not controlled solely by the user.
| flossposse wrote:
| If Apple really wanted to maximize privacy, they wouldn't be
| constantly collecting so much information in the first place
| (capture the network traffic from an apple device sometime -
| it's crazy). User interactions on Apple devices definitely
| seem to be surveilled for "safety" purposes.
|
| From my perspective, Apple's behavior indicates that what
| they want to maximize is their own control, and their
| position as the gatekeeper others must pay in order to get
| access to you.
| bee_rider wrote:
| The author helpfully emphasized the interesting question at the
| end
|
| > This future worries me because it doesn't really matter what
| technical choices we make around privacy. It does not matter if
| your model is running locally, or if it uses trusted cloud
| hardware -- once a sufficiently-powerful general-purpose agent
| has been deployed on your phone, the only question that remains
| is who is given access to talk to it. Will it be only you? Or
| will we prioritize the government's interest in monitoring its
| citizens over various fuddy-duddy notions of individual privacy.
|
| I do think there are interesting policy questions there. I mean
| it could hypothetically be mandated that the government must be
| given access to the agent (in the sense that we and these
| companies exist in jurisdictions that can pass arbitrary laws;
| let's skip the boring and locale specific discussion of whether
| you think your local government would pass such a law).
|
| But, on a technical level--it seems like it ought to be possible
| to run an agent locally, on a system with full disk encryption,
| and not allow anyone who doesn't have access to the system to
| talk with it, right? So on a technical level I don't see how this
| is any different from where we were previously. I mean you could
| also run a bunch of regex's from the 80's to find whether or not
| somebody has, whatever, communist pamphlets on their computers.
|
| There's always been a question of whether the government should
| be able to demand access to your computer. I guess it is good to
| keep in mind that if they are demanding access to an AI agent
| that ran on your computer, they are basically asking for a lossy
| record of your entire hard drive.
| _boffin_ wrote:
| Unreasonable search?
| bee_rider wrote:
| > (in the sense that we and these companies exist in
| jurisdictions that can pass arbitrary laws; let's skip the
| boring and locale specific discussion of whether you think
| your local government would pass such a law)
|
| Anyway the idea of what's a reasonable search in the US has
| been whittled away to almost nothing, right? "The dog smelled
| weed on your hard drive." - A cop, probably.
| chgs wrote:
| Boring locale specific discussion.
| cryptonector wrote:
| > The author helpfully emphasized the interesting question at
| the end
|
| We're already there. AI or not doesn't affect the fact that
| smartphones gather, store, and transmit a great deal of
| information about their users and their users' actions and
| interests.
| walrus01 wrote:
| It's a _good thing_ that encrypted data at rest on your local
| device is inaccessible to cloud based "AI" tools. The problem is
| that your average person will blithely click
| "yes/accept/proceed/continue/I consent" on pop up dialogs in a
| GUI and agree to just about any Terms of Service, including
| decrypting your data before it's sent to some "cloud" based
| service.
|
| I see "AI" tools being used even more in the future to
| permanently tie people to monthly recurring billing services for
| things like icloud, microsoft's personal grade of office365,
| google workspace, etc. You'll pay $15 a month forever, and the
| amount of your data and dependency on the cloud based provider
| will mean that you have no viable path to ever stop paying it
| without significant disruption to your life.
| Animats wrote:
| > Who does your AI agent actually work for?
|
| Yes. I made that point a few weeks ago. The legal concept of
| principal and agent applies.
|
| Running all content through an AI in the cloud to check for
| crimethink[1] is becoming a reality. Currently proposed:
|
| - "Child Sexual Abuse Material", which is a growing category that
| now includes AI-generated images in the US and may soon extend to
| Japanese animation.
|
| - Threats against important individuals. This may be extended to
| include what used to be considered political speech in the US.
|
| - Threats against the government. Already illegal in many
| countries. Bear in mind that Trump likes to accuse people of
| "treason" for things other than making war against the United
| States.
|
| - "Grooming" of minors, which is vague enough to cover most
| interactions.
|
| - Discussing drugs, sex, guns, gay activity, etc. Variously
| prohibited in some countries.
|
| - Organizing protests or labor unions. Prohibited in China and
| already searched for.
|
| Note that talking around the issue or jargon won't evade
| censorship. LLMs can deal with that. Run some ebonics or
| leetspeak through an LLM and ask it to translate it to standard
| English. Translation will succeed. The LLM has probably seen more
| of that dialect than most people.
|
| _" If you want a vision of the future, imagine a boot stepping
| on a face, forever"_ - Orwell
|
| [1] https://www.orwell.org/dictionary/
| iugtmkbdfil834 wrote:
| A cynic in me is amused at the yet unknown corporation being
| placed under investigation due to a trigger phrase in one of
| the meetings transcribed incorrectly.
|
| Your point is worth reiterating.
| Terr_ wrote:
| Or poisoned-data that sets up a trap, so that a system will
| later confabulate false innocence or guilt when certain
| topics or targets come up.
| crooked-v wrote:
| "Grooming" in particular is the angle that Republicans want to
| use to illegalize any kind of gender-nonconforming behavior, up
| to and including desired states like "women wearing pants is
| crossdressing, and doing so around children is a felony".
| nashashmi wrote:
| The most depressing realization in all of this is that the vast
| treasure trove of data that we used to have in the cloud thinking
| it was not scannable even for criminal activity has now become a
| vector where we shall have thought police coming down upon us for
| simple ideas of dissent.
| AlexandrB wrote:
| A lot of people tried to sound the alarm. It's not "the cloud",
| it's "other people's computers". And given that other people
| own these machines, their interests - whether commercial or
| ideological - will always come first.
| tokioyoyo wrote:
| To be fair, most people understand that risk. It's just it is
| very convenient in a lot of scenarios and some businesses
| might have not even started without it. Privacy is not that
| big of a concern for a big chunk of people. And they're
| basically voting with their wallets.
| like_any_other wrote:
| "Voting with their wallets", where all the choices are
| picked by entities hostile to consumer privacy and
| autonomy, _and then they mislead you about those choices_.
|
| People who bought LG (and now most other, now "smart") TVs
| did not in any meaningful way "vote" to be spied on and
| support DRM - simply all the TVs in a store would spy and
| show ads, and not disclose any of it at time of sale.
| tokioyoyo wrote:
| Well, it shows how it's not the biggest issue for an
| average customer, as it, in theory, can make the TV
| cheaper. If people cared about absence of ads that much,
| all social media would not be ad-ridden since people
| would stop using it.
| Terr_ wrote:
| Plus the machines they don't technically own--like
| Microsoft's attempts to force online accounts, bloatware,
| telemetry, etc.
| vaylian wrote:
| Trump will take office on Monday. If he chooses to declare some
| progressive idea as anti-american, a lot of people, who
| previously said nothing illegal, could face hostility from
| "patriots".
|
| I hope this doesn't happen. But I wouldn't be surprised if it
| did. Old data can become toxic waste.
| blueblimp wrote:
| > Yet this approach is obviously much better than what's being
| done at companies like OpenAI, where the data is processed by
| servers that employees (presumably) can log into and access.
|
| No need for presumption here: OpenAI is quite transparent about
| the fact that they retain data for 30 days and have employees and
| third-party contractors look at it.
|
| https://platform.openai.com/docs/models/how-we-use-your-data
|
| > To help identify abuse, API data may be retained for up to 30
| days, after which it will be deleted (unless otherwise required
| by law).
|
| https://openai.com/enterprise-privacy/
|
| > Our access to API business data stored on our systems is
| limited to (1) authorized employees that require access for
| engineering support, investigating potential platform abuse, and
| legal compliance and (2) specialized third-party contractors who
| are bound by confidentiality and security obligations, solely to
| review for abuse and misuse.
| chefandy wrote:
| I have to say -- I'm kind of amazed that anyone would expect
| privacy out of chat bot companies and products. You're
| literally having a "conversation" with the servers of companies
| that built their entire product line using other people's
| professional and personal output whether they approved, or even
| knew about it or not. Less a "it's better to ask for
| forgiveness than permission" sort of thing than a "we'd rather
| just not ask and be pretty cagey about it if they ask, and then
| if they prove it, tell them they tacitly agreed to it by not
| hiding it from us even though they had no way to know we were
| looking at it" sort of thing. Frankly I'm astonished that open
| ai, specifically, promises as much as they do in their privacy
| policy. Based on their alleged bait-and-switch tactics quietly
| swapping out models or reducing compute for paying customers
| after the initial "gee wiz look at that" press cycle, I can't
| imagine those privacy policies will have much longevity when
| the company gets a more stable footing... and whooops looks
| like they figured out how to extract the training data from the
| models! And it's different data since we extracted it from the
| model so the old privacy policy doesn't apply! Haha sorry,
| that's business and we're building a techno utopian society
| here, so you should feel honored to be included! You think
| Altman wouldn't sell that in a _heartbeat_ to try and fund some
| big moonshot product if they get clobbered in the marketplace?
| Never mind the sketchy girlfriend-in-an-app-class chatbots.
|
| Don't get me wrong -- I absolutely think the privacy SHOULD be
| there, but I'm just shocked that anyone would assume it was.
| Maybe I'm being overly cynical? These days when I think I might
| be, in the end, it seems I wasn't being cynical enough.
| notfed wrote:
| Cynically, I think most people know this in this kind of
| situation, but like clockwork media sources will suddenly
| dramatize things for clicks, money, lawsuits, or politics,
| and people will nod their heads not because they agree with
| the accusations, but because they have preconceived bias
| against the defendant company.
| lowbatt wrote:
| Maybe a little off topic, but is there a way for a distributed
| app to connect to one of the LLM companies (OpenAI, etc.) without
| the unencrypted data hitting an in-between proxy server?
|
| An app I'm building uses LLMs to process messages. I don't want
| the unencrypted message to hit my server - and ideally I wouldn't
| have the ability to decrypt it. But I can't communicate directly
| from client -> LLM Service without leaking the API key.
| simonw wrote:
| "But I can't communicate directly from client -> LLM Service
| without leaking the API key."
|
| There is a way you can do that right now: the OpenAI WebRTC API
| introduced the idea of an "ephemeral key":
| https://platform.openai.com/docs/guides/realtime-webrtc
|
| This provides a way for your server to create a limited-time
| API key for a user which their browser can then use to talk to
| OpenAI's API directly without proxying through you.
|
| I love this idea, but I want it for way more than just the
| WebRTC API, and I'd like it for other API providers too.
|
| My ideal version would be a way to create an ephemeral API key
| that's only allowed to talk to a specific model with a specific
| pre-baked system prompt (and maybe tool configuration and
| suchlike) and that only works for a limited time and has a
| limited token budget.
| lowbatt wrote:
| interesting, will check that out. thanks!
| michaelmrose wrote:
| Will such processing be cheap enough to be done by a box that
| plugs into a customers router to handle such? Would they buy
| them? Notably not just for this use case but others
| whyage wrote:
| Check out https://www.opaque.co/
| ozgune wrote:
| > Apple even says it will publish its software images (though
| unfortunately not the source code) so that security researchers
| can check them over for bugs.
|
| I think Apple recently changed their stance on this. Now, they
| say that "source code for certain security-critical PCC
| components are available under a limited-use license." Of course,
| would have loved it if the whole thing was open source. ;)
|
| https://github.com/apple/security-pcc/
|
| > The goal of this system is to make it hard for both attackers
| and Apple employees to exfiltrate data from these devices.
|
| I think Apple is claiming more than that. They are saying 1/ they
| don't keep any user data (data only gets processed during
| inference), 2/ no privileged runtime access, so their support
| engineers can't see user data, and 3/ they make binaries and
| parts of the source code available to security researchers to
| validate 1/ and 2/.
|
| You can find Apple PCC's five requirements here:
| https://security.apple.com/documentation/private-cloud-compu...
|
| Note: Not affiliated with Apple. We read through the PCC security
| guide to see what an equivalent solution would look like in open
| source. If anyone is interested in this topic, please hit me up
| at ozgun @ ubicloud . com.
| saagarjha wrote:
| Some of the core elements of the boot process are not source
| available, unfortunately.
| natch wrote:
| From Apple's document on Advanced Data Protection:
|
| >With Advanced Data Protection enabled, Apple doesn't have the
| encryption keys needed to help you recover your end-to-end
| encrypted data.
|
| Apple doesn't have the keys. Somebody else might. Somebody other
| than you. Also, I think they meant to say decryption keys,
| although they're probably just dumbing down terminology for the
| masses.
|
| >If you ever lose access to your account, you'll need to use one
| of your account recovery methods
|
| "You'll need to use." Not "there is no way except to use."
|
| >Note: Your account recovery methods are never shared with or
| known to Apple.
|
| "shared with or known to Apple." Not "shared with or known to
| anyone else."
|
| The encryption is there, I believe that. I just don't know how
| many copies of the keys there are. If the only key is with me, it
| would be super easy for Apple to just say that. I believe that
| they have said that in the past, but the wording has now changed
| to this hyper-specific "Apple does not have the key" stuff.
| musicale wrote:
| As you suggest, the wording should be clarified to say that the
| key is never copied, is only stored on your device, is not
| accessible to others, etc.
| natch wrote:
| Maybe they are unable to make that clarification, if it would
| be false.
| systoll wrote:
| It does say
|
| > It's protected with the new key which is controlled solely
| by the user's trusted devices
|
| I think main thing they're avoiding is an explicit guarantee
| that the key cannot be retrieved from your phone by a third
| party.
| rglover wrote:
| > We are about to face many hard questions about these systems,
| including some difficult questions about whether they will
| actually be working for us at all.
|
| _And how_. I 'd lean towards no. Where we're headed feels like
| XKEYSCORE on steroids. I'd love to take the positive, optimistic
| bent on this, but when you look at where we've been combined with
| the behavior of the people in charge of these systems (to be
| clear, not the researchers or engineers, but c-suite), hope of a
| neutral, privacy-first future seems limited.
| ActorNightly wrote:
| Given how politics and companies evolved, I actually trust
| those people in charge of XKEYSCORE systems more than ever.
| They may wear suits, but those people usually come from some
| military background, and have a sense of duty towards defending
| US, from threats both foreign and domestic, and historically
| have not really abused their powers no matter what the
| administration is. XKEYSCORE for example, wasn't really about
| hacking people, it was just about collecting mass metadata and
| building profiles, well within the legal system, and the blame
| should be on the companies that didn't provide privacy tools,
| because any big government could have build the same system.
|
| Meanwhile, the anti anti-establishment Republican Party since
| 2016 who cried about big tech turned out to be the biggest pro-
| establishment fans, giving Elmo an office in a white house and
| Zucc bending a knee to avoid prosecution.
|
| With these new systems, Id rather have smart people who only
| work in US defensive forces because of a sense of duty
| (considering they could get paid much more in the private
| sector) in charge.
| schmidtleonard wrote:
| > well within the legal system
|
| It's not a search if we don't find anything, and it's not a
| seizure if we charge the money with the crime. These are
| court approved arguments, so they must be correct
| interpretations.
|
| Point is: modern bureaucrats have proven that they are
| absolutely willing to abuse power, even in the best of times
| when there is no real domestic political strife.
| saagarjha wrote:
| > historically have not really abused their powers
|
| How would you know?
| HeatrayEnjoyer wrote:
| I absolutely do not trust it, but AFAIK the military
| doesn't feed much intelligence to law enforcement on US
| soil. (We'll see if that's still the case in the near
| future.)
| rainonmoon wrote:
| We _do_ know - that they demonstrably _have_ abused their
| powers. I didn 't realise it was possible to know about
| XKEYSCORE with no context or understanding of the Snowden
| leaks but GP seems to have missed that the "suits" "in
| charge of XKEYSCORE", the NSA, have repeatedly illegally
| wiretapped American citizens, to say nothing of the FISA
| abuses, Five Eyes, etc. Regardless of how you feel about
| the three-letter agencies' impacts on the rest of the
| world, the thought that anyone on Hacker News would
| consider these programs defensible is shocking.
| rglover wrote:
| Unfortunately, there are far too many examples of those very
| people abusing these tools. They shot the "sense of honor and
| duty" argument point blank just for allowing these things to
| exist in the first place.
|
| If what you say is true, there would have been more than one
| honorable person to step up and say "hey, wait a minute." In
| the case of XKEYSCORE, there was precisely one, and he's
| basically been marooned in Russia for over a decade (and
| funny enough, XKEYSCORE still exists and is likely still
| utilized in the exact same way [1]).
|
| Never underestimate the effect the threat of character
| destruction--and by extension, loss of income--will have on
| even the most honorable person's psyche. In situations
| involving matters like these, it's always far more likely
| that the "pressure" will be ratcheted up until the compliance
| (read: keep your mouth shut) rate is 100%.
|
| [1] https://documents.pclob.gov/prod/Documents/OversightRepor
| t/e...
| toss1 wrote:
| Good thoughts but as you point out about Elmo & Zucc, there
| is no way it stays with just the responsible people. It will
| also not be limited to protest. Just look at what Florida,
| Texas, and other states are doing about women's healthcare -
| any general agent worth its salt and with a bit of data will
| know about any woman's periods, pregnancies, miscarriages,
| and travel - which is being criminalized ....
| rapjr9 wrote:
| My guess is that the main purpose of agents will be to train
| the AI on your data. Companies have run out of data on the
| internet for training AI's, so they'll use agents as an excuse
| to get access to your personal real-time data. This has always
| been the business model, you are the product.
| jrm4 wrote:
| "The goal of encryption is to ensure that only two parties, the
| receiver and sender, are aware of the contents of your data.
|
| Thus, AI training on your data breaks this, because it's another
| party.
|
| You now don't have encryption."
|
| Thanks for coming to my blah blah blah
| dialup_sounds wrote:
| The article has nothing to do with model training.
| peppertree wrote:
| Is embeddings enough to preserve privacy? If I run the
| encoder/decoder on device and only communicate with server in
| embeddings?
| nostradumbasp wrote:
| For a short time maybe, but that kind of activity might look
| like something interesting enough for someone to target you and
| remove all privacy from your entire existence.
| hallh wrote:
| No, the original text can largely be recovered from
| embeddings[0] if you know which embedding model was used.
|
| [0] https://arxiv.org/abs/2310.06816
| klik99 wrote:
| > You might even convince yourself that these questions are
| "privacy preserving," since no human police officer would ever
| rummage through your papers, and law enforcement would only learn
| the answer if you were (probably) doing something illegal.
|
| Something I've started to see happen but never mentioned is the
| effect automated detection has on systems: As detection becomes
| more automated (previously authored algorithms, now with large AI
| models), there's less cash available for individual case workers,
| and more trust at the managerial level on automatic detection.
| This leads to false positives turning into major frustrations
| since it's hard to get in touch with a person to resolve the
| issue. When dealing with businesses it's frustrating, but as
| these get more used in law enforcement, this could be life
| ruining.
|
| For instance - I got flagged as illegal reviews on Amazon years
| ago and spent months trying to make my case to a human. Every
| year or so I try to raise the issue again to leave reviews, but
| it gets nowhere. Imagine this happening for a serious criminal
| issue, with the years long back log on some courts, this could
| ruin someones life.
|
| More automatic detection can work (and honestly, it's inevitable)
| but it's got to acknowledge that false positives will happen and
| allocate enough people to resolve those issues. As it stands
| right now, these detection systems get built and immediately
| human case workers get laid off, there's this assumption that
| detection systems REPLACE humans, but it should be that they
| augment and focus human case workers so you can do more with less
| - the human aspect needs to be included in the budgeting.
|
| But the incentives aren't there, and the people making the
| decisions aren't the ones working the actual cases so they aren't
| confronted with the problem. For them, the question is why save
| $1m when you could save $2m? With large AI models making it
| easier and more effective to build automated detection I expect
| this problem to get significantly worse over the next years.
| smallmancontrov wrote:
| The UK Post Office scandal is bone-chilling.
|
| Update this to a world where every corner of your life is
| controlled by a platform monopoly that doesn't even provide the
| most bare-bones customer service and yeah, this is going to get
| a lot worse before it gets better.
| Vampiero wrote:
| And that's the early game.
|
| Imagine when AI will be monitoring all internet traffic and
| arresting people for thoughtcrime.
|
| What wasn't feasible to do before is now quite in reach and
| the consequences are dire.
|
| Though of course it won't happen overnight. First they will
| let AI encroach every available space (backed by enthusiastic
| techbros). THEN, once it's established, boom. Authoritarian
| police state dystopia times 1000.
|
| And it's not like they need evidence to bin you. They just
| need inference. People who share your psychological profile
| will act and speak and behave in a similar way to you, so you
| can be put in the same category. When enough people in that
| category are tagged as criminals, you will be too.
|
| All because you couldn't be arsed to write some boilerplate
| shakna wrote:
| It's already arresting the wrong people [0].
|
| [0] https://www.theregister.com/2023/08/08/facial_recogniti
| on_de...
| HeatrayEnjoyer wrote:
| We need strong and comprehensive regulations. Some places
| have enacted partial solutions but none anywhere near as
| complete as needed. EU has GDPR and some early AI laws, India
| has the IT Act that requires companies to provide direct end-
| user support.
| BlueTemplar wrote:
| That's why there are transparency laws that indirectly forbid
| the use of black box decision systems like these for anything
| government-related.
| CoffeeOnWrite wrote:
| Also AI for accountability laundering. It gives plausible
| deniability. It's a sociopathic manager's dream.
| rainonmoon wrote:
| This. They're digital sniffer dogs, a pretext to lend
| credibility to vibes-based policing.
| drysine wrote:
| >Imagine this happening for a serious criminal issue, with the
| years long back log on some courts, this could ruin someones
| life.
|
| It can be much scarier.
|
| There was a case in Russia when a scientist was accused in a
| murder that happened 20 years ago based on 70% face recognition
| match and fake identification as an accomplice by a criminal.
| [0] He spent 10 months in jail during "investigation" despite
| being incredibly lucky to have an alibi -- archival records of
| the institute where he worked, proving he was in an expedition
| far away from Moscow at that time. He was eventually freed but
| I'm afraid that police investigators that used very weak face
| recognition match as a way to improve their work performance
| stats are still working in the police.
|
| [0] https://lenta.ru/articles/2024/04/03/scientist/
| ithkuil wrote:
| And probably there are other people in jail convicted using
| the same method that just were unlucky enough to not have a
| bulletproof alibi?
| drysine wrote:
| I don't know, but it seems quite likely, unfortunately.
| There were quite a few other cases when fake evidence was
| planted by police.
|
| It's not the only problem with technology -- it's claimed
| that there has been over hundred cases of false DNA matches
| not caused by malice or processing errors.[0] In theory,
| DNA match must not be considered by courts as 100%
| accurate, but in fact it is.
|
| On the other hand, there were cases when human rights
| advocates or journalists were claiming that innocent people
| were jailed but that turned out to be false, like people
| getting caught on camera doing the same kind of crime again
| after they served their sentence.
|
| [0] https://www.kommersant.ru/doc/5825384
| asddubs wrote:
| same with stenographs. We keep inventing new methods for
| the police to make up evidence.
| drysine wrote:
| I don't understand, could you elaborate on that?
| asddubs wrote:
| I apologize, I got the terms mixed up, I meant a
| polygraph
| drysine wrote:
| Yep, I got the impression that courts consider polygraph
| only when the results implicate the accused. Good thing
| that by law they cannot force you to get questioned with
| polygraph attached.
| polygon87 wrote:
| I know it's the wrong way to think but things like this make
| me glad about a digital footprint... good chance I'm liking a
| TikTok comment or reading an HN thread at the same time as
| any crime, just statistically.
| asddubs wrote:
| that's not going to get you off the hook. anything that
| could be faked via account sharing is going to be discarded
| (not to mention that tiktok and similar platforms will not
| collaborate with you to build an alibi by giving access to
| this data, only the police to build a case)
| rainonmoon wrote:
| Grave consequences are not a rarity. Automated decision
| making in immigration and housing classify people with zero
| recourse or transparency, locking them out of a place to live
| (and in the case of Australia, locking them up in offshore
| detention for years).
| t0bia_s wrote:
| It could also be used to eliminate political opponents,
| minorities, etc. Persecution with collective guilt bases on
| digital footprints wasn't easier ever.
| divan wrote:
| There was a good thread on this phenomenon (called
| "accountability sinks") [1]
|
| [1] https://news.ycombinator.com/item?id=41891694
| deathanatos wrote:
| TFA makes some rather basic errors.
|
| First,
|
| > _Prior to 2011, most cloud-connected devices simply uploaded
| their data in plaintext._
|
| > _Around 2011 our approach to data storage began to evolve.
| [...] began to roll out default end-to-end encryption [...] This
| technology changed the way that keys are managed, to ensure that
| servers would never see the plaintext content of your messages._
|
| "changed the way that keys are managed" is at a confused
| contradiction with "uploaded their data in plaintext". If you're
| going from TLS - E2EE, then yeah, "changed the way keys are
| managed" miiight make sense, though that's not how I'd phrase it.
| Then later,
|
| > _On the one hand they can (1) send plaintext off to a server,
| in the process resurrecting many of the earlier vulnerabilities
| that end-to-end encryption sought to close. Or else (2) they can
| limit their processing to whatever can be performed on the device
| itself._
|
| We're still confusing "transmit plaintext" with plaintext being
| available to the server; the clear option of "use TLS" is
| omitted. It doesn't really undermine the argument -- the server
| would still have access to the data, and could thus maliciously
| train AI on it -- but it is surprising for a "cryptographer".
|
| > _For example, imagine that Apple keeps its promise to deliver
| messages securely, but then your (Apple) phone goes ahead and
| uploads (the plaintext) message content to a different set of
| servers where Apple really can decrypt it. Apple is absolutely
| using end-to-end encryption in the dullest technical sense... yet
| is the statement above really accurate? Is Apple keeping its
| broader promise that it "can't decrypt the data"?_
|
| No, no reasonable person would believe that (though I am sure
| that if the scenario ever came to be, Apple, or whoever, would
| likely argue "yes") since it would utterly scuttle the term
| "E2EE". If you say "Our product supports X", and then have to
| caveat away 100% of what makes X X, then it's just grift, plain
| and simple. (Now, whether grift sees regulatory action ... well.)
|
| > _Now imagine that some other member of the group -- not you,
| but one of your idiot friends -- decides to turn on some service
| that uploads (your) received plaintext messages to WhatsApp._
|
| > _In general, what we're asking here is a question about
| informed consent._
|
| I would sort of agree, but corporations will expose the consent
| here to the "friend", and then argue that because the friend
| consented to _your_ data being uploaded, it is fine. An argument
| for privacy regulations.
|
| (I don't think you have to go through all this ... work. Just
| upload the user's data. They'll complain, for a bit, but the
| market has already consolidated into at least an ologopoly, users
| have shown that, for the most part, they're going to keep using
| the product rather than leave, or else I'll be ending this
| comment with a free "2025 will be the Year of the Linux Desktop".
| What's gonna happen, _regulation_ to ensure a free market remains
| free1? Please. Cf. MS Recall, currently in the "complain" phase,
| but give it time, and we'll reach the "we heard your concerns,
| and we value your input and take your feedback with the utmost
| respect _ram it down their throats_ " stage.)
|
| (1free as in "dictated by the laws of supply & demand", not
| laissez-faire which is where the US will be headed for the next
| 4.)
|
| (and ... 2011? I'd've said 2013 is when we found out the 4A meant
| way less than we thought it did, leading to the rise in massive
| adoption of TLS. Less so E2EE.)
| lifeisstillgood wrote:
| So if I understand it
|
| 1. E2E encryption does work
|
| 2. But phones can send plaintext back to the cloud to get help
| doing AI things
|
| 3. And we tend not to know because it's all "assisstance"
|
| But the solution like anything is _pricing_. I mean yet again
| (uber, Airbnb) billions of dollars of VC money is used as subsidy
| so my photos can get OCR'd.
|
| If phones said "hey for a dollar fifty I can work out what the
| road sign says behind your dogs head in 32 photos your mum sent
| you last week" I think we woukd see a different threat landscape
|
| This is - again - unsustainable cash spending distorting markets
| and common sense. If the market was "we can OCR and analyse these
| insurance claims" the. Things like privacy and encryption _would
| be first class requirements_ and harder to sell and build.
|
| By spending a billion they can sell services to people without
| regulators to ask awkward questions and then they hope step 3.
| Profit.
|
| I short not even AI can spot patterns in encrypted data, it's
| only when plaintext gets sent around in the _hope_ of profit do
| we see a threat. That's seems a simple fix if not an easy one
| fragmede wrote:
| The article hinges on a bad assertion that
|
| > Apple can't rely on every device possessing enough power to
| perform inference locally. This means inference will be
| outsourced to a remote cloud machine.
|
| If you go look at Apple's site https://www.apple.com/apple-
| intelligence/ and scroll down, you get:
|
| Apple Intelligence is compatible with these devices. iPhone 16
| A18 iPhone 16 Plus A18 iPhone 16 Pro Max A18 Pro iPhone 16 Pro
| A18 Pro iPhone 15 Pro Max A17 Pro iPhone 15 Pro A17 Pro iPad Pro
| M1 and later iPad Air M1 and later iPad mini A17 Pro MacBook Air
| M1 and later MacBook Pro M1 and later iMac M1 and later Mac mini
| M1 and later Mac Studio M1 Max and later Mac Pro M2 Ultra
|
| If you don't have one of those devices, Apple did the obvious
| thing and disabled features on devices that don't have the
| hardware to do it.
|
| While Apple has this whole private server architecture, they're
| not sending iMessages off device for summarization, that's
| happening on device.
| flossposse wrote:
| Green, (the author), makes an important point: > a technical
| guarantee is different from a user promise. [...] End-to-end
| encrypted messaging systems are intended to deliver data
| securely. They don't dictate what happens to it next.
|
| Then Green seems to immediately forget the point they just made,
| and proceed to talk about PCC as if it were something other than
| just another technical guarantee. PCC only helps to increase
| confidence that the software running on the server is the
| software _Apple_ intended to be there. It doesn 't give me any
| guarantees about where _else_ my data might be transferred from
| there, or whether Apple will only use it for purposes I 'm okay
| with. PCC makes Apple less vulnerable to hacks, but doesn't make
| them any more transparent or accountable. In fact, to the extent
| that some hackers hack for pro-social purposes like exposing
| corporate abuse, increased security also serves as a better
| shield _against_ accountability. Of course, I 'm not suggesting
| that we should do away with security to achieve transparency. I
| am, however, suggesting that transparency, moreso than security,
| is the major unaddressed problem here. I'd even go so far as to
| say that the woeful state of security is enabled in no small part
| _by_ lack of transparency. If we want AI to serve society, then
| we must reverse the extreme information imbalance we currently
| inhabit wherein every detail of each person 's life is exposed to
| the service provider, but the service provider is a complete
| black-box to the user. You want good corporate actors? Don't let
| them operate invisibly. You want ethical tech? Don't let it
| operate invisibly.
|
| (Edit: formatting)
| p4bl0 wrote:
| This article should be accompanied by _Five things privacy
| experts know about AI_ [1]. They pair really well and should
| probably link to each other.
|
| [1] https://news.ycombinator.com/item?id=42695628
| fourthark wrote:
| (which discusses privacy issues on the _training_ side)
| ajb wrote:
| The real threat here is going to be when AI expands from being
| applied to accelerate the work of individuals, to being applied
| to the control of organisations. And it will be tempting to do
| that. We all know the limitations of managers, management
| hierarchies, metrics, OKRs etc. It's easy to think of a CEO
| deciding that all the communications between their employees
| should just be fed into an AI that they can query . (Ironically
| that would be easier to enforce if everyone was remote). It's
| quite possible that it would enable more effective organisations,
| as the CEO and upper level management can have a better idea of
| what is really happening. But it will reduce the already tenuous
| belief of the powerful that their ordinary staff are real human
| beings. And it will inevitably leak out from private
| organisations, as the executive class see no reason why they
| shouldn't have the same tools when running the country as when
| running a corporation.
|
| Advocates of mass surveillance like to point out that no human
| now needs to listen to your calls. But the real danger was never
| the guy in a drab suit transcribing your conversions from reel-
| to-reel tape. It was always the chief who could call for the
| dossier on anyone they were finding inconvenient, and have it
| looked at with an eye to making you never inconvenience them
| again.
|
| The full consequences of mass surveillance have not played out
| simply because no one had the means to process that much ad-hoc
| unstructured data. Now they do.
| TeMPOraL wrote:
| > _It 's easy to think of a CEO deciding that all the
| communications between their employees should just be fed into
| an AI that they can query . (Ironically that would be easier to
| enforce if everyone was remote)._
|
| This is already happening, whether the CEOs want it or not -
| when there's a legal issue requiring discovery, e-discovery
| software may be used to pull in all digital communications that
| can be accessed, and feed it all to AI for, among other things,
| sentiment analysis. Applications of GenAI for legal work, in
| general, is a hot topic in legal circles now.
| 1vuio0pswjnm7 wrote:
| "You've probably also noticed Silicon Valley's enthusiasm to find
| applications for this tech."
|
| A "solution" looking for a "problem". Like cryptocurrency. Like
| the "metaverse". And so on.
|
| When the "solution" brings new problems SillyCon Valley then
| proclaims they have a solution to the problem they created.
|
| "Even if these firms don't quite know how AI will be useful to
| their customers, they've already decided that models are the
| future."
|
| More often than not the "customers" are advertisers. Other
| computer users are just ad targets.
| TeMPOraL wrote:
| GenAI (and LLMs in particular) is a "solution looking for a
| problem" in the sense _petroleum_ or _electricity_ was, not in
| the sense cryptocurrency is. I.e. you take one look at it and
| it 's 100% obvious it's a _general_ technology that 's going to
| be useful in some way for all kinds of work people do, and be
| doing in the near future, _and_ itself will make possible
| things and endeavors that weren 't possible before.
|
| The devil is always in the details, but assuming that GenAI
| _might_ be important for your business is correct no matter
| what business you 're in, so it's worth it to pay attention and
| test it sooner rather than later.
| reader9274 wrote:
| For some reason I felt like this article had a lot of promise to
| start, but turned into fluff and I learned nothing. It covered
| very basic ideas of both AI and encryption, which I didn't expect
| from such a highly regarded expert.
| dathinab wrote:
| > They'll order your food and find the best deals on shopping,
| swipe your dating profile, negotiate with your lenders, and
| generally anticipate your every want or need.
|
| Why should they do so?
|
| I mean seriously.
|
| There is more money to make in telling you that the AI will buy
| you the beast deal but instead buy premeditated (i.e. bought)
| "okay" looking deals instead.
|
| Similar dating apps and the related ecosystems has a long history
| of scamy behavior in all kind of ways as they want to keep you
| using the app. And people with money have always found ways to
| highlight themself more. I.e. there is more money to make in
| "swiping for you" in way which looks on the surface honest but
| isn't.
|
| etc. etc.
|
| There is basically always more money to make in systematically
| deceiving people as long as you do it well enough so that people
| don't notice, or they don't have a real/realistic choice, i.e.
| are forced by the circumstances.
|
| So the moment you take the human and human conscience/moral out
| of the loop and also have no transparency there is a 0% of this
| ending well if regulators don't preclude abuse. And AI is pretty
| effective at removing transparency and the humanity out of the
| loop. With how things currently look, especially in the US, they
| (edit: they==regulators) are more likely to do the opposite
| (edit: i.e. remove consumer protections).
| xp84 wrote:
| "who does the ai work for"
|
| I think even ignoring the more scary government/cops questions,
| this gets to a key problem. Because for 25 years we've taught 2
| generations of digital natives as well as everybody else that
| everything Internet should be "free as in beer" paid for by
| trashy advertising, and that consumers' only cash cost should be
| hardware and ISP/data plan. Therefore the answer to "who's going
| to foot the bill for that AI" "must" be Big Adtech meaning the AI
| 100% will work for them and often directly against users'
| interests (just as the YouTube algorithm mindlessly but
| intentionally prefers to radicalize a user or drive them to
| obsession on any number of topics vs. having them arrive at a
| healthy level of satisfaction and then sign off and go outside).
|
| In my opinion a lot of these problems we see in the scary law
| enforcement scenarios would be easier to solve if we didn't
| expect everything to be ad-supported "free" and rather, we could
| be convinced to buy a $5000 piece of hardware for your home, that
| you control, that was privy to your encryption keys and performed
| all the power-insensitive AI processing for your family. That
| sounds a lot but compared to things like cars that people happily
| finance for $70,000 and smartphones which cost $1300 it is only
| weird because we aren't used to it.
___________________________________________________________________
(page generated 2025-01-18 23:01 UTC)