[HN Gopher] Disrupting deceptive uses of AI by covert influence ...
___________________________________________________________________
Disrupting deceptive uses of AI by covert influence operations
Author : Jimmc414
Score : 80 points
Date : 2024-05-30 17:06 UTC (5 hours ago)
(HTM) web link (openai.com)
(TXT) w3m dump (openai.com)
| catchnear4321 wrote:
| > Activity by a commercial company in Israel called STOIC,
| because technically we disrupted the activity, not the company.
|
| hmm. is this legal cya because a name was named? naming and
| shaming is a curious choice. taking care to specify what was
| disrupted is... well, it draws the eye.
|
| all others were called operations or groups. unlikely the groups
| were disrupted even if their operations were. no difference in
| pattern. just in categorization. company vs cell.
|
| what is the extra care for, exactly?
| giraffe_lady wrote:
| Probably all of them are state affiliated but only this one
| affiliated with a state the US considers an ally?
| thiagoharry wrote:
| This is what I thought reading the article. To be fair,
| before reading the article I was expecting to find zero
| disrupted activities related with the west and its allies,
| but the fact that they disrupted that AI usage by Israel was
| a good surprise. However, I really doubt that US does not do
| the same, spreading propaganda in Internet, based on the fact
| that they employ and finance several radios, and newspapers
| around the globe with this purpose (ex: Radio Free Asia).
| Giving how the US government and tech companies are close,
| with backdoors planted in CISCO routers, and information
| collected from social media, I find hard to believe that
| OpenAI or any other US-based company is trusted to report and
| disrupt activities like these when performed by US
| government, or perhaps some closer allies.
| giraffe_lady wrote:
| Yes agreed with all of that. I'm sure this was handled...
| carefully.
| grooly wrote:
| Meta also reported disrupting activity from the same company:
|
| https://www.reuters.com/technology/meta-identifies-networks-...
| kbigdelysh wrote:
| Based on the Reuters article, The Israeli company STOIC put
| many ai-generated comments on social media posed from Jewish,
| African Americans and concerned citizens praising Israel's
| handling of the war in Gaza.
| sturza wrote:
| The operations were:
|
| * A previously unreported operation from Russia, which we dubbed
| "Bad Grammar", operating mainly on Telegram and targeting
| Ukraine, Moldova, the Baltic States and the United States;
|
| * A persistent Russian threat actor posting content about Ukraine
| across the internet, known as "Doppelganger";
|
| * A persistent Chinese threat actor posting content across the
| internet to praise China and criticize its critics, known as
| "Spamouflage";
|
| * A persistent Iranian threat actor posting web content that
| supported Iran and criticized Israel and the US, known as the
| International Union of Virtual Media (IUVM);
|
| * A commercial company in Israel called STOIC, generating content
| about the Gaza conflict, and to a lesser extent the Histadrut
| trade unions organization in Israel and the Indian elections. We
| have nicknamed this operation "Zero Zeno" for the founder of the
| stoic school of philosophy, and to reflect the low levels of
| engagement that its various campaigns attracted.
| newzisforsukas wrote:
| No US based operations targeting foreign or domestic entities?
| I guess they already have that in house?
| red-iron-pine wrote:
| we've been talking about GPT-fueled propaganda campaigns on
| HN since before COVID... so very likely in-house.
|
| Likely in-house for many of the other actors above; this may
| just be low-hanging fruit, or operations designed to hit
| specific demographics in specific countries.
| mjburgess wrote:
| With OpenAI's breathless "fears" about AI safety, their
| anthropomorphising a chatbot interface, and so on.. one should be
| aware here that OpenAI isn't against deception, it just wishes to
| control its manner.
|
| Note also, that in the west corporate propaganda is permissible,
| whilst state propaganda is orwellian.
|
| It's no mystery then that much state propaganda appears under a
| corporate guise. If OpenAI were against corporate propaganda,
| much of its business model would vanish overnight.
| nottorp wrote:
| OpenAI will protect us against the evils of AI! They just need
| a state sanctioned monopoly or at least mandatory "safety
| measures" that will raise the barrier to enter the market to
| beyond what your average startup can raise.
| exe34 wrote:
| what matters is what the oligarchs control. in the east, it's
| the state. in the west, the corporations. democracy is just the
| ointment to help with the pain, for now. eventually it won't be
| needed.
| vintermann wrote:
| > A previously unreported operation from Russia, which we dubbed
| Bad Grammar
|
| Good! Great that they don't give criminals cool names ...
|
| > Activity by a commercial company in Israel called STOIC,
| because technically we disrupted the activity, not the company.
| We nicknamed this operation Zero Zeno, for the founder of the
| stoic school of philosophy
|
| Never mind.
| mamonster wrote:
| Russian bots are targeting 9GAG?
|
| What year is it?
| red-iron-pine wrote:
| they're still pretty active on the *chan's, if I believe threat
| intelligence reporting. no surprise they're in 9GAG.
|
| also slashdot, hackernews, and various reddits. it call all be
| automated, and failing that, plenty of folks willing to do the
| "mechanical turk" approach and shitpost for pay.
| mamonster wrote:
| They are for sure on 4chan(AFAIK there is a pretty obvious
| propaganda thread that is updated daily), but I thought 9Gag
| was dead? Do people still get memes there?
| jazzyjackson wrote:
| maybe its just the dev environment, lacking good antibot
| mechanisms they can test new delpoyments without showing
| their hand to reddit etc
| dekhn wrote:
| I have very little confidence that OpenAI has the technical
| expertise or company culture required to deal with APT (advanced
| persistent threats).
| ActionHank wrote:
| Also, the article doesn't state it outright, but the
| implication is that they only stopped bad actors after they had
| already been using their services, with no indication of how
| long they were using them for.
| baxtr wrote:
| I recommend anyone interested in the subject to study Meta's
| Adversial Threat Reports.
|
| They started their counter operations a while back and
| regularly draft extensive reports [1].
|
| Reading through those reports it becomes obvious how much
| effort this requires.
|
| I'm not sure how much effort OpenAI can put into this at all.
|
| [1] https://transparency.meta.com/de-de/metasecurity/threat-
| repo...
| lolinder wrote:
| This is a slight tangent, but I would _love_ to see a
| sentiment analysis graph of HN perceptions of Meta /Facebook
| over time for the past two years or so. Subjectively, my
| sense is that releasing Llama 2/3 gained them a lot of good
| will in this community and attitudes have generally become
| much more positive towards them as OpenAI's star has fallen.
|
| In terms of developer goodwill, they may be the real winners
| of this whole gold rush.
| baxtr wrote:
| I had a similar thought recently.
|
| They really do a good job on many fronts right now.
|
| Additionally the whole TikTok debate will help them a lot
| too.
| ben_jones wrote:
| Facebook, I mean Meta, has benefitted from the
| enshittification of the tech industry.
|
| Next to TikTok they're better moralistically. Next to Sam
| Altman, I mean OpenAI, they're better moralistically. Next
| to other companies doing layoffs they're better
| moralistically (biggest exit packages?)
| brokenmachine wrote:
| I'm not a fan of them using our data to train their AI,
| with very obscure opt-out procedures.
|
| Not unexpected but just what you'd expect.
|
| But I haven't used Facebook for about a decade, so
| -\\_(tsu)_/-
| summarity wrote:
| Looking at open roles, it seems like they're fleshing out their
| security research program: https://openai.com/careers/research-
| program-manager-security...
| rozap wrote:
| I think if the mossad is included in your threat model, you may
| as well just pack up and go home.
| rurp wrote:
| Maybe in some cases but certainly not for systemically
| important platforms, as OpenAI aspires to be.
|
| Google and Microsoft don't just throw up their hands at the
| threat of nation-state actors; doing so would be hugely
| irresponsible.
| rozap wrote:
| All technologies live on some scale of exploitability.
| Mobile phones, for example, are somewhere on that spectrum,
| and google and apple try their best to secure them. But
| there are still state actors that have really sophisticated
| attacks that worm their way into devices. To be clear, I
| think the security teams at google and apple shouldn't
| throw their hands up and give up, but I think you need to
| be honest with yourself when building capability, to accept
| that it'll get misused. If the misuse doesn't outweigh the
| benefit to humanity, then it's pretty easy to sleep at
| night.
|
| On the scale of exploitability, I think some of the LLMs we
| have now are pretty high up there, beyond mobile phones -
| harder to secure and with bigger consequences when misused.
| The intelligence agencies of pariah states like Russia,
| Iran and Israel will absolutely use this stuff to sway
| public opinion and get large numbers of people killed. I
| think employees of OpenAI need to be honest with themselves
| about this, rather than trying to say "oh we'll just do
| security/trust/safety" better than the a nation state.
|
| I think there are a number of ways out of this which are
| consistent. You could convince yourself that AI's benefit
| to humanity as a whole over time will outweigh these issues
| now [citation needed], or you could take a nihilistic
| approach and say that you just wanna print stacks of money
| and go relax on the beach. Those are at least arguments you
| can make. But I think it's intellectually dishonest to say
| "Bad stuff won't happen with our tech because our team will
| out compete the mossad".
| renegade-otter wrote:
| I argue here that sentient AI destroying the world is effectively
| fiction. Beware of the people, state actors, and those who get
| ultra rich on the hype and then flood our politics with money:
| https://renegadeotter.com/2024/04/22/artificial-intelligence...
| cheald wrote:
| Given that any of these bad actors can spin up local inference
| infrastructure to run their operations pretty trivially, what
| _practical_ effect does this have beyond just "hey, look, we're
| doing something"?
| thegrim33 wrote:
| Exactly .. every state actor will already have their own local
| infrastructure for all this, there's no way they're actually
| dependent on using OpenAI. Most likely this is just a disinfo
| campaign - "hey look guys, we caught some bad covert actors, so
| there's totally no nation states making massive use of LLMs to
| manipulate online discourse, look we caught them, everything is
| secure, don't worry about it".
|
| The funny part is, if it is all a disinfo campaign, then the
| very resources behind the campaign will downvote and attack
| this post and try to silence/discredit such voices. Or, maybe
| I'm just completely wrong and I'll get downvoted for having
| dumb beliefs. There's actually very little way to tell the
| difference between the two scenarios. In addition, does the act
| of me pointing out that malicious actors would want to downvote
| this now make it less likely for them to actually do so, as it
| would show their hand? You can recurse down these rabbit holes
| forever. Fun stuff.
| ImPostingOnHN wrote:
| Your comment violates the site rules (commenting on
| downvotes) and cheapens the conversation with the same
| unfalsifiable conspiracy victim rhetoric, so you should
| expect to see downvotes for that alone, even if none of the
| rest of the comment existed.
| ben_w wrote:
| Even if you're not skilled enough to stop the FSB from misusing
| your services, it's worth putting in the effort to stop the
| next Jim Jones from doing so.
|
| And learning to stop a Jim Jones, even knowing that you need
| to, may help you stop you from being named by the UN as
| complicit in a genocide, the way the UN explicitly named
| Facebook as having played a "determining role" in the Rohingya
| genocide:
| https://web.archive.org/web/20220217165239/https://www.reute...
|
| Of course, if you want to prevent even the most capable of
| intelligence agencies from spinning up their own local LLMs at
| all, then at a minimum you can't ever publish the model weights
| themselves -- a solution which itself is the cause of around
| 50% of the criticisms people around here make of OpenAI.
| hermitdev wrote:
| Honestly, who cares what the fuck the UN thinks at this
| point? They've lost all credibility. Hell, they just held a
| moment of silence for the Butcher of Tehran, an individual
| that, as a head of state, has repeatedly called for the
| destruction of Israel and the USA, the genocide of Jews. Not
| just words, but actions, too. Likely behind the Oct 7th
| attacks against Israel as well, that literally triggered the
| current war in Gaza.
|
| So, the UN should take a hard long look in the mirror before
| it accuses anyone else of being complicit in genocide.
| nerdjon wrote:
| Something about them caring about this bothers me a bit.
|
| Like, yeah ok it is a good thing that they are trying to curtail
| this abuse.
|
| However, it kinda feels like we are just delaying the inevitable?
| With open source models like LLAMA, detecting and doing anything
| about this will become impossible in a year or 2 (in reality its
| already possible).
|
| It just feels very, "look at us doing this good thing but ignore
| what we unleashed on the world" so please pat us on the back
| here.
|
| Maybe I am just pessimistic, but I am not about to give OpenAI
| any praise here for this when they are the reason we are in the
| current situation.
| UberFly wrote:
| My feelings exactly. As usual it feels like lawyers are saying
| "just do this even though it's BS so in a future court room you
| can bring it up". You're not pessimistic, just realistic.
| ADeerAppeared wrote:
| > It just feels very, "look at us doing this good thing but
| ignore what we unleashed on the world" so please pat us on the
| back here.
|
| It is that.
|
| These AI firms have been very vocal about their "ends justify
| the means" attitude. (Which is doubly damning considering how
| little these firms care about any purported AI safety for their
| AGI dreams. Months of "AI is going to kill us all" while they
| couldn't even be arsed to stop using CSAM-contaminated training
| data, nevermind using quality training data to ensure
| alignment)
|
| And "Foreign state actors generated some propaganda texts" is
| very small as AI-problems in the here-and-now go.
| S0y wrote:
| >We've terminated accounts linked to covert influence operations;
| >no significant audience increase due to our services.
|
| I can't help but find this extremely funny. you mean to tell me
| that massive operations trying to use your service didn't
| actually gain anything from using your service?
|
| OpenAI wasting time and resources shutting them down and the bad
| actors wasting time and resources using chatGPT no benefit.
|
| It just reads like a massive waste of time for everyone involved.
| quinncom wrote:
| Hilarious, I love this take.
|
| I'd guess that "covert ops" using ChatGPT are amateurs, by
| definition (because it's clearly not covert). Surely, the
| serious operators would use offline local LLMs.
| mensetmanusman wrote:
| You're thinking too much like an engineer. News like this
| teaches people that they are clearly ahead in capabilities far
| beyond any government or other corporation.
|
| Anytime they can get good publicity like this is good for their
| business.
| dialup_sounds wrote:
| I get the sense that they would be using ChatGPT to _save_ the
| time and resources required to produce content, rather than
| expecting it to produce better results.
| barryrandall wrote:
| How effective were OpenAI's safety/security measures?
|
| With these confirmed bad actors, what percentage of their usage
| attempts were disrupted by automatic safety measures?
|
| Of the blocked attempts, is there any evidence that the influence
| actor successfully circumvented those protections?
| asne11 wrote:
| > we repeatedly observed cases where our models refused to
| generate the text or images that the actors asked for
|
| So the protections were bypassed at least some of the time.
| Many of these are demonstrably trivial to bypass.
| hex4def6 wrote:
| "Our Usage Policies prohibit the use of our services to deceive
| or mislead others. This includes deceiving people about the
| authorship or source of content generated using our models, by
| engaging in activities such as: ... Creating automated systems
| that falsely present as being a real person, or don't disclose to
| people that they are interacting with AI (unless it's obvious
| from the context)."
|
| Suspiciously absent from the list of takedowns is anything on
| Reddit. Given the amount of obviously-chatGPT puppets on there,
| I'm somewhat 'surprised'.
|
| Also a good reminder that if you're using ChatGPT for anything,
| they can and will examine the traffic and contents of your
| messages / replies.
|
| I wonder if in the future they may offer a "service" to content
| providers. Say you're a reddit / forum / etc. You get supplied an
| API that you can run queries through that match any content that
| may have been generated through their service. You can then tag
| those accounts as bots. Of course, such an API will have costs
| associated with it...
| hermitdev wrote:
| > I wonder if in the future they may offer a "service" to
| content providers. Say you're a reddit / forum / etc. You get
| supplied an API that you can run queries through that match any
| content that may have been generated through their service. You
| can then tag those accounts as bots. Of course, such an API
| will have costs associated with it...
|
| I'm not a lawyer, but this sounds like racketeering to me.
| Basically extorting "protection" money from sites.
| 082349872349872 wrote:
| We were hearing a little while back that it had become
| exceedingly difficult to concoct diplomatic cover for
| intelligence agents, in a world full of socials.
|
| Does the availability of AI mean that (as long as you own, or at
| least can retcon, several socials?) spinning up entire innocuous
| provenance networks for fresh identities is once again a push-
| button solved problem?
| rurp wrote:
| It might help against casual checks, but a large web of
| invented social accounts will be uncoverable by any serious
| actor. If there's no corroborating trail outside of social
| media for any of those accounts it will be very possible to
| sniff out the fakery.
| tivert wrote:
| > We were hearing a little while back that it had become
| exceedingly difficult to concoct diplomatic cover for
| intelligence agents, in a world full of socials.
|
| > Does the availability of AI mean that (as long as you own, or
| at least can retcon, several socials?) spinning up entire
| innocuous provenance networks for fresh identities is once
| again a push-button solved problem?
|
| AI doesn't defeat biometrics. I'd bet the problem with social
| media for intelligence agents is it allows facial recognition
| to uncover their true identity. If the search uncovers a real
| profile and a fake AI generated one with fake AI generated
| friends, it's still fishy to an adversary.
|
| Also if you're an agent and have gone to a country under one
| name, you can never go back under a different name without
| being trivially caught.
___________________________________________________________________
(page generated 2024-05-30 23:02 UTC)