[HN Gopher] Human study on AI spear phishing campaigns
___________________________________________________________________
Human study on AI spear phishing campaigns
Author : DalasNoin
Score : 117 points
Date : 2025-01-05 13:40 UTC (9 hours ago)
(HTM) web link (www.lesswrong.com)
(TXT) w3m dump (www.lesswrong.com)
| TechDebtDevin wrote:
| Grandma is fkd
| noman-land wrote:
| We are all grandma.
| pavel_lishin wrote:
| I've had coworkers, and so has my spouse, who has fallen for
| the "iTunes gift cards for the CEO" trick. I think _grandma_
| is no longer an accurate stand-in for a tech-unsavvy person
| who is vulnerable to spearphishing attempts.
| consumer451 wrote:
| I would argue that pretty much everyone could be socially
| engineered into dropping their guard for a moment.
| e40 wrote:
| I get about 10 emails per week from my "CEO" to pay an
| invoice. I've even gotten a few text messages. Oddly, the
| emails never have an attachment. Is this because Google
| (Workspace account) is removing it?
|
| I've always wondered if it is 10 different orgs doing the
| campaigns, or the same one. If the same one, why send 10?
| noman-land wrote:
| This is somehow not considered to be an active warzone
| when it clearly is. The slightest misstep could ruin your
| life.
| pavel_lishin wrote:
| > _I 've always wondered if it is 10 different orgs doing
| the campaigns, or the same one. If the same one, why send
| 10?_
|
| My bet is that one criminal group is selling software to
| enable this, with very similar default settings. Then ten
| groups by the software, and each one ends up sending you
| a very similar email.
| terribleperson wrote:
| This is one of the terrifying, probably already happening threats
| presented by current LLMs.
|
| Social engineering (and I include spearphishing) has always been
| powerful and hard to mitigate. Now it can be done automatically
| at low cost.
| cluckindan wrote:
| If the study was done with target consent, it might be biased
| with inflated click-through rates due to the targets expecting
| benign well-targeted spear-phishing messages.
|
| If it was done without target consent, it would certainly be
| unethical.
| neom wrote:
| They got IRB approval. The authors framed the emails as part of
| a marketing study involving "targeted marketing emails."
| jt2190 wrote:
| It seems like "the subject clicked a link in an email" is
| equated to "being phished", but I'm not certain that is a good
| definition.
| Terr_ wrote:
| I'm certain that someday I'm going to be dinged on a really
| shallow kind of work security test because I decided to
| investigate a link into a sandbox/honeypot environment.
| RaptorJ wrote:
| These phish testing companies always stick a header
| (X-PHISH-TEST or some such) on the email so the email
| server can white-list -- easy to just Outlook blackhole
| filter anything with that header after you've seen one
| test.
| Terr_ wrote:
| What stops an attacker from abusing the same header?
|
| It could be kinda-secure if the header had to have a
| payload which matched a certain value pre-approved for a
| time-period. However an insider threat could see the test
| going on and then launch their own campaign during the
| validity window.
| joe_the_user wrote:
| That, after thirty years, email security still depends on the
| wisdom of individuals not clicking the wrong link, is appalling.
|
| The situation involves institutions happy to opaque links to
| email as part of their workflow. What could change this? All I
| can imagine is state regulation but that also is implausible.
| concerndc1tizen wrote:
| The same is true for operating systems. Why don't they sandbox
| properly?
|
| We have sandboxing on mobile apps. Why can't we have the same
| for desktop?
| andersa wrote:
| After all these years, Microsoft is _finally_ rolling out
| win32 app isolation, so maybe we are finally on the good
| path...
| fassssst wrote:
| Developers initially revolted against Microsoft UWP and Mac
| App Store.
| wetpaws wrote:
| Walled garden is not a substitute for security
| Y_Y wrote:
| Not because they isolated the applications though! Because
| they were shit, and that's not a requirement.
| joe_the_user wrote:
| Asking for fully bug free software is nice but unrealistic.
| Browsers are ostensibly somewhat sandboxed too but there are
| always new zero-days 'cause browsers are essentially OSes
| with many moving parts.
|
| However, it reasonable to expect _a single hole_ to be fixed.
| The "email hole" has been discussed for decades but here we
| are.
| makeitdouble wrote:
| Email is still the running blood of the internet. While we
| mostly get away with Slack and others for in-group
| communication, anything going outside, especially to
| customers, still goes through emails.
|
| At that scale, expecting a core issue to be quickly (or
| ever) fixed is just unrealistic. I honestly wonder if
| fundamentally it will ever be fixed, or if instead we get a
| different communication path to cover the specific use
| cases we do care about security.
|
| PS: the phone is now 2 century olds, and we sure couldn't
| solve scamming issues...
| throw10920 wrote:
| > Why can't we have the same for desktop?
|
| Morally? No reason why, and people are working on it
| (slowly).
|
| Practically? Because sandboxing breaks lots of things that
| users and developers like, such as file picking (I hate
| snaps), and it takes time to reimplement them in a sandbox in
| the way that people expect them to work. If it requires the
| developers' cooperation, then it's even slower, because
| developers have enough APIs to learn as it is.
| bandrami wrote:
| And to the extent you mitigate some of those user
| complaints (as flatpak etc. are doing) you are basically
| re-opening the exact same holes that you developed the
| sandbox to get away from
| rawgabbit wrote:
| I blame Microsoft. As a consumer OS its default stance should
| be no, the user does not intend to grant god permissions to
| this embedded or external script when they clicked it. Instead
| the user should have been challenged with a dialog, do you want
| to install this App and then execute?
| bennythomsson wrote:
| To which everybody will click yes. They have been conditioned
| by too much half-baked crap out there that requires it and
| the need to go on with their lives instead of having tp start
| investigating things they anyway don't have a clue about (and
| don't want to, not being IT folks).
| bongodongobob wrote:
| This is why you don't daily drive a local admin and leave UAC
| enabled. If you were using an unprivileged account, you'd be
| getting UAC prompts.
| richardw wrote:
| Email and web browsing relies on "deny lists" rather than
| "allow lists". So anything goes but you block bad addresses,
| rather than nothing until you get
| permissions/trust/credibility. This helped growth of all the
| networks but means indefinite whack a mole.
|
| I think (but am not sure) that something using trust networks
| from the ground up would be better in the long term. Consider
| anything dodgy until it has built trust relationships.
|
| Eg email servers can't just go for it. You need time to warm up
| your IP address, use DKIM etc. People can't just friend you on
| FB without your acceptance so it's a lot safer than email, if
| still not perfect. A few layers of trust would slow bad actors
| down significantly.
|
| A trust network wouldn't be binary. Having eg a bunch of spam
| accounts all trust each other wouldn't help getting into your
| social or business network.
|
| Thoughts from experts?
| MetaWhirledPeas wrote:
| > Email and web browsing relies on "deny lists" rather than
| "allow lists". So anything goes but you block bad addresses,
| rather than nothing until you get
| permissions/trust/credibility.
|
| But this is fundamental to an open Internet. Yes going
| whitelist-only would stop bad actors but it would also hand
| over the entire internet to the megacorps with no avenue for
| individual success.
| joe_the_user wrote:
| Email and browsers shouldn't be glibly equated.
|
| Email as it is presently is a constant opening to phishing
| and spear fishing. Browser exploits are common too but it's
| harder (not impossible) to make them personal. And phishing
| doesn't have to rely on a browser exploit - a fake login
| page is enough.
|
| It's logical to have a whitelist (or disallow) email links
| but still allow browsers to follow links.
| ano-ther wrote:
| I just received a corporate IT security training link. From an
| external address and with a cryptic link. After a previous
| training which asked us not to trust external emails
| (spoofable) especially not with unknown links.
|
| IT wasn't amused when I reported it as phishing attempt.
| nickpinkston wrote:
| Haha - amazing. I've had the same thought, and I'm sure the
| scammers have too.
| perching_aix wrote:
| Technologically, email aliases have been working wonders for me
| in personal use. No idea if it could be rolled out effectively
| for nontechnical users at an organizational scale though, even
| with automation.
|
| It also does little against compromised mailboxes - heck, a
| sufficiently advanced spear fish might even have better chances
| if the user misunderstands the security improvements this would
| provide.
|
| But I think other than this, there's not much else to fix. Some
| people are malicious, others get compromised. No fixing that.
| rapind wrote:
| A good start would be ditching HTML in email. Plain text is
| perfectly suitable for non-marketing emails (and marketing
| emails are just chaff at this point anyways).
|
| I'll die on this hill.
| MetaWhirledPeas wrote:
| One word deserves so much blame for the current state of the
| internet: marketing
| LeftHandPath wrote:
| They built their phishing emails using data scraped from public
| profiles. Fascinating.
|
| I have to wonder if, in the near future, we're going to have a
| much higher perceived cost for online social media usage.
| Problems we're already seeing:
|
| - AI turning clothed photos into the opposite [0]
|
| - AI mimicking a person's voice, given enough reference material
| [1]
|
| - Scammers impersonating software engineers in job interviews,
| after viewing their LinkedIn or GitHub profiles [2]
|
| - Fraudsters using hacked GitHub accounts to trick other
| developers into downloading/cloning malicious arbitrary code [3]
|
| - AI training on publicly-available text, photo, and video, to
| the surprise of content creators (but arguably fair use) [4]
|
| - AI spamming github issues to try to claim bug bounties [5]
|
| All of this probably sounds like a "well, duh" to some of the
| more privacy and security savvy here, but I still think it has
| created a notable shift from the tech-optimism that ran from
| 2012-2018 or so. These problems all existed then, too, but with
| less frequency. Now, it's a full-pressure firehose.
|
| [0]: https://www.wsj.com/politics/policy/teen-deepfake-ai-
| nudes-b...
|
| [1]: https://www.fcc.gov/consumers/guides/deep-fake-audio-and-
| vid...
|
| [2]: https://connortumbleson.com/2022/09/19/someone-is-
| pretending...
|
| [3]: https://it.ucsf.edu/aug-2023-impersonation-attacks-target-
| gi...
|
| [4]: https://creativecommons.org/2023/02/17/fair-use-training-
| gen...
|
| [5]: https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-
| stands-f...
| hibikir wrote:
| This lines up well with the success rates I have seen from expert
| phishers. When I worked at a certain well known company with
| strong security, a demon called Karla would succeed at
| spearphishing a bit over 50% of the security team.
|
| AI now means much less skilled people can be as good as she was.
| Karla as a Service. We are doomed.
| richdougherty wrote:
| "The cost-effective nature of AI makes it highly plausible
| we're moving towards an agent vs agent future."
|
| Sounds right. I assume we will all have AI agents triaging our
| emails trying to protect us.
|
| Maybe we will need AI to help us discern what is really true
| when we search for or consume information as well. The amount
| and quality of plausible but fake information is only going to
| increase.
|
| "However, the possibilities of jailbreaks and prompt injections
| pose a significant challenge to using language models to
| prevent phishing."
|
| Gives a hint at the arms race between attack and defense.
| ninkendo wrote:
| What defines a successful spear phishing? Is it just clicking a
| link?
|
| My process when I see a sketchy email is to hover over the
| links to see the domain. Phishing links are obvious to anyone
| who understands how URLs and DNS works.
|
| But working for a typical enterprise, all links are "helpfully"
| rewritten to some dumbass phishing detection service, so I can
| no longer do this.
|
| At my current company I got what I assumed was a phishing
| email, I hovered over the links, saw they were pointing to some
| dipshit outlook phishing detection domain, and decided "what
| the hell, may as well click... may as well see if this phishing
| detection flags it" [0]...
|
| ... and it turns out it was not only not legit, but it was an
| internal phishing test email to see whether I'd "fall for" a
| phishing link.
|
| Note that the test didn't check if I'd, say, enter my
| credentials into a fraudulent website. It considered me to have
| failed if I merely clicked a link. A link _to our internal
| phishing detection service_ because of course I'm not trusted
| to see the actual link itself (because I'd use that to check
| the DNS name.)
|
| I guess the threat model is that these phishers have a zero-day
| browser vulnerability (worth millions on auction sites) and
| that I'd be instantly owned the moment I clicked an outlook
| phishing service link, so I failed that.
|
| Also note that this was a "spear phishing" email, so it looked
| like any normal internal company email (in this case to a
| confluence page) and had my name on it. So given that it looks
| nearly identical to other corporate emails, and that you can't
| actually see the links (they're all rewritten), the takeaway is
| that you simply cannot use email to click links, ever, in a
| modern company with typical infosec standards. Ever ever. Zero
| exceptions.
|
| - [0] My threat model doesn't include "malware installed the
| moment I click a link, on an up to date browser", because I
| don't believe spear phishers have those sort of vulnerabilities
| available to burn, given the millions of dollars that costs.
| bongodongobob wrote:
| Problem is Outlook now obfuscates the shit out of links,
| something something safesearch or along those lines. When I
| hover over a link, I now have no idea where it wants to take
| me unless I copy and paste it and look through the 500
| character link to find where it actually wants to take me.
| DoctorOetker wrote:
| Is this the same Karla as in Fight Club?
| pavel_lishin wrote:
| I believe that was Marla.
| justinl33 wrote:
| this research actually demonstrates that AI will reduce the
| phishing threat long-term, not increase it. Yes, the 50x cost
| reduction is scary, but it also completely commoditizes the
| attack vector.
| perching_aix wrote:
| I'm sorry but I'm not sure I follow. How do you mean that the
| commoditization of spear fishing will reduce phishing threats
| long term? To me that implies the exact opposite would happen?
| 101008 wrote:
| I made a purchase yesterday from Meta (Oculus). A few minutes
| after payment, I received an email asking to click to confirm it
| was me.
|
| It came from verify@verification.metamail.com, with
| alert@nofraud.com cc. All red flags for phishing.
|
| I googled it because it had all the purchase information, so
| unless a malicious actor infiltrated Meta servers, it has to be
| right. And it was, after googling a bit. But why do they do such
| things?i would expect better from Meta.
| tomashubelbauer wrote:
| I experienced the exact same thing when I bought the Flipper
| Zero. A "hacker device" and the email communication following
| the sale being made was straight out of a phishing email
| campaign book. I don't remember the details, it has been a
| while, but it was wild how sketchy the emails looked. I hope
| they have improved the email templates since.
| frizlab wrote:
| I got way worse. I was fined for leaving an unattended
| baggage at the train station for a bit. The fine came through
| an SMS message redirecting to a domain which I had to whois
| to verify was owned by the train company...
| throw10920 wrote:
| It's always infuriating getting email from Amazon or my bank
| "here's signs of potential phishing emails/texts" that doesn't
| include an exhaustive list of every email address and phone
| number that that organization will try to contact me from. That
| should be _table stakes_ when it comes to phishing avoidance,
| and it 's something that can _only_ be done by the business,
| not the customer.
|
| Yes, like you say, there's always the chance that someone
| hijacked an official domain - that's where other things like a
| formal communication protocol ("we will never ask for your
| password", "never share 2FA codes", "2FA codes are separate
| from challenge-response codes used for tech support") and rules
| of thumb like "don't click on shortened links" come in. Defense
| in depth is a must, but the list of official addresses should
| be the _starting point_ and it isn 't.
| ben_w wrote:
| > i would expect better from Meta
|
| I'm surprised you would expect better.
|
| Everything I hear about their processes, everything I
| experience as a user, says their software development is all
| over the place.
|
| Uploading a video on mobile web? I get the "please wait on this
| site" banner and no sign of progress, never completes. An
| image? Sometimes it's fine, sometimes it forgets rotation
| metadata. Default feed? Recommendations for sports teams I
| don't follow in countries I don't live in. Adverts? So badly
| targeted that I end up reporting some of them (horror films)
| for violent content, while even the normal ones are often for
| things I couldn't get if I wanted to such as a lawyer
| specialising in giving up a citizenship I never had. Write a
| comment? Sometimes the whole message is deleted *while I'm
| typing* for no apparent reason.
|
| Only reason I've even got an account is the network effect. If
| the company is forced to make the feed available to others, I
| won't even need this much.
|
| If they stopped caring about quality of their core product,
| what hope a billing system's verification emails?
| onemoresoop wrote:
| Yes, but to receive a message that is not from them after a
| transaction you just did with them is quite bad.
| makeitdouble wrote:
| Looking at what No Fraud does [0], it sounds like Meta has
| either spun off the first party hardware store from their usual
| infra, or straight asked a third party to deal with it, and to
| insulate their main business they split the email domains.
|
| Most companies are already splitting domains for customer and
| corporate communication, that's a step in the same direction.
|
| While you're right it sounds fishy as hell, it's also mildly
| common IMO and understadable, especially when e-commerce is not
| the main business, and could be a reflection of how anti-
| phishing provisions are pushing companies to be a lot more
| protective of the email that comes from their main domain.
|
| [0] https://www.nofraud.com/faq/
| tucnak wrote:
| You should check your browser extensions!
| 015a wrote:
| "Look, humans will adapt to the ever-increasing and accelerating
| nightmares we invent. They always have before. Technology isn't
| inherently evil, its how it is used that can be evil, its not our
| fault that we make it so accessible and cheap for evil people to
| use. No, we can't build safeguards, the efficient market
| hypothesis leaves no room for that."
| imiric wrote:
| Mostly accurate, except I would change the last sentence to:
|
| "We take safety very seriously. Look how much safer our SOTA
| model is based on our completely made up metrics. We will also
| delay releasing these models to the public until we ensure
| they're safe for everyone, or just until we need to bump up our
| valuation, whichever comes first."
| bennythomsson wrote:
| How did they generate these? If I try with ChatGPT then it
| refuses, citing a possible violation of their content policy.
| Even when I tell it that this is for me personally, it knows who
| I am, and that it's just for a test -- which obviously I could be
| just pretending, but again, it knows who I am but still refuses.
| qwerty2343242 wrote:
| You can host open source llm offline.
| ben_w wrote:
| They team specifically "use AI agents built from GPT-4o and
| Claude 3.5 Sonnet". The question here is "how did they manage
| to do so" not "what else can do it with less effort".
|
| As those two are run by companies actively trying to prevent
| their tools being used nefariously, this is also what it
| looks like to announce they found an unpatched bug in an
| LLM's alignment. (Something LessWrong, where this was
| published, would care about much more than Hacker News).
| a1j9o94 wrote:
| If you're using ChatGPT directly as opposed to the API, the
| system prompts could be driving it.
|
| Also, in section 3.6 of the paper, they talk about just
| switching fishing email, to email helps.
|
| Or said differently, tell it that it's for a marketing email,
| and it will gladly write personalized outreach
| serviceberry wrote:
| While I broadly agree with the concerns about using LLMs for
| "commoditized", large-scale phishing, isn't the study a bit
| lacking? Specifically, "click through" is a pretty poor metric
| for success.
|
| If I receive a unique / targeted phishing email, I sure will
| check it out to understand what's going on and what they're
| after. That doesn't necessarily mean I'm falling for the actual
| scam.
| dwood_dev wrote:
| I hate the InfoSec generated phishing tests.
|
| They all pass DKIM, SPF, etc. Some of them are very convincing.
| I got dinged for clicking on a convincing one that I was
| curious about and was 50/50 on it being legit (login from a
| different IP).
|
| After that, I added an auto delete rule for all the emails that
| have headers for our phish testing as a service provider.
| Retr0id wrote:
| It's worth noting that "success" here is getting the target to
| click a link, and not (for example) handing over personal
| information or credentials.
| webdevladder wrote:
| I believe I was the target of employment-flavored spear phishing
| a few months ago. Could have been a researcher like the OP.
|
| - 3 new email chains from different sources in a couple weeks,
| all similar inquiries to see if I was interested in work (I
| wasn't at the time, and I receive these very rarely)
|
| - escalating specificity, all referencing my online presence, the
| third of which I was thinking about a month later because it hit
| my interests squarely
|
| - only the third acknowledged my polite declining
|
| - for the third, a month after, the email and website were
| offline
|
| - the inquiries were quite restrained, having no links, and only
| asking if I was interested, and followed up tersely with an open
| door to my declining
|
| I have no idea what's authentic online anymore, and I think it's
| dangerous to operate your online life with the belief that you
| can discern malicious written communications with any certainty,
| without very strong signals like known domains. Even realtime
| video content is going to be a problem eventually.
|
| I suppose we'll continue to see VPN sponsorships prop up a
| disproportionate share of the creator economy.
|
| In other news Google routed my mom to a misleading passport
| renewal service. She didn't know to look for .gov. Oh well.
___________________________________________________________________
(page generated 2025-01-05 23:00 UTC)