[HN Gopher] Thanksgiving 2023 security incident
___________________________________________________________________
Thanksgiving 2023 security incident
Author : nomaxx117
Score : 268 points
Date : 2024-02-01 19:59 UTC (3 hours ago)
(HTM) web link (blog.cloudflare.com)
(TXT) w3m dump (blog.cloudflare.com)
| jshier wrote:
| Fascinating and thorough analysis! I guess if you think an
| account is unused, just delete it!
| phyzome wrote:
| Probably safer to rotate the credentials and then schedule it
| for deletion later. Then if you discover it _wasn 't_ unused
| after all, you have an easier recovery... :-)
| sebmellen wrote:
| The most surprising part of this is that Cloudflare uses
| BitBucket.
| Cthulhu_ wrote:
| How so? It integrates well with the other Atlassian products
| they use.
| infecto wrote:
| Maybe but maybe not. I don't like Bitbucket but there are a
| number of large companies where they worry about using services
| owned by competitors in one of their verticals.
| kccqzy wrote:
| Bitbucket doesn't have to be a service. It can be an old-
| fashioned downloaded software that you install on your own
| machines. Not everything is SaaS.
| infecto wrote:
| Not sure what you mean? If you are alluding to the OP that
| said it was surprising...I don't think he found it
| suprising they they use Bitbucket over Mercurial. I think
| its safe to assume he meant bitbucket over a Github.
|
| In the git universe there is a pretty short list of
| services, locally or hosted that you would probably use as
| an entity as large as cloud flare.
| toyg wrote:
| Integrates with Jira and the rest of Atlassian's stuff, and
| it's just another git server at the end of the day.
| sevg wrote:
| > Even though we believed, and later confirmed, the attacker had
| limited access, we undertook a comprehensive effort to rotate
| every production credential (more than 5,000 individual
| credentials), physically segment test and staging systems,
| performed forensic triages on 4,893 systems, reimaged and
| rebooted every machine in our global network including all the
| systems the threat actor accessed and all Atlassian products
| (Jira, Confluence, and Bitbucket).
|
| > The threat actor also attempted to access a console server in
| our new, and not yet in production, data center in Sao Paulo. All
| attempts to gain access were unsuccessful. To ensure these
| systems are 100% secure, equipment in the Brazil data center was
| returned to the manufacturers. The manufacturers' forensic teams
| examined all of our systems to ensure that no access or
| persistence was gained. Nothing was found, but we replaced the
| hardware anyway.
|
| They didn't have to go this far. It would have been really easy
| not to. But they did and I think that's worthy of kudos.
| readyplayernull wrote:
| > The manufacturers' forensic teams examined all of our systems
| to ensure that no access or persistence was gained. Nothing was
| found, but we replaced the hardware anyway.
|
| Aha, the old replace-your-trusted-hardware trick.
| zitterbewegung wrote:
| Manufacturers have had security vulnerabilities for hardware
| to the point that the firmware on device couldn't be trusted
| to be replaced so they said to get new hardware so it's not a
| bad strategy.
| AzzyHN wrote:
| In a corporate environment, standard procedure when an
| employee's computer gets infected is to re-image it. Even if
| it was a stupid virus that was immediately caught, the
| potential risk of undetected malware running amuck is just
| too high.
|
| Now imagine, instead of Steve from HR's laptop, it's one of
| Cloudflare's servers.
| syncsynchalt wrote:
| Honestly I wish we'd had an excuse/reason to do an org-wide
| prod creds refresh like this at some places I've been.
|
| You find some scary things when you go looking for how exactly
| some written-by-greybeard script is authenticating against your
| started-in-1990s datastore.
| barkingcat wrote:
| I think they did have to do that far though.
|
| Getting in at the "ground floor" of a new datacentre build is
| pretty much the ultimate exploit. Imagine getting in at the
| centre of a new Meet-Me room
| (https://en.wikipedia.org/wiki/Meet-me_room) and having
| persistent access to key switches there.
|
| Cloudflare datacentres tend to be at the hub of insane amounts
| of data traffic. The fact that the attacker knew how valuable a
| "pre-production" data centre is means that cloudflare probably
| realized themselves that it would be a 100% game over if
| someone managed to get a foot hold there before the regular
| security systems are set up. It would be a company ending event
| if someone managed to install themselves inside a data centre
| while it was being built/brought up.
|
| Also remember, at the beginning of data centre builds, all
| switches/equipment have default / blank root passwords
| (admin/admin), and all switch/equipment firmware are old and
| full of exploits (you either go into each one and update the
| firmware one by one or hook them up to automation for fleet
| wide patching) Imagine that this exploit is taking place before
| automation services had a chance to patch all the firmware ...
| that's a "return all devices to make sure the manufacturer
| ships us something new" event.
| vasco wrote:
| What I think they meant is customers would keep paying them.
| And they are right, one just has to look at Okta, Solarwinds
| and other providers that have been owned, not done half of
| this and somehow are still in business. Everyone whistles to
| the side and pretends they shouldn't switch vendors, rotate
| all creds, cycle hardware, because it saves lots of work and
| this stuff falls under "reasonable oopsie" to the general
| public, when in fact there should be rules about what to do
| in the event of a breach that should be much stricter. So
| they do some partial actions to "show work" in case of
| lawsuits and keep going. The old engineers leave, new ones
| come in, and now you have systems who are potentially owned
| for years to come.
|
| It takes some honesty and good values by someone in the
| decision-making to go ahead with such a comprehensive plan.
| This is sad because it should be tablestakes, as you say
| correctly, but having seen many other cases, I think although
| they did "the expected", it's definitely above and beyond
| what peers have done.
| swyx wrote:
| do manufacturers share some of the cost of this kind of
| security related return or is this a straight up "pay twice
| for the same thing" financial hit?
| eastdakota wrote:
| We have very good relations with our network vendors (in
| this case, Cisco, Juniper, and Arista). The CEOs of all of
| them 1) immediately got on a call with me late on a
| weekend; 2) happily RMAed the boxes at no cost; and 3) lent
| us their most senior forensics engineers to help with our
| investigation. Hat tip to all of them for first class
| customer service.
| swyx wrote:
| shows how much they value you as a partner and i'm sure
| they appreciate your overall business.
|
| thanks Matthew! love the transparency and dedication to
| security as always. really sucks to have this be
| continuing fallout from Okta's breach. wish large scale
| key rotation was more easily automatable (or at least as
| a fallback, there should be a way to track key age on
| clientside? so that old keys stick out like a sore
| thumb). i guess in the absence of industry standard key
| rotation apis someday you might be able to "throw AI at
| it".
| tgsovlerkhgsel wrote:
| > It would be a company ending event if someone managed to
| install themselves inside a data centre while it was being
| built/brought up.
|
| It wouldn't. Most people like to assume the impact of
| breaches to be what it _should_ be, not what it actually is.
|
| Look at the 1-year stock chart of Okta and, without looking
| up the actual date, tell me when the breach happened/was
| disclosed.
| mschuster91 wrote:
| > Look at the 1-year stock chart of Okta and, without
| looking up the actual date, tell me when the breach
| happened/was disclosed.
|
| The problem with this is that while _security minded_
| people know what Okta is and why to stay the fuck away from
| handing over your crown jewels to a SaaS company is
| warranted, C-level execs _don 't care_. They only care
| about their golf course or backroom deal friends and about
| releasing PR statements full of buzzwords like "zero
| trust", "AI based monitoring" and whatever.
|
| The stock markets don't care either, they only look at the
| financial data, and as long as there still are enough
| gullible fools signing up, they don't care and stonk goes
| up.
| nolok wrote:
| > It would be a company ending event
|
| Given they got out of cloudbleed without any real damage let
| alone lasting damage, I disagree.
|
| (I don't disagree with your point about how bad of a problem
| this would be, I'm just insisting that security failure is
| not taken seriously at all by anyone)
| Hrundi wrote:
| I don't remember any companies that ended thanks to
| cloudbleed, but I'd be happy to be proven wrong
| chx wrote:
| Presuming taviso is not exaggerating and why would he CF's
| reply to cloudbleed was ... not quite nice.
|
| https://twitter.com/taviso/status/1566077115992133634
|
| > True story: After cloudbleed, cloudflare literally
| lobbied the FTC to investigate me and question the legality
| of openly discussing security research. How come they're
| not lobbying their DC friends to investigate the legality
| KF?
|
| For those not familiar with the history , this tweet
| started the cloudbleed disclosure to cloudflare:
|
| https://twitter.com/taviso/status/832744397800214528
|
| > Could someone from cloudflare security urgently contact
| me.
|
| This followed: https://blog.cloudflare.com/incident-report-
| on-memory-leak-c...
| sneak wrote:
| > _Imagine getting in at the centre of a new Meet-Me room and
| having persistent access to key switches there._
|
| This wouldn't get you much. We already assume the network is
| insecure. This is why TLS is a thing (and mTLS for those who
| are serious).
| noizejoy wrote:
| > We already assume the network is insecure.
|
| Maybe naively, I wish this assumption became universal.
| sophacles wrote:
| I suspect "we" is a much smaller group than you imagine.
| I've gotten pcaps from customers as recently as this year
| that include unencrypted financial transaction data. These
| were captured on a router, not an end host, so the traffic
| was going across the client's network raw.
| tptacek wrote:
| This is why old secops/corpsec security hands are so religious
| about tabletop exercises, and what's so great about
| BadThingsDaily+ on Twitter. Being prepared to do this kind of
| credential rotation takes discipline and preparation and, to be
| frank, most teams don't make that investment, including a lot
| of really smart, well-resourced ones.
|
| If Cloudflare is in a position where their security team can
| make a call to rotate every secret and reimage every machine,
| and then that happens in some reasonable amount of time, that's
| pretty impressive.
|
| + https://twitter.com/badthingsdaily?lang=en
| akira2501 wrote:
| It'd be more impressive if they actually got all the
| credentials.
|
| It's good that you think you can absorb a complicated
| security task, it's useless if you have no way to test or
| verify this action.
| swyx wrote:
| yes but this is a nice #2. not many fortune 500s would 1)
| even know they were breached and 2) if they were breached,
| have the breach be so contained.
| schainks wrote:
| Having seen the small number of DEFCON talks that I've seen, I
| would have absolutely gone that far.
| orenlindsey wrote:
| Cloudflare is showing how to correctly respond to attacks.
| Other companies should take note.
| ldoughty wrote:
| The nuclear response to compromise should be the standard
| business practice. It should be exceptional to deviate from it.
|
| If you assume that they only accessed what you can prove they
| accessed, you've left a hole for them to live in. It should
| require a quorum of people to say you DON'T need to do this.
|
| Of course, this is ideal world. I'm glad my group is afforded
| the time to implement features with no direct monetary or user
| benefit.
| marcinzm wrote:
| > we were (for the second time) the victim of a compromise of
| Okta's systems
|
| I'm curious if they're rethinking being on Okta.
| Icathian wrote:
| The challenge being, who else could possibly handle
| Cloudflare's requirements? I imagine the next step is to build
| their own, and that's obviously not an easy pill to swallow.
| amluto wrote:
| Why not? Cloudflare already operates a system that can help
| customers to require SSO for access to their services -- why
| not try to capture more of that vertical by becoming an IdP?
| whalesalad wrote:
| They already run their own zero trust infrastructure for
| customers, kinda surprised they are not dogfooding it.
| https://www.cloudflare.com/plans/zero-trust-services/
| mikey_p wrote:
| Did you read the article?
|
| They are using zero trust and explained that it's why the
| scope of the security incident was extremely limited.
| jgrahamc wrote:
| We use our Zero Trust stuff extensively. In fact, we built
| it for ourselves initially.
| tomschlick wrote:
| They are, but they don't have management for user accounts,
| 2fa, etc. You setup a connection to something like Okta,
| Google Apps, O365, SAML, etc to be your persistent user db
| and cloudflare just enforces it.
|
| I wouldn't be surprised if they are working on first party
| IAM user support though.
| margalabargala wrote:
| There are good reasons not to dogfood critical services
| like that; it can make recovering from unexpected issues
| much harder if you introduce mutual dependencies.
|
| For example, if Slack devops team were to exclusively
| communicate over Slack, then a Slack outage would be much
| harder to resolve because the team trying to fix it would
| be unable to communicate.
| BytesAndGears wrote:
| My company will only give us new laptops that are preinstalled
| with Okta's management system.
|
| I am grandfathered in to an old MacBook that has absolutely no
| management software on it, from the "Early Days" when there was
| no IT and we just got brand new untouched laptops.
|
| They offered me an upgrade to an M1/M2 pro, but I refused,
| saying that I wasn't willing to use Okta's login system if I
| have my own personal passwords or keys anywhere on my work
| computer.
|
| Since that would hugely disrupt my work, I can't upgrade. Maybe
| I can use incidents like this to justify my beliefs to the IT
| department...
| verve_rat wrote:
| Why do you need personal passwords on your laptop to do your
| work? I'm not understanding this.
| BytesAndGears wrote:
| Fair question, but I use a lot of things that are varying
| degrees of helpful for my work:
|
| * personal ChatGPT and copilot subscriptions, since company
| doesn't pay for these
|
| * Trello account for keeping track of my todo list
| (following up with people, running deploys)
|
| * Obsidian for keeping notes, as a personal knowledge-base
| (things like technologies and reminders)
|
| * Apple account for music, copy/paste, sharing photos from
| my travel with coworkers, synching docs related to my work
| visa and taxes
|
| * Personal slack login for communicating with my partner in
| our private server
|
| * personal GitHub account credentials for synching my
| private dotfiles repo with my neovim config. basically
| can't work without my dotfiles, but I could theoretically
| email these to myself or something, to prevent this one.
|
| And sure, I could be stubborn and not use any of this, but
| I'd be way less productive and kinda miserable.
| MichaelZuo wrote:
| Why don't you just do those on a second, personal,
| laptop?
|
| Does your workplace restrict you from bringing it in?
| BytesAndGears wrote:
| Convenience, I suppose... and that doesn't solve all of
| the issues (eg Copilot)
|
| I'm fine with it because I know there's no management
| software on this laptop, but yeah it's a totally
| different story if I had to use a newer one with SSO and
| management software
| jen729w wrote:
| I've been in the same situation. With two laptops you
| lose the ability to, say, send email directly to your
| task system.
|
| It's really easy to say 'don't use your personal stuff at
| work', but when work is some locked-down behemoth whose
| view of productivity software is 'just use Office', and
| you're really trying to be better at your job, using your
| own tools can be the only solution.
|
| And in my situation, yeah, they didn't want you bringing
| things in. I worked in a secure area.
| pests wrote:
| Then you need to let the employer see your lack of
| productivity when you are limited by the locked-down
| system.
|
| Finding solutions to work around the systems, on your own
| time and dime, only hurts in the long run.
|
| They think everything is fine. Nothing will ever get
| fixed. Voice these concerns.
| Xeyz0r wrote:
| Why carry a second laptop when you can log in wherever
| you need to on your work laptop? It's easier for me to
| store all my passwords in a password manager and log in
| to the websites I need from my work laptop.
| hinkley wrote:
| * Jetbrains
|
| * Stack Overflow
|
| * Job Search sites
|
| I don't remember if Jetbrains needs a password to get to
| personal licenses, but they definitely do to use their
| bug database. I suspect they're not the only one.
|
| Letting other people blow off steam can be an act of
| self-preservation. Insisting that people only ever do
| 100% work things at work or on work hardware slightly
| raises your low-but-never-zero chances of being murdered
| by coworkers. Or less ironically, hilariously intense
| bridge-burning activities.
|
| Also most of this conversation is happening during work
| hours so I think we can infer that grandparent is being a
| little hypocritical.
| _boffin_ wrote:
| Let me get this straight... you're taking privileged
| company information and transferring it to personal...
|
| I'm now understanding how people get sued when going from
| company to company.
| BytesAndGears wrote:
| Certainly nothing privileged! Moreso just reminders about
| "follow up with person x" and that kind of thing
| _boffin_ wrote:
| Let me ask you one follow up question: if you and the
| company had a disagreement of sorts and they examined
| your activity, would you believe they find nothing that
| they'd deem privileged?
| sneak wrote:
| > _personal Trello account for keeping track of my todo
| list._
|
| > _personal Obsidian for keeping meeting notes, and
| recording conversations as a personal knowledge-base_
|
| I'm not a lawyer, but I'm pretty sure these could subject
| a _lot_ of your other personal data to potential subpoena
| should your employer get sued by a sufficiently
| determined attacker.
|
| Don't cross the streams.
| BytesAndGears wrote:
| Ooh actually that's the most compelling reason I've
| heard. I think I might actually separate some of those
| accounts with this reasoning.
| marksomnian wrote:
| Also it's a violation of Obsidian's license:
|
| > Obsidian is free for personal and non-profit use.
| However, if you use Obsidian for work-related activities
| that generate revenue in a company with two or more
| people, you must purchase a commercial license for each
| user. Non-profit organizations are exempt from this
| requirement.
|
| https://obsidian.md/license
| kccqzy wrote:
| You use so many personal accounts for work that it's
| unfathomable for me. If some hacker manages to hack into
| your account and find so much valuable information about
| your work, your work is going to be mightily pissed about
| it. Imagine you are working on an upcoming product launch
| and the attacker used your personal account to leak the
| launch. Or imagine they just decided to leak your
| company's internal source code. Or imagine they simply
| use the technical information in your personal attacks to
| steal user data (even Cloudflare says they worry about
| this: "Our aim was to prevent the attacker from using the
| technical information about the operations of our network
| as a way to get back in").
|
| You are making your work take on an extraordinary risk in
| hiring you.
| deathanatos wrote:
| The parent's view does seem a bit extreme, but there is
| always some overlap. Whatever HR system you have is going
| to be in a weird area of personal/employee overlap, as
| it'll need to have a password that your personal life has
| access to. (As tax documents, pay stubs, benefits stuff,
| etc. all impact the "personal" side of one's life. E.g., I
| need to store -- in my personal archives -- the years W-2.)
|
| Also, people just do things for convenience. (Although I
| tend to pipe these passwords over an SSH connection, so
| that they're not resident on the work laptop. Though there
| is a good argument to be had about me permitting my work
| laptop SSH access to my personal laptop. From a technical
| standpoint, my employer could hack/compromise my personal
| laptop. From a legal and trust standpoint, I presume they
| won't.)
| sneak wrote:
| It's not extreme at all, it's the bare minimum that
| professionals do.
|
| Absolutely _none_ of my personal stuff ever touches a
| corporate machine. Ever. I wouldn 't even log in to the
| W2 downloading app as an employee from the work machine.
|
| Granting work ssh keys access to your personal machine is
| crazy; if your work machine gets compromised, they steal
| your entire personal system's home directory too. Why
| would you unnecessarily expand the blast radius of a
| compromise like this?
| pbhjpbhj wrote:
| >From a technical standpoint, my employer could
| hack/compromise my personal laptop. From a legal and
| trust standpoint, I presume they won't.)
|
| You trust all personnel with access to your employers
| network?
|
| What's more surprising is that they trust you to setup
| adhoc ssh connections to arbitrary endpoints; unless
| you're the person in charge of network security?
|
| Would anyone notice if you, or an intruder, dumped
| terabytes of data over that connection?
|
| I don't work in IT but this just doesn't feel right to
| me.
| yjftsjthsd-h wrote:
| > if I have my own personal passwords or keys anywhere on my
| work computer.
|
| Well... don't do that? Why would you ever have personal
| anything on a work computer?
| Symbiote wrote:
| A Github account, for one possible example.
| pizzalife wrote:
| Well, why? It just seems risky. Everything you make on
| your work laptop / during work hours is typically owned
| by your employer. If your employer is paying you to
| contribute to OSS, don't use your personal github
| account. Just don't ever mix personal and company
| accounts on company hardware.
| electroly wrote:
| Note that if you do make a second account, at least one
| of them must be a paid account. A single person cannot
| have multiple free accounts and GitHub does not care if
| it's because one is for work; it's in the TOS.
| deathanatos wrote:
| This is why I use a separate Github account for work?
|
| (& then just rotate the credentials on it when you part
| ways with the employer.)
|
| Some of my co-workers even do a Github account per
| employment.
| _boffin_ wrote:
| That's me--a GitHub account per employer with employee
| email.
| _boffin_ wrote:
| Doesn't matter. No personal stuff on company devices.
|
| I just don't understand any rational otherwise.
| babypuncher wrote:
| So you just don't listen to music at work?
| _boffin_ wrote:
| Do you have a phone and headphones?
| yjftsjthsd-h wrote:
| Okay, I'll bite; what about a github account? You don't
| generally own code you write for an employer, so why
| would you be an personal repos from a company machine?
| (Likewise, there's generally no good reason for the
| company to have access to personal repos, so those
| security domains should never overlap)
| samcat116 wrote:
| > new laptops that are preinstalled with Okta's management
| system
|
| Okta doesn't make device management software, thats made by
| companies like Jamf. Okta can integrate with them but Okta
| isn't what manages your laptop at all.
|
| > I wasn't willing to use Okta's login system if I have my
| own personal passwords or keys anywhere on my work computer.
|
| Do not do this, its not a personal device.
| michaelt wrote:
| _> Do not do this, its not a personal device. _
|
| You think nobody's logged into their personal spotify on
| their work computer? All those guys wearing headphones in
| the office have brought in CDs to play in their laptop CD
| drives?
|
| And that business traveller away from their partner and
| kids for a week+ isn't going to video call them? Or watch
| some netflix in their hotel room in the evening?
|
| That's so unrealistic, you could write IT security policy
| for a Fortune 100 company :)
| jrockway wrote:
| Which "nation state" do we think this was?
| meowface wrote:
| For these kinds of attacks it's nearly always China, Russia,
| US, or sometimes Iran. 95% chance it's either China or Russia,
| here.
| 2OEH8eoCRo0 wrote:
| When has it been the US?
| AzzyHN wrote:
| We do a lot of hacking
| toyg wrote:
| Stuxnet?
| toyg wrote:
| Their response program being called "Code Red" is likely a
| hint.
| godzillabrennus wrote:
| China.
| jedahan wrote:
| The writeup contains indicators, including IP addresses, and
| the location of those addresses. In this case, the IP address
| associated with the threat actor is currently located in
| Bucharest, Romania.
| tomschlick wrote:
| No nation state is going to use IPs from their own country if
| they don't want to be caught. They will use multiple layers
| of rented VPS's with fake identities to pay for those
| resources.
| jrockway wrote:
| Yeah. I've dealt with definitely-not-nation-states before,
| and their pattern was to sign up for free/cheap CI services
| (CircleCI, Github Actions, that sort of thing) and launch
| their attacks from there. The VPS thing also sounds very
| very plausible to me, I figured there was a long tail, but
| until I was looking up every network that was attacking us,
| I really had no idea how deep the long tail goes. I now
| feel like half the world's side hustle is to rent a server
| that they never update and host a couple of small business
| websites there.
| bredren wrote:
| > I now feel like half the world's side hustle is to rent
| a server that they never update and host a couple of
| small business websites there.
|
| Do you mean people are offering build / host services for
| small biz, and leaving their servers in such a state they
| can be owned and used as jump points for intrusion?
|
| Reason I ask is long-hosted small business websites are
| sometimes established with the intent to legitimize some
| future unrelated traffic.
| CubsFan1060 wrote:
| M247 is commonly used by VPN providers (https://www.reddit.co
| m/r/PrivateInternetAccess/comments/8xwn...)
| lijok wrote:
| Which nation state has good enough employment protection laws
| that they can take weekends off while doing recon on a top
| value target?
| icepat wrote:
| Yes, they must have been a member of the Norwegian Foreningen
| Svartehattehackere. They are a very strong union.
| toyg wrote:
| Might be a coincidence. A certain nation-state is currently
| engaged in all-out war; the intruder might have been summoned
| to another, more urgent task.
| belltaco wrote:
| Great write up.
|
| > Over the next day, the threat actor viewed 120 code
| repositories (out of a total of 11,904 repositories
|
| > They accessed 36 Jira tickets (out of a total of 2,059,357
| tickets) and 202 wiki pages (out of a total of 14,099 pages).
|
| Is it just me or 12K git repos and 2 million JIRA tickets sound
| like a crazy lot. 15K wiki pages is not that high though.
|
| > Since the Smartsheet service account had administrative access
| to Atlassian Jira, the threat actor was able to install the
| Sliver Adversary Emulation Framework, which is a widely used tool
| and framework that red teams and attackers use to enable "C2"
| (command and control), connectivity gaining persistent and
| stealthy access to a computer on which it is installed. Sliver
| was installed using the ScriptRunner for Jira plugin.
|
| > This allowed them continuous access to the Atlassian server,
| and they used this to attempt lateral movement. With this access
| the Threat Actor attempted to gain access to a non-production
| console server in our Sao Paulo, Brazil data center due to a non-
| enforced ACL.
|
| Ouch. Full access to a server OS is always scary.
| Aeolun wrote:
| > Is it just me or 12K git repos and 2 million JIRA tickets
| sound like a crazy lot. 15K wiki pages is not that high though.
|
| I think my org has on the order of 3 repositories per dev? They
| seem to have 3200 employees, with what I assume to be a
| slightly higher rate of devs, so you'd expect around 6-7
| thousand?
|
| 2M Jira tickets is probably easily achieved if you create
| tickets using any automated process.
| trollied wrote:
| They might create a JIRA ticket for each customer support
| interaction. Would make sense.
| ummonk wrote:
| The number of repositories sounds really high. The number of
| tickets doesn't.
| spenczar5 wrote:
| 12k git repos can happen if the team uses github enterprise
| with forking internally.
|
| It can also happen in franken-build systems which encourage
| decoupling by making separate repos: one repo that defines a
| service's API, containing just proto (for example). A second
| repo that contains generated client code, a third with
| generated server code, a fourth for implementation, a fifth
| which supplies integration test harnesses, etc...
|
| Sound insane? It is! But its also how an awful lot of stuff
| worked at AWS, just as an example.
| sesm wrote:
| I can relate to this, it seems that code hosting providers
| push their users into having more repos with their CI
| limitations. I've noticed that with GitHub Actions, I assume
| Atlassian does the same.
| OJFord wrote:
| They probably have thousands of devrel/app engineer type
| example/demo/test repos alone - it doesn't say they're
| _active_.
|
| 2M tickets - in my 4.5y at present company we've probably
| averaged about 10 engineers and totalled 4.5k tickets.
| Cloudflare has been around longer, has many more engineers,
| might use it for HR, IT, etc. too, might have processes like
| every ticket on close opens a new one for the reporter to test,
| etc. It sounds the right sort of order of magnitude to me.
| pkkim wrote:
| At least 25% of that is a single Jira project consisting of
| formulaic tickets to capture routine changes in production, and
| a large portion of that is created by automated systems. There
| may be other such projects too.
|
| Source: former Cloudflare employee
| Aeolun wrote:
| Reading this 2 months after the fact feels a bit late, but I
| guess it's better for your stock price if these revelations
| happen with remediation already in hand?
|
| Since they didn't really have reason to believe my data was
| accessed, maybe that's ok. I know from firsthand experience how
| hard rotating all your credentials across the whole org is.
| Cthulhu_ wrote:
| The final security report was only released yesterday, and the
| amount of work they did to make sure all of their systems were
| secure after the incident was A Lot; two months is pretty quick
| for a project of that scale IMO.
| Aeolun wrote:
| Yes, but if after two months they'd found out that customer
| data _had_ been compromised, that would be a little late for
| me to do anything about it.
| nemothekid wrote:
| What do you expect them to do? It sounds like you are
| complaining that they weren't able to instantly ascertain
| if customer data had been compromised.
| eastdakota wrote:
| Had customer data been impacted we would have disclosed it
| immediately.
| htrp wrote:
| ^ eastdakota is part of the cloudflare mgmt team (CEO)
| BytesAndGears wrote:
| Writeups and actions like this from cloudflare are exactly why I
| trust them with my data and my business.
|
| Yes, they aren't perfect. They do some things that I disagree
| with.
|
| But overall they prove themselves worthy of my trust,
| specifically because of the engineering mindset that the company
| shares, and how serious they take things like this.
|
| Thank you for the blog post!
| Xeyz0r wrote:
| Nobody is perfect, but Cloudflare indeed inspires confidence.
| Especially thanks to cases where they don't hesitate to talk
| about the issue and how they resolved it. It's precisely these
| descriptions of such situations that demonstrate their ability
| to overcome any challenges.
| overstay8930 wrote:
| We're one of their larger enterprise customers and stuff like
| this makes it easy to get renewals approved easily, keeping
| engineers in the loop makes it such an easy sell.
| OJFord wrote:
| > The one service token and three accounts were not rotated
| because mistakenly it was believed they were unused.
|
| Eh? So why weren't they revoked entirely? I'm sure something's
| just unsaid there, or lost in communication or something, but as
| written that doesn't really make sense to me?
| stepupmakeup wrote:
| Rotating could have been manual and the person in charge wanted
| to save time. Stress could be a factor too.
| htrp wrote:
| blameless post mortem most likely
|
| Great call out too
|
| > Note that this was in no way an error on the part of AWS,
| Moveworks or Smartsheet. These were merely credentials which we
| failed to rotate.
| phyzome wrote:
| Betting they have a new item in their compromise runbook. :-)
| londons_explore wrote:
| Am I the only one who just sees a totally blank page?
|
| Viewing the HTML shows it's got an empty body tag, and a single
| script in the <head> with a URL of
| https://static.cloudflareinsights.com/beacon.min.js/v84a3a40...
| chankstein38 wrote:
| No, that's also what I see. I'm not sure why you're getting
| downvoted.
|
| EDIT: re-opened the link a few minutes later and now I see the
| post
| overstay8930 wrote:
| Happened to me on my iPhone too
| londons_explore wrote:
| So after the Okta incident they rotated the leaked credentials...
|
| But I think they should have put honeypots on them, and then
| waited to see what attackers did. Honeypots discourage the
| attackers from continuing for fear of being discovered too.
| fierro wrote:
| >The one service token and three accounts were not rotated
| because mistakenly it was believed they were unused.
|
| This odd to me - unused credentials should probably be deleted,
| not rotated.
| pbhjpbhj wrote:
| This smells weird, surely? I'd be looking at who chose not to
| rotate those particular credentials.
|
| 1: "what are these accounts?"
|
| 2: "oh they're unused, they don't even appear in the logs"
|
| 1: "we should rotate them"
|
| 2: "no, let's keep those rando accounts with the old
| credentials, the ones we think might be compromised ... y'
| know, for reasons"
|
| ?
| muzso wrote:
| > The threat actor searched the wiki for things like remote
| access, secret, client-secret, openconnect, cloudflared, and
| token. They accessed 36 Jira tickets (out of a total of 2,059,357
| tickets) and 202 wiki pages (out of a total of 14,099 pages).
|
| In Atlassian's Confluence even the built-in Apache Lucene search
| engine can leak sensitive information and this kind of access (to
| the info by the attacker) can be very hard to track/identify.
| They don't have to open a Confluence page if the sensitive
| information is already shown on the search results page.
| kccqzy wrote:
| > Analyzing the wiki pages they accessed, bug database issues,
| and source code repositories, it appears they were looking for
| information about the architecture, security, and management of
| our global network; no doubt with an eye on gaining a deeper
| foothold.
|
| For a nation state actor, the easiest way to accomplish that is
| to send one of their loyal citizens to become an employee of the
| target company and then have the person send back "information
| about the architecture, security, and management" of the target
| company.
|
| Fun (but possibly apocryphal) fact: more than a decade ago in a
| social gathering of SREs at Google, several admitted to being on
| the payroll of some national intelligence bureaus.
| toyg wrote:
| Not if such citizens are sanctioned. Code Red. Hint hint.
| elashri wrote:
| I think this probably was a name after the famous Code Red
| worm [1], not a reference to China.
|
| [1] https://en.wikipedia.org/wiki/Code_Red_(computer_worm)
| orenlindsey wrote:
| Cloudflare being compromised would be enormous. Something between
| 5 and 25% of all sites use CF in some fashion. An attacker could
| literally hold the internet hostage.
| htrp wrote:
| >They did this by using one access token and three service
| account credentials that had been taken, and that we failed to
| rotate, after the Okta compromise of October 2023. All threat
| actor access and connections were terminated on November 24 and
| CrowdStrike has confirmed that the last evidence of threat
| activity was on November 24 at 10:44.
|
| Okta hitting everywhere
___________________________________________________________________
(page generated 2024-02-01 23:00 UTC)