[HN Gopher] CrowdStrike global outage to cost US Fortune 500 com...
___________________________________________________________________
CrowdStrike global outage to cost US Fortune 500 companies $5.4B
Author : Terretta
Score : 133 points
Date : 2024-07-24 18:28 UTC (4 hours ago)
(HTM) web link (www.theguardian.com)
(TXT) w3m dump (www.theguardian.com)
| lnxg33k1 wrote:
| So, here's 10 bucks, we good?
| throwaway240403 wrote:
| Worse than than, 10$ gift card
| lawn wrote:
| I heard it had been withdrawn?
| metadat wrote:
| _CrowdStrike offers a $10 apology gift card to say sorry
| for outage_ - https://news.ycombinator.com/item?id=41058261
| (129 comments)
|
| According to the discussion in the thread, you're correct.
| Also, it was a $10 giftcard for .. uber eats. Where you
| can't get anything for less than ten bucks.
| lawn wrote:
| That's truly some dark comedy.
| rqtwteye wrote:
| These are deeply insulting. My company sometimes sends
| out $10 cards for DoorDash. To actually get something I
| would have to add at least another $10 myself.
|
| I wonder if Uber Eats and DoorDash give these out for
| free to companies as promotion. I bet most people who use
| the cards spend another $20 or more.
| ck45 wrote:
| Even worse, the vouchers got canceled and can't be
| redeemed.
| dylan604 wrote:
| I guess it's easier to buy X number of gift cards and
| allow that company to deal with the individuals rather
| than paying each customer $10 individually.
|
| Also, does the IT manager get that giftcard? Do they
| share it with the rest of the team? Does the CTO get the
| card and shares it with the rest of the C-suite. What's
| the proper way of handling that other than reject with a
| harsh laugh in their face at the offer?
| bluSCALE4 wrote:
| Or a year membership of geek squad.
| bdcravens wrote:
| Bankruptcy coming ....
| 7thaccount wrote:
| I'm curious if that will actually happen as these companies
| need to use something to check their compliance box.
| femiagbabiaka wrote:
| Correct. To move away there have to be alternatives. Who here
| is building the alternative?
| SoftTalker wrote:
| Yeah I think it will be quickly forgotten. Most people
| already have where I work. It was just another IT fail in a
| long line of them.
| dylan604 wrote:
| This was what I kept thinking about when it happened. Every
| job I've ever had has had days of some system or other
| being down for some amount of time. Hell, even WFH has put
| me offline when I've had extended power outage after a
| major storm. The only thing different about this was that
| it was all the companies at the same time because of one
| glitch. For all of the companies that did not use this
| system, it was just another day of the week.
| xyst wrote:
| There are competitors in this space. Palo Alto Networks comes
| to mind. Whether competitors have same issues as ClownStrike
| remains to be seen.
| AmericanChopper wrote:
| The two problems I see with this story are
|
| 1) These types of products cause incidents all the time,
| this was just a very high impact one that happened to
| affect everybody all at once.
|
| 2) Their product is very good compared to the competitors.
|
| All products in this space are black boxes, but CS is one
| of the least black-boxy, the alerts it produces are decent,
| the tooling is comes with is especially good (from an
| operator perspective), and the reporting it produces is
| exactly the sort of thing decision makers in big enterprise
| love to see.
|
| I doubt there's going to be much churn from this,
| definitely not an existential amount. As much as I
| personally can't stand the organisation, I think they
| absorbed most of the bad press on behalf of all the service
| providers they took offline.
| hnlmorg wrote:
| Unlikely. Big enterprises are slow to change and switching
| provider would be a _massive_ change.
|
| They're more likely to investigate running two platforms
| simultaneously or, more likely, talk to competitors purely as a
| bargaining chip to negotiate a bigger discount for the next
| renewal.
| dylan604 wrote:
| You can sue your provider and still continue to use that
| provider. Look at the Apple vs Samsung lawsuits as an example
| hnlmorg wrote:
| The Apple vs Samsung situation is very different to this
| but I do take your point.
|
| The end result is still the same though, CrowdStrike will
| lose a lot of income and confidence but they're not going
| out of business.
|
| Frankly, even if they were to, I'm certain they'd end up
| getting bailed out anyway. But I can't see it getting to
| the point to begin with.
| chasd00 wrote:
| yeah that's my gut feeling. Also, no one at these enterprises
| are to blame they can all point fingers at CloudStrike and
| abuse their account reps. That way they don't have to put in
| the work to actually switch vendors while appearing like
| they're "on it". I bet they all get pretty nice discounts on
| renewal like you said but that will be it.
| CivBase wrote:
| They were valued near $100B prior to this incident and even now
| their valuation is north of $60B. Even if they were held
| financially responsible for all $6B in damages - which is
| definitely not going to happen - that doesn't seem like a
| company-ending scenario.
|
| Their biggest problem is obviously going to be with customer
| retention, but there are huge technical and regulatory hurdles
| their customers would have to go through to switch to a
| competitor. I'm sure many of their customers will accept this
| as an isolated incident and be quick to accept CrowdStrike's
| assurances that this won't happen again.
|
| I think Delta Airlines is probably in more trouble right now
| than CrowdStrike.
| d1sxeyes wrote:
| As far as I can tell, CS is still up 75% over 12 months.
| dheera wrote:
| The real problem is their stock crashed, so their best
| engineers will probably leave, and then it will become
| increasingly worse in a feedback cycle until possible eventual
| bankruptcy.
|
| Ideally the stock price should be pushed up to attract better
| engineers to go in and fix shit. If stockholders agree to bid
| up the stock 100% YoY, hell, even I'd look for a job there, and
| help fix shit in return for some juicy RSUs.
|
| If you take away their funding, you can only expect worse in
| the future.
| chgs wrote:
| Crashed? Since the start of the year CRWD is up 5%
| dheera wrote:
| Since the beginning of time everything is up 10000000%
|
| It crashed in the last week. That's what matters.
| cjpearson wrote:
| But since most engineers probably started before last
| week, wouldn't they be still up quite a bit? Even with
| the crash it's up 75% over the past year.
| dheera wrote:
| They just lost a lot though, losing incentive to stay.
| But they provide a public good, so it would help for the
| public to bid the stock up to get the employees to stay,
| fix things, and vest it rather than ditching the company.
| ridgeguy wrote:
| Lawsuits may crater CrowdStrike. I'll be surprised if they
| carry adequate insurance against this large a screwup.
| btbuildem wrote:
| I think what's more likely is their stock will drop for a
| while, this will blow over, and everyone will continue to pay
| their licence fees as if nothing ever happened. Don't
| underestimate the inertia, ineptitude, and resistance to change
| that permeates the upper echelons of large corps.
| xedrac wrote:
| Looks good on a resume:
|
| - Wrote code responsible for $5.4 billion
| solardev wrote:
| How does a company this big not have automated tests for their
| config files, and not have gradual/staggered rollouts for their
| deployments?
|
| Is there some good reason for this approach (need to get config
| updates into the wild as quickly as possible to combat zero-days
| or zero-hours?) or was this just a massive oversight?
|
| Side rant... their postmortem took forever to get to the point,
| first explaining all their jargon and product names. Makes me
| really appreciate the Cloudflare ones.
| blackeyeblitzar wrote:
| Often this is not because the team doesn't know about these
| things but because they have low staffing or other priorities
| or deadlines. This event looks to me like company rot that can
| be laid at the CEO's feet
| lupusreal wrote:
| My bet is they have some normal process for updates that has
| testing but that process is only enforced by policy, not code,
| and somebody simply decided it was a waste of their time.
| geodel wrote:
| Yeah, management should sternly tell _All code / config must be
| tested before deployment to prod_. Millions of companies have
| issued this order and after that they are running free of
| problem for decades.
|
| It is so straightforward and it always works.
| toomuchtodo wrote:
| Without regulation, best practices are simply opinions and
| suggestions. "Please do" is insufficient for critical
| infrastructure. See: financial infra regulatory apparatus.
|
| Incentives, outcomes, the usual.
| specialist wrote:
| Regulations bad. The invisible foot of Freedom Markets(tm)
| will fix exactly these kinds of market failures. Rationally
| speaking.
| devoutsalsa wrote:
| How many conversations have you had with people at work about
| how something was a bad idea, only to unsuccessfully avoid it?
| jvanderbot wrote:
| And conversely, how many good ideas have you proposed to have
| them put on "next quarter"'s schedule?
| rfoo wrote:
| > not have automated tests for their config files
|
| They very likely have automated tests. However, what if bug
| only triggers 90% of the time and you hit the lucky 10% during
| automated tests? Of course you can run tests 100 times but...
| is this a common practice? Moreover, we have both code and
| anecdotal evidences that the bug may indeed happen randomly.
| Tavis Ormandy posted a rough analysis of the crash context:
| https://x.com/taviso/status/1814762302337654829. It looks like
| the crash is caused by first checking if an uninitialized
| pointer is NULL, and if not, dereferencing it. If the
| uninitialized leftover data just happened to be zero, no crash
| happens.
|
| And anecdotally, we saw people reporting that repeatedly
| rebooting their machines for 15+ times fixed the problem for
| them - because eventually you got lucky and in a boot it didn't
| happen and CrowdStrike managed to update itself to not crash.
|
| > not have gradual/staggered rollouts for their deployments
|
| No idea. Maybe their poor reliability guy got overrided by
| another team, like "how dare you delaying our important
| definition update? we're racing with threat actors!". I hope
| they learned their lesson.
| pantalaimon wrote:
| The crash was triggered by a config file that just contained
| null bytes as payload
| rfoo wrote:
| This is a false rumor. Please stop spreading
| misinformation.
| zelphirkalt wrote:
| Running tests 100 times only helps, if you have some
| sufficiently randomized input data, so that the issue can
| happen.
| chasd00 wrote:
| hey they're moving fast and breaking things..
| 51Cards wrote:
| I am wondering if the update passed the test farm just fine,
| but when the file was moved to the update distribution system
| that's when the issue happened. The file copied as all nulls
| and there was no validation check that the file posted ok.
| Compounded by no validation check on the file after downloading
| by the end system. Compounded by not having a staged roll-out
| process for updates.
| mberning wrote:
| I think this is going to be a huge boon for dell. We had so many
| older computers that got completely hosed. Lots of Latitude 5400s
| died completely. All will need replacements.
| seattle_spring wrote:
| How did it hose them completely? I thought the problem is
| easily fixed by removing the offending config?
| Thaxll wrote:
| Never heard of servers with bad hardware that never rebooted
| for years?
| leetbulb wrote:
| Many people here have only ever used cloud servers.
| umanwizard wrote:
| The Latitude 5400 is a laptop.
| SoftTalker wrote:
| How? The fix is to just remove the one bad "Channel File." Are
| there machines where that does not resolve the problem?
| nerdjon wrote:
| Hardware failure hosed or just needing a re-image hosed?
|
| Unless the hardware was already near failure I don't see how
| this could cause hardware failure. The worst case scenario was
| the machine just constantly rebooting but after 3 (I think,
| somewhere around that number) it should have launched into
| WindowsRE.
| nottorp wrote:
| It's my understanding that CrowdStrike customers buy that thing
| to check a box in some security audit, not because it provides
| any other benefit.
|
| Let's blame bullshit compliance?
| wepple wrote:
| What's that understanding based on?
| nottorp wrote:
| All previous comments on HN about the incident... I've seen
| absolutely no one praising the thing as a security solution
| but a lot of people posting that it's bought to pass audits.
| gruez wrote:
| >not because it provides any other benefit.
|
| It probably does provide benefits against some clueless intern
| in accounting downloading a macro-enabled excel file that has a
| malware enabled.
| LASR wrote:
| I'm pretty certain CS has contracts that limit their liabilities
| in events like this.
|
| Probably a refund is all they'll be on the hook for.
|
| Sadly, damage done like this is just chalked up to an accident,
| and swept under the rug.
| Terretta wrote:
| > _contracts that limit their liabilities... refund is all
| they'll be on the hook for_
|
| By cashing in this $10 Uber Eats coupon you agree to hold
| harmless...
|
| - https://news.ycombinator.com/item?id=41058261
|
| - https://techcrunch.com/2024/07/24/crowdstrike-
| offers-a-10-ap...
| mirashii wrote:
| "A few people on twitter are saying this thing happened. We
| didn't actually talk to them, we didn't look at the emails
| and verify their authenticity ourselves, we just trusted some
| twitter screenshots and wrote a blogspam article stating it
| as truth.
|
| We put absolutely no critical thought into whether this was a
| likely thing, and we completely ignored the many government
| and media reports that are credibly sourced which state that
| there are known phishing scams and other threat actors trying
| to capitalize on this incident."
|
| I highly doubt this is something that Crowdstrike actually
| did.
|
| Edit: Amazingly they did, the article has been updated with a
| statement. Amazingly stupid all around.
| vb-8448 wrote:
| It depends on the actual root cause, gross negligence won't
| save them, regardless of what they put in the contracts.
|
| From my point of view, one of the greatest problem for them is
| that they bypassed customers deployment policies.
| sithadmin wrote:
| >one of the greatest problem for them is that they bypassed
| customers deployment policies
|
| Caveat emptor. Falcon and other similar security products
| often push updates at-will, and they're fully transparent
| about this if you actually read the contract terms and
| understand the vendor's approach to operations. I have worked
| with many clients that elect not to use such tools in certain
| sensitive environments, specifically to mitigate the risk of
| being impacted by something like CrowdStrike's 7/19 event.
| abnercoimbre wrote:
| Do we have more insight into the nature or reasons behind the
| bypassing?
| gruez wrote:
| >From my point of view, one of the greatest problem for them
| is that they bypassed customers deployment policies.
|
| Do you really want to wait until for the
| weekly/monthly/quarterly deployment window to deploy a
| detection update for a 0day, or a new type of malware?
| vb-8448 wrote:
| Well, at least it's up to me to decide, not CS.
| gruez wrote:
| You're free to choose an EDR vendor that allows you to
| defer definition updates. Remember, this is enterprise
| sales for multi-billion dollar companies, so the usual
| excuse of "take it or leave it" doesn't really apply.
| logicchains wrote:
| Hopefully this will make BigCos think twice about forcing their
| employees to fill their computers with "security" malware that
| slows productivity to a crawl.
| disruptiveink wrote:
| The value proposition of Crowdstrike is exactly that: something
| that you can deploy to tick the regulatory checkbox of "we have
| endpoint protection from a reputable company everywhere"
| without consuming outrageous system resources.
|
| That's why they have so many enterprise customers. They're the
| only game in town that won't slow down your servers arbitrarily
| while still convincing an auditor that you do have an
| antivirus.
|
| Too bad they also crash your whole system every now and then.
| leandrod wrote:
| Free just got cheaper.
|
| Yeah, I know using free software isn't a panacea. Still it would
| be a step in the right direction, plus I could not refrain from
| the cheap shot at M$ Windows.
| xnyan wrote:
| Cloudstrike customers voluntarily agreed to allow Cloudstrike
| to push kernel drivers. What should Microsoft have done to
| prevent this?
| ck45 wrote:
| Move Windows Defender into user space and enforcing the same
| for all security software.
| IcyWindows wrote:
| This has nothing to do with how Defender works.
|
| Crowdstrike shipped a driver that they marked as a
| mandatory boot driver. The Windows OS could have had more
| recovery options otherwise.
| gruez wrote:
| This isn't your personal computer/homelab where you can get
| away with using common sense antivirus or even windows
| defender. Software like crowdstrike are often used in
| industries where they're mandated to install such software for
| compliance reasons (eg. PCI-DSS). Even if you were using linux
| you'd still need to install it, and crowdstrike previously had
| issues with their linux agent. It was just uncommon enough that
| it didn't hit the news.
| chgs wrote:
| This is why the important thing is diversity. The more
| diverse your ecosystem they less likely you are to suffer a
| catastrophic failure
|
| If half your tills are windows/defender and half
| linux/crowdstrike then half your tills are going to be
| working.
| gruez wrote:
| Except that seems like a maintenance nightmare day to day.
| There's bugs in the linux version but not the windows
| version, not to mention having to write two sets of
| software. Imagine having to get your app's prod to work on
| both windows AND linux.
| midtake wrote:
| Agreed. It should be deployed entirely on Linux. Rip and
| rebuild is much easier on Linux. Using Windows as a
| server should be seen as a dark pattern in 2024.
|
| For EMS, hospitals, Windows makes sense on the server
| because they don't know any better. For anyone remotely
| technologically competent, Windows shouldn't even be
| considered an option other than as workstations. Linux on
| the server is the only way and no one can convince me
| otherwise.
| gruez wrote:
| >Using Windows as a server should be seen as a dark
| pattern in 2024.
|
| >Linux on the server is the only way and no one can
| convince me otherwise.
|
| Now meet the sysadmin that thinks the same, but for
| windows for clients. At the risk of overgeneralizing,
| people are only for "diversity" when it means supporting
| their preferred underdog platform (eg. linux desktop).
| When they're the dominant incumbent it's suddenly "dark
| pattern", "they don't know any better" and "no one can
| convince me otherwise".
| BillSaysThis wrote:
| $5.4B seems way too low given the number of flights Delta
| cancelled and will be on the hook to refund.
|
| https://www.washingtonpost.com/transportation/2024/07/23/del...
| gruez wrote:
| Delta's revenue is only $58.05 billion in 20213. I'm not sure
| how a day or two of canceled flights is going to be anywhere
| near $5.4B.
| hysan wrote:
| Delta only started to recover today, so it's more like 4-5
| days of canceled flights. Not saying it's a huge difference
| but 1-2 greatly underplays how bad it was for people flying
| Delta vs any other airline.
| magic_man wrote:
| Way more than delta flights were grounded. If you include all
| the hours lost by all those people and assign some monetary
| value it's probably more than that. Not only that, but all
| the hospitals. God forbid someone died because of it.
| Surprised crowd strike isn't bankrupt
| jvanderbot wrote:
| Crowdstrike's impact was more than Delta's cancelled flights?
| How did you arrive at the conclusion it was limited to Delta?
|
| Even still, Delta had a _really bad time_ recovering, and is
| still cancelling flights days later. It 's not just "a day or
| two". At a billion a week, with 30-40% cancellation rates,
| that's 300-400M just for this one customer. And that's _just_
| lost revenue. Imagine the extra costs: customer service
| complaints, hardware / IT restoration, extra wages for
| flight attendants working double duty to keep the remaining
| flights going. Madness.
|
| Even just in travel segment, how many hotels, car rentals,
| uber/lyft rides, etc were cancelled b/c of missed flights?
| How much do you think they _paid_ on top of the lost revenue
| to handle customer complaints, IT restoration, etc?
|
| The repair costs alone at a given airport must be staggering,
| as every terminal screen is BSOD and needs a tech to manually
| restore from bitlocker.
|
| https://www.cbsnews.com/news/delta-flight-cancellations-
| toda...
| gruez wrote:
| Well the comment I replied to didn't event try to quantify
| the total impact, and only mentioned delta flights being
| canceled. There might be a bazillion other ways it can add
| up to $5.4B, but "flights Delta cancelled and will be on
| the hook to refund" does not come anywhere close to
| justifying that number.
| moralestapia wrote:
| It's more like a week of downtime which totals to about a
| billion based on the data you brought in.
|
| So, just _one_ of the affected companies brings the total to
| $1B, wouldn 't you say $5B is actually a low estimate?
| jjav wrote:
| I would like to live in a world where crowdstrike is directly
| liable for every dollar lost and has to pay for all of it. Only
| then would these companies start to take quality seriously.
|
| Of course in our real world, they are unlikely to pay anything
| at all and just continue operating as-is.
| ghostly_s wrote:
| It's absolutely ludicrous the US airline industry still has
| no contingency plan for "a bad software update is pushed by a
| vendor to our Windows systems."
| SketchySeaBeast wrote:
| If the computers in a computer reliant massively
| distributed organization go down, what's the alternative?
| ThunderSizzle wrote:
| Computers shouldn't auto update. Updates need to be
| scheduled and monitored and tested.
| lazide wrote:
| Every CTO's easy answer - cool, we'll just only do them
| once every 5 years then.
|
| With modern tech stacks, we're talking hundreds of
| updates a quarter. If not more.
| whoknowsidont wrote:
| >I would like to live in a world where crowdstrike is
| directly liable for every dollar lost and has to pay for all
| of it.
|
| That wouldn't make any sense.
|
| The blame realistically lies within each company who allowed
| a critical point of failure just so they get checkbox
| software and not actually have to expend the effort of making
| sure the company is able to function with their chosen
| infrastructure.
| margalabargala wrote:
| Exactly. The customer is to blame here, not the seller.
| Buyer beware.
|
| If airlines skimp on maintenance and planes crash, well,
| all those dead passengers should have picked an airline
| that takes maintenance more seriously. Next time they'll
| know better.
| autoexecbat wrote:
| It shouldn't be one or the other, but both
| Terretta wrote:
| > _I would like to live in a world where crowdstrike is
| directly liable for every dollar lost and has to pay for all
| of it._
|
| Or a world where the _regulator_ that required the checkbox
| -- _even if the firm can demonstrate a superior way of
| achieving the same objective_ -- should pay for it.
| leros wrote:
| Crowdstrike's contracts and terms of service will have
| clauses about how much liability they have when things go
| wrong. I have no idea what Crowdstrike's policy is, but
| pretty often, the liability is limited to the amount of money
| you've paid them during the outage.
|
| I've been involved in procurement at a big corporation and
| one thing we always modified in contracts was making the
| vendor 100% liable for any damages caused by their outages,
| but many vendors wouldn't make that modification.
| cwilkes wrote:
| I've heard of that for civil engineering firms but the
| amount of damages is capped at the yearly contract amount.
|
| Which in this case is probably a lot less than what these
| companies are paying in clean up costs.
| betaby wrote:
| Well, 'security' check boxes have consequences.
| paxys wrote:
| Today in "random number pulled out of someone's ass"
| mrinterweb wrote:
| Can't wait for the class action lawsuit. The total impact is
| likely greater than $5.4B. A significant number of people must
| have died due to the impact this had on hospitals and emergency
| services.
| btbuildem wrote:
| Realistically speaking, the liability lies with every
| individual organization that installed the corporate spyware on
| their systems.
| chasd00 wrote:
| hmm i was talking with my little sister about picking up some
| of their stock because these things always just blow over and
| the stock reverts back to where it basically was in time.
|
| I don't think many enterprises will switch because of the
| effort required and, instead, they'll just yell at the account
| reps for a while and then go back to paying the invoice.
| However, a big lawsuit is something i didn't think of.
| aeyes wrote:
| Everyone signed their terms of use:
| https://www.crowdstrike.com/software-terms-of-use/
|
| Section 6.1:
|
| THERE IS NO WARRANTY THAT THE SOFTWARE OR ANY OTHER CROWDSTRIKE
| OFFERINGS WILL BE ERROR FREE, OR THAT THEY WILL OPERATE WITHOUT
| INTERRUPTION OR WILL FULFILL ANY OF SOFTWARE USER'S PARTICULAR
| PURPOSES OR NEEDS. THE SOFTWARE AND ALL OTHER CROWDSTRIKE
| OFFERINGS ARE NOT FAULT-TOLERANT AND ARE NOT DESIGNED OR
| INTENDED FOR USE IN ANY HAZARDOUS ENVIRONMENT REQUIRING FAIL-
| SAFE PERFORMANCE OR OPERATION. NEITHER THE SOFTWARE OR ANY
| OTHER CROWDSTRIKE OFFERINGS ARE FOR USE IN THE OPERATION OF
| AIRCRAFT NAVIGATION, NUCLEAR FACILITIES, COMMUNICATION SYSTEMS,
| WEAPONS SYSTEMS, DIRECT OR INDIRECT LIFE-SUPPORT SYSTEMS, AIR
| TRAFFIC CONTROL, OR ANY APPLICATION OR INSTALLATION WHERE
| FAILURE COULD RESULT IN DEATH, SEVERE PHYSICAL INJURY, OR
| PROPERTY DAMAGE. SOFTWARE USER AGREES THAT IT IS SOFTWARE
| USER'S RESPONSIBILITY TO ENSURE SAFE USE OF SOFTWARE AND ANY
| OTHER CROWDSTRIKE OFFERING IN SUCH APPLICATIONS AND
| INSTALLATIONS.
| swat535 wrote:
| Parent is taking about deaths. You can't use terms of service
| limited iability regarding to death and TOD is not law.. I'm
| pretty sure it can be'litigated. You can't just say I am not
| responsible for death or injury and skip all regulatory
| requirements for safety critical systems
| rpeden wrote:
| It seems potentially tricky because they didn't just say
| they're not responsible for death or injury.
|
| They essentially got the customer to accept a contract that
| says the software isn't designed for use in systems where
| failure could cause death, and that the customer accepts
| responsibility for using it appropriately.
|
| I agree this whole incident was a massive blunder by
| CrowdStrike, but I'm not sure it makes sense to hold them
| liable for damage caused by customers using the product in
| a places they explicitly agreed not to use it in. In those
| cases, I think the organization that installed
| CrowdStrike's software in inappropriate places bears a lot
| of responsibility for the outcome, and their failure to
| understand the TOS they agreed to doesn't mean it's not a
| legally binding contract.
|
| It'll be interesting to see how it all plays out.
| ram_rar wrote:
| Can someone with a background in contract negotiation, vendor
| onboarding, and business continuity risk management share their
| expertise? We'd love to hear about typical vendor contract
| provisions that protect customers in situations like this.
|
| If damages can be demonstrated, what are the chances of airlines
| successfully claiming compensation? Or, in practice, do such
| cases usually result in significant discounts during the next
| contract renewal rather than actual damages paid out?
| com wrote:
| If the liability is capped at the cost of the duration of the
| incident (70 minutes from Crowdstrike's PR-messaging
| perspective) or one month's service charge - both pretty normal
| in standard contracts, then it's only outside of the contracts
| that some equity could be achieved. Not holding my breath
| though.
| Thaxll wrote:
| Imagine Apple / Google pushing an update that bricked 2b+ mobile
| devices.
| sandworm101 wrote:
| Does anyone else feel a little sympathy for CrowdStrike? They
| pushed out something they should not have. OK. That is bad. But a
| couple days on and the bulk of the difficulties seem to be from
| how windows handled the situation: The BSODs, the boot loops, the
| inability to recover from a basic fault. I feel that if this did
| happen in a linux environment (it could) that it would be easier
| to isolate and boot systems into some sort of temporary mode.
| Linux would communicate and offer options. The windows-specific
| trend of just abandoning all hope, giving up and throwing the
| BSOD at the user ... CrowdStrike didn't create that.
| betaby wrote:
| Why there should be any sympathy for them? Their business is
| shitty boss-ware. I care about boss-ware makers as much as I
| care about tobacco companies.
| timtom123 wrote:
| No? They did this on Linux too.
|
| https://www.theregister.com/2024/07/21/crowdstrike_linux_cra...
| disruptiveink wrote:
| What do you mean? They wrote the kernel driver. With great
| power comes great responsability.
|
| If you're writing a kernel driver that is deployed throughout a
| great portion of Fortune 500, with the money that that entails,
| then you should definitely be able to afford to pay people to
| write defensive code and have proper pipelines in place.
| sandworm101 wrote:
| And there is only the one kernel. No easy rollback option at
| boot, a previous version to at least get the system online.
| monksy wrote:
| "We can't hold back releases from going into prod.. we have to
| deliver"
|
| "We don't have enough time to write tests"
|
| "Developers should be able to test their own code"
| ajma wrote:
| But they apologized with a $10 gift card.
|
| Soo. $5.4B - $10
| throw7 wrote:
| The preliminary post incident review is here:
|
| https://www.crowdstrike.com/falcon-content-update-remediatio...
|
| It boils down to the "Content Validator" had a bug and gave a
| false positive.
|
| It's kind of crazy that the 'rapid response content' update was
| then free to go out direct to production machines with zero
| actual live testing.
|
| That's either due to c-suite excel cost-cutting/maximize profit
| or silicon valley yolo.
| slaw wrote:
| Silicon valley working at midnight? It was a contractor from
| India.
| Havoc wrote:
| Don't worry...they're handing out $10 vouchers to make up for it
|
| https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-ap...
___________________________________________________________________
(page generated 2024-07-24 23:15 UTC)