Post Ak68ysmERIBmawl0im by shana@mastodon.gamedev.place
(DIR) More posts by shana@mastodon.gamedev.place
(DIR) Post #Ak65noOM6RMtwlcQDo by foone@digipres.club
2024-07-19T18:50:37Z
0 likes, 0 repeats
Starbuck's ordering system appears to be down. They sell something like 5 million coffees a day, and that whole system is just down.Man, Cloudstrike isn't going to get sued over this, they're going to have their heads put on a pike as a reminder for the next ten generations that YOU DON'T DEPLOY ON FRIDAY.
(DIR) Post #Ak65wUNDtq8IT3E79M by ftg@mastodon.radio
2024-07-19T18:53:16Z
0 likes, 0 repeats
@foone Until the next time somebody does it."This gives us time to fix it over the weekend!"
(DIR) Post #Ak665o8JnreEMChNAG by mgleadow@mastodon.green
2024-07-19T18:54:29Z
0 likes, 0 repeats
@foone I think it was Thursday when they deployed, but still. Also, proven rollback techniques against terminal failures in tenor installs can be useful...
(DIR) Post #Ak66Hc7EJHicYnoq5w by humanhorseshoes@mastodon.world
2024-07-19T18:55:13Z
0 likes, 0 repeats
@foone Worked for a company that routinely deployed on Friday evenings and left it up to whoever was on call to work it out. No documentation ever.
(DIR) Post #Ak66PkttX2787gSovA by zorinlynx@tiggi.es
2024-07-19T18:57:04Z
0 likes, 0 repeats
@foone Or better yet, you don't deploy all your customers at EXACTLY THE SAME FREAKING TIME.
(DIR) Post #Ak66YzTSiUR9vdVIm0 by slothrop@chaos.social
2024-07-19T18:57:58Z
0 likes, 0 repeats
@foone The real IT doom loop is pushing a faulty update that breaks all the coffee shops.Who will be awake to fix the computers now?
(DIR) Post #Ak678Hjko9XYhlxCwC by madopal@mastodon.social
2024-07-19T18:53:39Z
0 likes, 0 repeats
@foone
(DIR) Post #Ak67GCAHPPuLjw8JJQ by nohillside@smnn.ch
2024-07-19T19:06:06Z
0 likes, 0 repeats
@foone it would have been equally bad any day, IMHO the lesson is rather „don’t roll out potentially critical changes to all systems at once“ (and even that is a lesson we already learned quite often).
(DIR) Post #Ak67XYpGPQ20wdaEa0 by admin@mastodon.slightlycyberpunk.com
2024-07-19T19:08:16Z
0 likes, 0 repeats
@foone I dunno, I feel like lately these kinds of massive outages just get seen the way we would see a natural disaster...if one company's system goes down, they lose out to a competitor and the person responsible for that outage gets fired. But if *everyone's* systems go down, there's no relative loss and the person responsible just implemented "industry standards" and cannot be blamed and everyone just kinda gets it fixed and moves on...
(DIR) Post #Ak67eHrTscZyMGpOQi by foone@digipres.club
2024-07-19T19:09:42Z
0 likes, 0 repeats
@madopal finally, someone who gets the reference
(DIR) Post #Ak68rdBrzVJaE9I8Cu by PJ_Evans@mas.to
2024-07-19T19:26:03Z
0 likes, 0 repeats
@foone I noticed last night that Ancestry was down, without warning. That was about 10PM, so i wonder if they were hit by it.
(DIR) Post #Ak68ysmERIBmawl0im by shana@mastodon.gamedev.place
2024-07-19T19:26:39Z
0 likes, 0 repeats
@foone But they deployed on Thursday evening their time! You only need to test things when it's actually Friday your time, it's the rules. 🤷♀️
(DIR) Post #Ak6SaBzdr2yO2LXABc by igb@mastodon.hccp.org
2024-07-19T23:01:01Z
0 likes, 0 repeats
@foone head put on a Pike’s Place amirite?
(DIR) Post #Ak6gKS1ls1HOUWuyXI by lightweight@mastodon.nzoss.nz
2024-07-20T01:40:48Z
0 likes, 0 repeats
@foone given it's Starbucks, I'm unexpectedly finding myself siding with CrowdStrike on this one... they've done the world a service.
(DIR) Post #AkDQ1SdFA0znhh5V2W by velinion@dice.camp
2024-07-23T07:41:14Z
0 likes, 0 repeats
@fooneEven more important, don't deploy an untested update!Also, they have absolutely no phased rollout of updates just in case of a bad update?Just how abysmally bad are the procedures and policies at CrowdStrike anyways?