[HN Gopher] Siblings miss crucial life-extending treatment becau...
___________________________________________________________________
Siblings miss crucial life-extending treatment because of
CrowdStrike outage
Author : nullindividual
Score : 110 points
Date : 2024-07-20 15:06 UTC (8 hours ago)
(HTM) web link (www.kiro7.com)
(TXT) w3m dump (www.kiro7.com)
| nullindividual wrote:
| Full title:
|
| Siblings miss crucial life-extending treatment at Seattle
| Children's because of CrowdStrike outage
| sgbeal wrote:
| Just FYI: site is inaccessible from outside the US.
| linacica wrote:
| Ooof yeah, just found out about too, first time seeing
| something not available in my country
| h2odragon wrote:
| https://archive.ph/yfbek
| throwaway3306a wrote:
| While I appreciate the effect this kind of downtime can have, I
| just don't understand these stories.
|
| Presumably it was planned in advance, so the patients know the
| time of their appointment and the doctor knows what was planned,
| and everything necessary to physically perform the treatment is
| already prepared at the hospital. What's stopping them from doing
| it without filling it into a digital system? Why is it impossible
| to make a paper record and fill it into the computer system
| later?
|
| If somebody was literally dying, would they stand around the
| computer like confused characters in a The Sims game who can't
| find the door, instead of saving the life? And if not, why is
| this less urgent case different?
| j-bos wrote:
| See some of the comments from affected medical staff on the
| main outage story, but the tldr is, tightly coupled systems.
| throwaway3306a wrote:
| I get that, my point is, why is it absolutely necessary to
| use the computer system? Why don't they just knock on the
| door, go grab the medicine and tools, apply it, then fill it
| into the system later?
|
| I understand they would just postpone whatever can be
| postponed to save the headache, I don't get the stories about
| life/health threatening situations.
| MostlyStable wrote:
| I can imagine that for something like this procedure, which
| is an infusion of medication into the brain it sounds
| like?, that the "tools" to perform the procedure themselves
| are computer based or computer dependent. It might not be
| as simple as injecting a drug into an IV line.
|
| Note that I am not a doctor and have absolutely no specific
| knowledge beyond what is in the original article, but I am
| guessing at potential explanations.
|
| Additionally, the article states that there is some "wiffle
| [sic] room" around the timing of the infusions. So it may
| be that the delay is not quite as serious as the title
| makes it sound.
| throwaway3306a wrote:
| Presumably they would fix these computers first thing
| during the night from a backup? If not, is this really
| about CrowdStrike, and not about a hospital unable to
| keep their absolutely critical computers backed up and
| restored in a timely manner?
|
| Again, I understand that restoring a complex net of
| servers is hard and takes time. But they surely have
| local hospital IT admins for these absolutely critical
| computers who are always available on site and can do it
| individually - it's not like there will be more than a
| hundred of these at a particular hospital? Hack it a
| little if you have to, disable the SSO etc - all that can
| be fixed later.
| advael wrote:
| The unfortunate fact of the matter is that centralizing
| IT systems around large corporate products, including the
| on-prem software and any cloud services, necessarily
| means less local control of what can go wrong and how it
| can be mitigated, and thus often problems that simply
| can't be fixed, even by competent on-prem staff. Even
| when it is possible, it's often highly illegal, and most
| organizations do a lot to beat risk-aversion into
| everyone on their staff, and of course I mean aversion to
| risk of breaking rules or protocols, not risk like
| "someone dying"
|
| I think it's always a mistake to outsource control of a
| mission-critical system, but that is exactly what large
| tech companies have been encouraging every organization
| that will listen to them to do for decades now
| throwaway3306a wrote:
| I have trouble accepting that. Even if they had to unplug
| the computer from the network and disable SSO and
| antivirus in safe mode, it's possible to get the computer
| operational. Even if they had to reinstall the OS and the
| critical software from scratch. There are solutions, the
| question is - did they even try? If not, why? And is
| CrowdStrike really to blame if they didn't? I just don't
| think so.
| advael wrote:
| Who in the org do you expect to have that competency, and
| do you think hospitals aren't keeping crucial things like
| credentials or software that gates access to things in
| the cloud when literally everyone in the world is
| encouraged to at every turn?
|
| The culture of organizational IT is broken because a lot
| of powerful companies found it profitable to break it and
| leave something inadequate in its place
| throwaway3306a wrote:
| I expect the local admins to be able to install a fresh
| OS not connected to the enterprise network. And I expect
| them to have physical copies of stuff like disk
| encryption keys, also backups of OS installations and
| images, and all critical software. If they don't have
| that or can't use it during an outage, the problem is
| incompetent IT management that has no business running a
| hospital, not CrowdStrike. Something else would take them
| out sooner or later.
|
| Again, we had all of this for a forest logging operation
| - is it too much to expect at a hospital?
| advael wrote:
| I agree with you, and kind of even agree that crowdstrike
| may not directly be at fault. But my point is that this
| competency is bled out of hospitals by external forces,
| primarily two: distant administration from companies that
| buy and manage multiple hospitals, often applying the
| same "efficiency" mindset that stripmines other
| industries in the name of profit, and the cloudtech
| sector, that is Google, Amazon, and Microsoft in
| particular, are very aggressive about selling their
| services along with demands that everything be given to
| their platforms, which often involves purging technicians
| who want on-site redundancy. This makes the systems more
| brittle, but also often causes people with the competency
| you're advocating to be fired
| WWLink wrote:
| I agree with this sentiment. If you ask me, the entity
| that comes out looking the worst from this Crowdstrike
| debacle are the companies that bought their service.
| Crowdstrike made a poorly designed and maintained
| product. I heard multiple people on reddit say it's the
| best of that type of product, but what the hell? Why does
| it need kernel-level control?
|
| Why did we get here? If you're installing kernel-level
| software you might as well run a kiosk that only runs
| presigned code and runs off a read-only system image. And
| a lot of the machines in question DO APPEAR to be kiosk
| settings (like hospital data entry terminals).
|
| It's easy to sit back and armchair, I'm sure there will
| be many cybersecurity experts who would figuratively jump
| at my throat for suggesting that trusting a vendor to run
| a rootkit on your computers is a bit incompetent. LOL. :D
| advael wrote:
| At this point I just assume any "cybersecurity expert"
| that defends Microsoft's nonsense is a cop
| kmeisthax wrote:
| Everyone installing Crowdstrike seems like they want to
| build locked-down kiosks but haven't heard of Windows
| Embedded yet. Or at least I'm assuming there's an
| Embedded configuration that lets you do AMFI[0]-tier code
| signing enforcement.
|
| [0] AppleMobileFileIntegrity, the daemon and kext on iOS
| that enforces very strict code signing.
| lambdaone wrote:
| Absolutely. The risk being managed is the risk to the
| CEO/CTO's jobs, not the risk to life.
| toast0 wrote:
| Hospital IT sucks. Look at a news report about a
| ransomware or this and it can easily be a few weeks for
| them to get back in shape. This one is hopefully easier
| because reportedly CloudStrike can sometimes pull an
| update before it BSODs and most windows machines auto
| restart on BSOD, so just leaving things unattended may be
| enough.
|
| Restore from backup or reimaging fresh often means you
| need a working backup or image server, which at a lot of
| these places is also a Windows server and is likely also
| running the same endpoint protection, and is likely also
| boot looping.
|
| Restore from zero isn't something any IT wants to do, and
| many of them aren't prepared to do it either.
|
| Like it or not, hospital care revolves around the
| electronic medical records systems, and while Kaiser
| Southern California in the 90s was using amber screens
| and some sort of mainframe, afaik, almost everyone is on
| EPIC now, which is a windows application with all the
| baggage that contains. Even before EPIC took over Kaiser,
| they were running terminal emulators on Windows.
|
| IMHO, it would be better for them to put together a
| ground up desktop distribution with exactly what they
| need, but that has user training costs and development
| costs.
| anotherhue wrote:
| It's because these computers are a means of corporate
| control. Policies and checks and procedures and whatever
| are all delivered through them.
|
| It's preferable, from the corporate perspective, to have
| everything fail temporarily than to relinquish this level
| of workforce management.
|
| If this is hard to imagine, just think of a Lyft driver
| from the perspective of Lyft Inc.
| ciabattabread wrote:
| _Hotels_ have difficulty with paper and pen bookings when
| their computers are down. You expect a modern hospital to
| function in those circumstances?
| dopylitty wrote:
| Yes
| pasquinelli wrote:
| the hospital better function.
|
| what you're saying is, if the less important service
| fails, of course the more important one will fail too.
| j-bos wrote:
| Have you ever worked a job that requires high degree of
| physical world logistics? In times where the primary
| coordination mechanism is down, any action becomes much
| slower to implement and often at a direct cost to
| implementing other actions.
|
| With regard to this case, I don't know any specifics, but I
| can imagine tools require digital calibration, inventories
| not tracked outside digital systems, certain meds behind
| digital access control, and emergency response striained to
| the point where complicated non emergency procedures would
| be more risk than benefit.
| throwaway3306a wrote:
| I have managed IT departments that managed hundreds of
| locations and thousands of computers running Windows XP
| and Windows Server 2003, no cloud at all. And I went
| through several similar outages (similar in impact on our
| operations, not cause or impact on others). Our first
| priority was to get the critical computers that operated
| machinery running - we did that hours (1-2) after the
| problem started. Then we played around with the servers
| and network for few weeks - but critical stuff was
| operable, albeit with lesser capacity and efficiency.
|
| And we were managing forests and waterways, not hospitals
| and human lives.
| TeMPOraL wrote:
| That's all fine, but this time, no one could get those
| computers back up in the first few hours, since they were
| stuck in a boot loop. Plus, systems like hospitals had to
| be running all that time. Plus, at the scale this outage
| is reported to be - banks, stores, factories, phones,
| emergency services, CNC machines, networking, aircon - I
| imagine everyone was confused and trying to figure out if
| _anything_ works.
|
| I'm happy nothing significant was hit over here in
| Poland; reading the main HN thread on the outage feels
| like reading war reports.
| throwaway3306a wrote:
| If it's stuck in a boot loop, the first thing I do is
| call the local admins and tell them to take a fresh SSD
| and a Windows installation USB drive with them. Plug the
| new SSD, reinstall the OS and copy the files from the old
| one. Computer running in less than an hour.
|
| That's literally what we did to restart our forest
| logging machinery. Are human lives less critical than
| that?
| saulpw wrote:
| You might consider that things have changed in the past
| 20 years. Also that medicine operates differently than
| forest logging.
| throwaway3306a wrote:
| Things haven't changed in IT so much. I am not in ICT
| management anymore, but I write software for the modern
| enterprise systems and networks - I'm reasonably up to
| date.
|
| Ad medicine - hence my question, I'd really like to know
| what's the blocker. So far it seems the blocker is bad IT
| management, regulation and liability, not impossibility
| to perform the treatment.
| j-bos wrote:
| Your answers indicate that you have not worked in n
| environment heavily dependent on ever shifting physical
| world logistics. You might try talking to some
| coordinators on the ground of a hospital, rescue center,
| consteuction site, theme park, or military operation for
| insight.
| throwaway3306a wrote:
| I talked to people in charge of the operations on a daily
| basis for years. I really don't think these
| considerations have changed that much since my times of
| leadership of an entire department managing just that.
| nullindividual wrote:
| From having seen the infusion process myself, I take it
| that it requires precision measurements over an extended
| period of time. This would seem unreasonable requirement
| for staff to perform.
|
| Again, from what I've seen, infusions are not just "throw
| it in an IV bag and wait".
| throwaway3306a wrote:
| If it requires a computer, why was that operationally
| critical computer not restored from a backup within hours
| after the problem started? This has nothing to do with
| CrowdStrike or other bugs - it could've simply failed
| hardware wise and the hospital should have been able to
| replace it immediately.
| nullindividual wrote:
| You have a naive view of how modern operations work, I
| must say. This shows when you suggest endpoints have
| backups. We're back to the mainframe/terminal times where
| all software is running on a web server or other
| centralized application server, which is also in a boot
| loop, somewhere else.
|
| Failed hardware is different, but hospitals likely have
| very few computers just 'lying around'. Especially the
| highly regulated machines, such as those which are
| attached to MRIs and the like.
|
| CFR 21 Part 11 was the bane of my existence. Software
| that can be installed and configured in a matter of
| minutes? That's a six month project, at least. Sure,
| backups are great, but then you've got a significant
| process to get it back up and running.
|
| These aren't early-2000 logging operations.
|
| I see you'll never be convinced, but this is how modern
| operations work. Being a hospital (or other industry with
| heavy government regulations) make operations that much
| worse.
| throwaway3306a wrote:
| You misunderstood me, I am easily convinced that this is
| the case - what I don't get is how they could let it be
| the case.
| zdragnar wrote:
| I can get stories like call centers, but I absolutely don't
| understand how life critical systems aren't air gapped and
| rigidly controlled.
|
| Fail safe is the only acceptable failure mode for any
| critical system. Crowdstrike failed here, but they're not the
| only thing that can go wrong with computers. Where is the
| redundancy?
| TeMPOraL wrote:
| Life-critical systems are air-gapped. Just no one
| considered systems running Epic to be life-critical. It
| turns out they are, probably more so than most.
|
| Also, air-gapping helps only so much when network dies and
| hospitals can't exchange patient information or send images
| from MRIs and X-rays to radiologists.
| Rinzler89 wrote:
| _> and hospitals can't exchange patient information or
| send images from MRIs and X-rays to radiologists_
|
| My dentist literally took a photo of my x-ray with his
| phone and sent it to to my orthodontist via Whatsapp and
| everything went quick and smooth, much faster than the
| official channels. Solutions to get a job done quickly
| and efficiently in case of emergency always exist,
| they're just not "by the book".
| czl wrote:
| Imagine a news story about a dentist that violated HIPPA
| (or equivalent) laws because they used Whatsapp /
| Facebook to share medical records. Will this news story
| be about a hero vs someone who got into trouble?
| Rinzler89 wrote:
| How would he get in trouble?
|
| Hippa doesn't apply in Europe but GDPR, and I don't see
| how that would be in violation since my information was
| exchanged only between the two parties with my consent,
| on an encrypted channel.
|
| They would only get into trouble if that info would leak
| in an identifiable way to unauthorized third parties and
| would cause damages (here there's no punitive damages
| like in the US). And people here tend to guard their
| WhatsApp chats pretty well since it's what everyone uses
| and it also contains their private chats so in a sense it
| can even be more secure than the official medical
| channels which are just more burocratic but offer no
| actual guarantee of more data security.
| czl wrote:
| > my information was exchanged only between the two
| parties with my consent, on an encrypted channel
|
| Say WhatsApp is found to have a security hole that has
| been leaking data to 3rd parties. What may be the fate of
| dentists / doctors that decided to use it an "encrypted
| channel" for medical records? Are doctors / dentists not
| fat targets for lawsuits? What might the guidance be from
| their lawsuit insurance policy?
| colechristensen wrote:
| It's not clear to me that this case is actually life
| threatening. They have a regular procedure which even in the
| article they say they have wiggle room for timing.
|
| If all of your computers go down your throughput is going to go
| down because other kinds of organization are going to be slower
| to do ad hoc... so you triage.
| dsclough wrote:
| A nurse was unable to give my wife medication while in labor
| because the barcode on the bag of drugs wouldn't scan.
| Fortunately we just had to wait another 20 minutes to get a new
| bag from the pharmacy but I can easily imagine a world where
| doctors are unable to perform procedures they are physically
| capable of doing because of liability surrounding not using the
| computer systems as intended. Epic particularly has really done
| a number on the healthcare system.
| throwaway3306a wrote:
| I really, sincerely don't understand that. How does an
| unscannable barcode prevent a doctor/nurse from administering
| medicine they are holding in their hands?
| simmerup wrote:
| Because they're accepting the liability of it going wrong
| if they make an unusual choice to disregard the error
| throwaway3306a wrote:
| So the problem really isn't CrowdStrike or any computer
| at all, but dumb policy or regulation?
| advael wrote:
| I think a good remedy would be to completely remove
| "normal procedure" as a defense against liability. Our
| legal standard should defend people who break protocols
| if they know they will result in harm, and prosecute
| people who don't, or prosecute the people who make the
| protocols in those cases. Law should supercede corporate
| policy, not treat it as a form of law
| shouldbeone wrote:
| It doesn't. We do this all this time in rapid responses and
| cardiac arrest scenarios, when we can't wait for an order
| in the EHR; someone keeps track of the medications, doses
| and rough times of administration, and it's entered into
| the EHR later.
| toldyouso2022 wrote:
| Because the law doesn't want to do its job anymore, so it
| created useless bureaucracy to make their lives easier and
| human life hell
| lambdaone wrote:
| An awful lots of apparently useless bureaucracy exists
| because many people, left to themselves, are often very,
| very stupid.
|
| Bureaucracy certainly stops smart people from doing the
| right thing, but more often, it stops stupid people from
| doing the wrong thing. Hack away at bureaucracy at your
| peril.
| ctxc wrote:
| Well said.
| newaccount74 wrote:
| It also stops smart people from doing stupid things.
| sigmoid10 wrote:
| The other commenter already said it: Liability. What if the
| scan is part of a procedure that ensures that the right
| drug is given to the right patient? Giving someone the
| wrong drug or even the wrong dose can cause serious harm.
| Imagine they kill someone that way and then during the
| investigation it turns out the they didn't scan the meds.
| It doesn't matter why they didn't scan it (lazy, forgetful,
| computer problem), it is en enormous legal risk for every
| party involved. Thousands of people die each year because
| of medical errors, so trying to prevent doctors from
| killing people by using strict procedures is very
| important. Even if it means that in extreme situations like
| this the procedure can cause harm as well. Overall it will
| save many, many more people than it will kill.
| throwaway3306a wrote:
| What if it turns out they harmed the patient by insisting
| on following the standard procedure during a worldwide
| outage? Isn't that the same kind of liability risk, and
| is the regulation really going to protect them in this
| case? If so, isn't that a hugely problematic regulation?
| FeepingCreature wrote:
| Yes, and yes. Welcome to our messed up society.
| AlotOfReading wrote:
| In that case the nurse or doctor has a strong defense of
| "I was following policy" for their insurance and boss.
|
| The people writing hospital policies or regulations
| aren't thinking about individual patient outcomes unless
| some notable news story came out recently, and even then
| it's maybe the third or fourth priority on a list a
| hundred items long.
| sigmoid10 wrote:
| We don't know that this is what actually happened in OP's
| case. I was referring to the comment you replied to and
| there it is pretty obvious that the regulation is exists
| to prevent harm from being done. But even if there is a
| clear justification, you would expose yourself to a
| lawsuit and need argue all this in court. I can totally
| understand why people don't want that, especially in the
| US. So if anything, you should blame the legal system.
| tzs wrote:
| If the barcode wouldn't scan there it might _also_ have not
| scanned correctly when that bag was being filled which
| could have led to it being filled incorrectly.
| throwup238 wrote:
| _> What 's stopping them from doing it without filling it into
| a digital system? Why is it impossible to make a paper record
| and fill it into the computer system later?_
|
| It's not filling in new data that's the problem - every person
| involved in treatment needs to be able to access the patient's
| medical records to check for contraindications. Allergies and
| drug interactions are a quick way to kill someone when
| injecting drugs directly into their veins even if they're
| already in a hospital.
|
| At a major hospital there's too many patients coming through
| and the data changes too frequently to keep paper backups.
| moralestapia wrote:
| >Her children's appointments were cancelled, the first they
| would miss in five years.
|
| They've been going every two weeks for the last five years. I
| doubt they wouldn't know what to do ...
| throwup238 wrote:
| It's a large children's hospital with thousands of
| employees treating tens of thousands of kids a year, not
| some rural family doctor with a list of patients that can
| fit on a single sheet of A4. They're not going to get the
| same staff every time and the staff isn't going to memorize
| the charts of every patient.
| renewiltord wrote:
| The truth is that many medical personnel are not agentic. They
| are human robots unable to act unless instructed to by a
| computer. The computer tells them when they can do something
| and they do it.
| CoastalCoder wrote:
| Do it really make sense to blame just CrowdStrike for this?
|
| They were one link in what appears to be a pretty fragile
| dependency graph.
|
| For example, wouldn't it possibly make sense to also blame:
|
| * Regulators / insurers / etc. who require passing the audits
| that mandate using services like this.
|
| * System designers who failed to implement disaster recovery
| plans for this scenario.
|
| * Auditors who failed to highlight this risk.
|
| * Device vendors who made medical equipment susceptible to this
| kind of DoS.
|
| * U.S. FDA / DEA who allowed and/or mandated systems with this
| kind of vulnerability.
|
| * Voters (in democracies) who ultimately bear responsibility for
| their government's actions/inactions.
|
| Etc.?
| TeMPOraL wrote:
| There's lot of blame to pass around, and a lot of systems to
| reconsider, but at least initially, the blame lies with people
| who had a kill switch to critical infrastructure in multiple
| countries, were fully aware of that fact, and yet were so
| careless they accidentally pulled it.
| stefan_ wrote:
| I don't exactly care who is blamed for this in the chain of
| stupidity, but it must happen. This corrosive attitude of "oops
| software problems nothing we can do" must end fast.
| Qem wrote:
| *because of CrowdStrike and Microsoft
| Bilal_io wrote:
| According to some comments in Yesterday's thread Debian was
| also hit with something similar back in April.
|
| Another comment in this thread quotes Crowdstrike's ToS which
| states that their software should not be used on critical
| systems.
|
| I blame the hospital for its inability to operate with pen and
| paper in the event of a computer crash or a power outage.
| lambdaone wrote:
| It's pretty difficult to operate a CT scanner with pen and
| paper, to name just one thing that fell over yesterday. CT
| scanners are life-critical.
| freehorse wrote:
| Why any machine related to the operation of a CT scanner
| itself has to be connected to the internet? The problem is
| not "using technology" in general. The problem is internet
| connectivity being not correctly identified as a liability
| in designing our technology infrastructure systems.
| nullindividual wrote:
| From what I've seen when I've had CTs (I am not a medical
| professional and have no direct ties to their industry),
| the machine sends the images in real time to a technician
| in another room. Those images are then sent to an offsite
| service for review by a radiologist, then returned to the
| doctor to give you the results and they're uploaded to
| Epic where I can review them online at my leisure.
|
| It's all on a network for a reason. If it's on a network,
| it has to comply with all regulations that govern the
| service.
| freehorse wrote:
| Data transfer can happen after data acquisition through
| another machine that can be connected. If that machine is
| compromised alternative channels can be found. There is
| no fundamental reason in how the systems work that before
| that step anything has to be connected online. Having
| things online is actually a liability, as evidenced by
| the mere fact that software like crowdstrike should be
| installed with kernel access on them. Why instead of
| going that path, we do not just segregate networks to
| make them more resilient? The only reason I see is all
| these -aaS business models. Nothing that really relates
| to the real needs of people or the healthcare system
| itself. After the ransomware attacks, instead of reducing
| attack surface the direction was to actually increase
| liability and risk by having another point that things
| can fail. I do not think anybody really learnt from that
| anything imo.
| nullindividual wrote:
| Coulda woulda shoulda, of course. Money. That's why.
| Bognar wrote:
| If some third party software you chose to install on your
| system added a kernel module and started causing kernel panics,
| would you blame the kernel maintainers?
|
| I'm sure if MS decides to remove the ability for third parties
| to write code that runs in kernel mode in the name of
| security/reliability/whatever, this site would immediately turn
| on a dime and say that Microsoft is evil for removing user
| control over their machines.
| bongodongobob wrote:
| "Well since it's not open source and they can't audit the
| code line by line (like I always do), they shouldn't use it
| so it still the user's fault." Probably. Tech nerds tend to
| be hilariously out of touch with big picture stuff beyond
| their basement lab.
| slaymaker1907 wrote:
| I'm not even sure Microsoft could actually restrict 3rd party
| code running in kernel mode like that from a legal
| perspective. There are a certain requirements about
| documenting interfaces in Windows from the 90s antitrust
| stuff.
| Twirrim wrote:
| It's probably been raised before, but the CrowdStrike terms of
| use (https://www.crowdstrike.com/software-terms-of-use/), section
| 6.1, have the usual blurb on them (emphasis mine):
|
| > Neither the software or any other Crowdstrike offerings are for
| use in the operation or aircraft navigation, nuclear facilities,
| communication systems, weapons systems, DIRECT OR INDIRECT LIFE-
| SUPPORT SYSTEMS, air traffic control, or any application OR
| INSTALLATION WHERE FAILURE COULD RESULT IN DEATH, SEVERE PHYSICAL
| INJURY, or property damage. SOFTWARE USER agrees that it is
| SOFTWARE USER'S RESPONSIBILITY TO ENSURE SAFE USE OF SOFTWARE AND
| ANY OTHER CROWDSTRIKE OFFERING IN SUCH APPLICATIONS AND
| INSTALLATIONS.
|
| We don't really think long and hard enough about isolation of
| systems, and what levels of access they actually need to be able
| to do their tasks. It's entirely practical to build completely
| isolated networks. US Government (and most major governments)
| operate classified networks with air gaps, network diodes and the
| like. We don't have to make everything actually internet
| accessible, while still retaining the ability to get data in to
| such isolated networks.
| nneonneo wrote:
| I suspect that if the operators of such facilities obeyed all
| of the terms of use for every product they wanted to use,
| they'd be using pen and paper for everything...
| niemandhier wrote:
| No, that's why medical and aviation products are expensive
| and these are difficult markets to penetrate.
|
| The degree of reliability that is required is insane, I
| cannot read the article since I am outside the US, BUT if
| these are the terms of service and the product was used in
| any area the was excluded under these terms, the entity that
| used the product might very well be guilty of gross
| negligence.
| Frieren wrote:
| and it was not. No plane was flying with that software. It was
| booking services and similar needs. Planes could fly just fine,
| it was impossible to book people, thou.
|
| My guess is that it's similar in this case. (Site is down)
| TeMPOraL wrote:
| Hospital networks and computers running Epic are very much
| indirect life-support systems, faulire of which can cause
| lots of injuries and deaths - as we're learning now in real-
| time.
| SoftTalker wrote:
| I'm hearing that most hospital cybersecurity insurance
| requires Crowdstrike (or a product like it) on all the
| endpoints, so if that's true the liability might fall back
| on them. It will be a protracted argument for sure.
| lainga wrote:
| Now comes the important question: did CrowdStrike's sales team
| try to sell solutions to operators of such critical systems? If
| so, how hard did they push?
| lambdaone wrote:
| Given this: https://www.theregister.com/2024/07/19/crowdstrik
| e_update_nh... _someone_ has certainly been selling
| Crowdstrike software to be used on life-critical systems.
| stackskipton wrote:
| Been in Hospital IT, yes they did. Sure, they talked about
| not putting on "Life Critical Systems" but just about every
| system at Hospital is life critical. This is extremely common
| in IT because I'm sure Microsoft Windows has similar clause.
| Hell, I bet Epic (Electronic Medical Records) has some clause
| about "Make sure you have backups" when their whole pitch is
| "throw out the paper!"
| blackeyeblitzar wrote:
| Of course they did. This type of fine print is meaningless
| and only there to push responsibility onto their customers.
| This type of issue shouldn't happen at large companies. We
| need regulations that create penalties and jail time for
| larger companies that have security incidents or other issues
| like this one.
| infamouscow wrote:
| We need regulations that create an environment for which
| those affected can directly seek retribution with full
| civil and criminal immunity.
|
| I'm sure there are lesser measures that _could_ work, but I
| 'm exceedingly confident the above will be extremely
| effective.
| devonkim wrote:
| We can look at a bit more regulated industries like
| airlines (see: Boeing) and see that it's not necessarily
| working whatever path we're on is certainly not that
| approach.
| nullindividual wrote:
| NT 4 EULA had a similar note, but it was limited in scope to
| JAVA.
|
| > 8. NOTE ON JAVA SUPPORT. THE SOFTWARE PRODUCT CONTAINS
| SUPPORT FOR PROGRAMS WRITTEN IN JAVA. JAVA TECHNOLOGY IS NOT
| FAULT TOLERANT AND IS NOT DESIGNED, MANUFACTURED, OR INTENDED
| FOR USE OR RESALE AS ON-LINE CONTROL EQUIPMENT IN HAZARDOUS
| ENVIRONMENTS REQUIRING FAIL-SAFE PERFORMANCE, SUCH AS IN THE
| OPERATION OF NUCLEAR FACILITIES, AIRCRAFT NAVIGATION OR
| COMMUNICATION SYSTEMS, AIR TRAFFIC CONTROL, DIRECT LIFE SUPPORT
| MACHINES, OR WEAPONS SYSTEMS, IN WHICH THE FAILURE OF JAVA
| TECHNOLOGY COULD LEAD DIRECTLY TO DEATH, PERSONAL INJURY, OR
| SEVERE PHYSICAL OR ENVIRONMENTAL DAMAGE.
| netsharc wrote:
| I wonder if that's MS's Java or Sun's. If it's from a third
| party they probably copy-pasted the relevant paragraphs.
|
| Quicktime also has a paragraph about not using them to
| operate nuclear facilities...
| minton wrote:
| QuickTime? The video stuff from Apple?
| netsharc wrote:
| Yeah, the one we had to install on Windows 98 to play MOV
| files. Which, I learnt yesterday[1]/[2], is the video
| format that turned to MP4.
|
| [1] https://obsproject.com/blog/obs-studio-hybrid-mp4
|
| [2] https://news.ycombinator.com/item?id=40951187
| kmeisthax wrote:
| The Quicktime nuclear clause also got copied into the
| iTunes EULA, which is _extremely funny_.
| cranberryturkey wrote:
| Well shit, I guess I have to pivot. I was going to
| operate my nuclear facility with Quicktime.
| sgbeal wrote:
| The disclaimer about not using Java for X, Y, and Z was
| part of the Java EULA for many years (and my still be - no
| idea).
| Wytwwww wrote:
| I don't think an EULA absolves you of any possible liability.
| Courts can more or less decide to ignore it if they see
| sufficient reasons to.
| akira2501 wrote:
| Why does the hospital get a pass? They have critical care
| equipment, with no redundancies, and no plan to deal with an
| outcome so predictable it's been in EULAs for decades.
|
| These lessons need to cut in all directions. If you want to
| profit off of treating disease you should be held to a much
| higher standard. Passing the buck off to your AV provider is
| convenient in the current atmosphere but it's incredibly
| short sighted.
| awinter-py wrote:
| with larger corp users there is often an MSA that overrides the
| TOS; no idea if that's the case here, but give hospital GC
| benefit of the doubt re due diligence
|
| also would be interesting if crowdstrike was installed by a
| reseller who specializes in healthcare / airlines, not the
| hospital or airline itself
|
| waivers not always enforced, per [1]
|
| > Courts will hold overbroad liability waivers unenforceable on
| public policy grounds. ... injurers routinely ignore these
| holdings and persist in requiring would-be plaintiffs to sign
| such unenforceable waivers anyway
|
| at least one federal case in florida[2] saying advertising a
| product as safe doesn't defeat the EULA, but it's florida
|
| 1. https://wp0.vanderbilt.edu/lawreview/wp-
| content/uploads/site...
|
| 2. https://casetext.com/case/justtech-llc-v-kaseya-us-llc
| niemandhier wrote:
| People ridicule the German fetish for doing things on paper and
| using cash, but many things tend to work here even if the
| computers stop working.
|
| My general practitioner once treated me during a power outage,
| all I had to do was come back and have my insurance scanned
| later.
| odiroot wrote:
| That's a really moot point because the legacy banking industry
| there got heavily affected.
|
| The nation's weird affinity for pen and paper is nothing but
| Luddism. Once you have experience living in a country with good
| e-governance you'd roll your eyes at Germany and their love for
| faxes.
| niemandhier wrote:
| Well I did not notice whatever affected the banking industry
| of my country, which at least for me makes my point.
|
| Good e-governance is incredibly difficult, which explains our
| love for faxes.
|
| E.g. Microsoft faced a lot of scrutiny in the congress
| hearing in June, some people go as far as saying MS is a
| danger to the national security of the USA.
|
| If the US, which is the home of these companies and can put
| pressure in ways the German government cannot, still can't
| force them to deliver secure systems, a fax ( at least an
| encrypted one) looks pretty attractive.
|
| When the EU parliament still used faxes, the US at least hat
| to break into the offices and manually install components to
| the machines to get access.
| Rinzler89 wrote:
| _> Good e-governance is incredibly difficult_
|
| And yet Estonia, a former impoverished communist country
| significantly less wealthy than Germany did it, and did it
| well. But no, Germans always have a laundry list bingo of
| FUD excuses as to why it can't possibly work. The bingo
| usually starts with "but m'uh privacy!" even though BAMF,
| Schufa and every law firm and government agency remotely
| interested in you can find out everything about you if they
| want to fine you for something.
| niemandhier wrote:
| 1.Estonia started more or less from scratch after 1990.
| Building a new system based on modern standards is easier
| than reforming one that is working under full load.
|
| 2. Estonia is much smaller than Germany, and afaik much
| more centralised.
|
| Estonia did a lot of things right, but countries like
| Britain, France and Germany can learn little from
| Estonia. The results just do not translate.
|
| 3. The German system as it is, is designed to make it
| slow to change. Even if hard right AFD would get the
| majority of votes in the next election, during one
| legislative term they could not change the system a lot.
| To do this, even the absurd long Merkle reign was not
| long enough.
|
| 3. might sound like a disadvantage, and it certainly is
| for the Germans, BUT if Germany would follow the likes of
| Hungary, Poland and Turkey, that would destabilise the
| whole continent.
|
| There is a joke, that claim that the German anthem
| actually is:" Stability, stability uber alles"
| Rinzler89 wrote:
| _> BUT if Germany would follow the likes of Hungary,
| Poland and Turkey, that would destabilise the whole
| continent._
|
| Kind of an unrelated point. Using faxes and paper based
| burocracy won't save you from crappy politicians
| implementing crappy policies or a government going crazy.
|
| Why are Germans so obsessed with correlating that crappy
| burocracy automatically means more political stability as
| some lame excuse for maintaining the inefficient and crap
| public burocracy? You can also have political stability
| with efficient burocracy. The key is political
| accountability and separation of democratic powers, to
| maintaining stability, nothing to do with using digital
| or paper for burocracy.
| hnbad wrote:
| > Even if hard right AFD would get the majority of votes
| in the next election, during one legislative term they
| could not change the system a lot. To do this, even the
| absurd long Merkle reign was not long enough.
|
| This is a widespread misunderstanding. The AfD could
| drastically change the system if it had the sufficient
| number of seats (or a willing partner). There are very
| different pathways depending on what your goal is. Merkel
| was a conservative who had no interest in "changing the
| system a lot" (quite the opposite). The AfD wants to
| drastically change things like the immigration system.
|
| It's a bit like how in the US the SCOTUS, thanks to the
| Trump appointed judges, effectively ruled that the
| President has a lot more power and is largely above the
| law (more so than these things already used to be the
| case before) but the Dems and Biden think the ruling is
| bad and thus refuse to do anything with that. They could
| also easily change the tune of the SCOTUS by appointing
| more judges but refuse to do so to avoid setting a
| precedent (as if the GOP needed it). "Centrists" often
| value decorum above succeeding at their stated goals.
|
| There's an episode of _Die Anstalt_ that plays through
| how the AfD could functionally abolish most of the
| constitution and democratic system within a single term
| if you want to know the specifics but it mostly comes
| down to abusing rules that exist because the system was
| created under the assumption that everyone would play
| fair (which is ironic given how things like the 5% hurdle
| are justified).
|
| > There is a joke, that claim that the German anthem
| actually is:" Stability, stability uber alles"
|
| I'm not sure where you heard that joke or whether this is
| a translation of the joke but the German anthem does not
| actually contain the words "uber alles". The German
| national anthem consists of a single stanza of the _Das
| Lied der Deutschen_ , the first stanza of which begins
| with "Deutschland, Deutschland uber alles".
|
| BTW it's a common misunderstanding that the "uber alles"
| ("above all") part is why only the final stanza was used
| by West Germany for its anthem. However that was
| originally meant as an appeal to German nationhood and
| unification "above" the monarchs although it was later
| adapted as an expression of national superiority. The
| actually problematic part comes later in the first stanza
| where it names rivers as boundaries - not only did that
| include East Germany (a separate country) but also parts
| that were no longer part of either of the two countries.
|
| As for why the second stanza didn't make the cut, I guess
| it was just a weird one to start a national anthem with
| as it celebrates German women, fidelity, wine and song
| and their "old respected fame" which at this point
| probably felt anachronistic and also wasn't as strong as
| the third stanza's "unity and justice and freedom" (with
| "unity" also having a new meaning when East Germany had
| become a separate country). That said, personally as a
| German I think it's a terrible anthem, especially given
| the melody was originally written to celebrate the Kaiser
| (first of the Holy Roman Empire and later of Austria-
| Hungary).
|
| Personally I would have preferred the original East
| German anthem _Auferstanden aus Ruinen_ , at least
| textually. It's also not great but at least it's better
| than digging through scraps to adapt an outdated poem set
| to a monarchist hymn.
| schroeding wrote:
| > even though BAMF, Schufa and every law firm and
| government agency remotely interested in you can find out
| everything about you if they want to fine you for
| something.
|
| This isn't true, IMO. German administrative offices in
| general don't talk to each other without your permission,
| with exceptions like the police. This is also the reason
| why it's e.g. a giant pain to change your name in Germany
| - everyone has their own independent database. Can you
| maybe clarify, please?
|
| BAMF has no data of German citizens ("Bundesamt fur
| Migration und Fluchtlinge", Federal Ministry for
| Migration and Asylum Seekers).
|
| Schufa is a private company which only gets data from
| other companies, not the government. If you don't allow a
| company to give your data to the Schufa (which, to be
| fair, you have to do for many things), they cannot
| legally get the data (and in this case you could force
| them to delete it via GDPR).
|
| German administrative authorities don't even have
| compatible databases. Like, if you go from Munich to
| Berlin, they have to basically enter your data manually.
| The software of the local municipalities
| (Einwohnermeldeamter) have no common API, and up until a
| few months, there wasn't even a unique ID for every
| citizen which could be used as a key in databases.
|
| Law firms only can get some data if a court allows it.
|
| If you get e.g. a ticket for speeding, and you didn't
| drive your car, but your spouse did and you don't tell
| them who the person in the driver seat on the picture is,
| there is nothing they can do (except forcing you to, from
| now on, keep a log book of who drives your car). They
| can't just call your local municipality and get the ID
| pictures of your spouse or something.
| schroeding wrote:
| Faxes - I fully agree.
|
| But there are areas where it definitely isn't just Luddism,
| especially in healthcare. See what happened in Finland [1].
| Yeah, you can break into a GPs office and steal the physical
| data quite easily, but that doesn't scale, hacking a
| centralized service does.
|
| (Sure, there are solutions which would be similarly resistant
| against hacks as paper - like saving the data on the actual
| insurance card. But those are not implemented - it has to be
| a centralized (often SaaS) solution, where hacks can scale
| nicely)
|
| [1] https://www.bbc.com/news/technology-54692120
| rootusrootus wrote:
| That is an upside, but what are the downside costs? Are there
| more errors? Do people die more often because something got
| overlooked because it was not shared? Certainly it is more
| expensive, but even putting that aspect aside I would want to
| know for sure that it was a net positive and not just in this
| one situation.
| ugh123 wrote:
| > but many things tend to work here even if the computers stop
| working.
|
| Sure, they work. At the pace of 1980's business speed. The U.S.
| is 4x bigger population and 6x gdp than Germany.
| 999900000999 wrote:
| https://crowdstrike.wd5.myworkdayjobs.com/crowdstrikecareers
|
| Looks like Crowdstrike outsources their SDET/QA while keeping
| most software engineers stateside.
|
| I generally don't have an issue with outsourcing, but it's
| obvious they're trying to save money on QA here. A few 200k SDETs
| could of probably caught this.
|
| I see this at tons of companies, they see QA as less important...
| alexchamberlain wrote:
| There are 3 axes of risk: probability that something goes
| wrong, the impact of something going wrong and the time to
| remediation when something goes wrong.
|
| You're arguing that on shoring QA would reduce the probability
| of something going wrong. I'm neither going to agree nor
| disagree.
|
| However, I think the failure here is to mitigate the impact of
| something going wrong. Their rollout plan was fundamentally
| flawed - it shouldn't have taken out so many machines at the
| same time. It should have been rolled out in stages, with only
| 1 machine at most at any given customer receiving early
| versions.
|
| It's best to assume a bug will get through 1 day or another,
| and spend some time mitigating the other axes too.
| 999900000999 wrote:
| My argument is they decided to cut cost on QA. It's very
| likely a higher paid QA team would of caught this.
|
| A higher paid QA might of told management, hey this is a very
| high risk change. If we're going to roll this out let's limit
| it to reduce the numbers of people affected.
|
| If you on shore your core development, but outsource all of
| your QA, I'm forced to assume you value QA less.
| ctxc wrote:
| "If we're going to roll this out let's limit it to reduce
| the numbers of people affected." Ime this is something
| senior developers would themselves do - and not only for
| changes they deem "high risk", but also by default.
|
| I say this because this case a data file was changed.
| Probably done thousands of times without an issue.
|
| QA would have never said "we need a staged rollout for
| this". Developers and those who set the process should do
| it.
| thomasjudge wrote:
| The flawed version was only up for about an hour and 18
| minutes. In that time it was able to have the impact that it
| did
|
| https://www.crowdstrike.com/blog/falcon-update-for-
| windows-h...
| lambdaone wrote:
| I had to get someone life-critical medicine yesterday. My GP
| practice's computers were down because, presumably, of
| Crowdstrike. Manual pen-and-paper processes saved the day.
|
| I wonder how many people didn't get so lucky?
| Baeocystin wrote:
| I was due to pick up my ADHD meds yesterday, and couldn't, for
| crowdstrike and reversion to paper reasons.
|
| For me, it's mildly annoying, but I've got an emergency supply.
| The lines of truly desperate people with much more urgent needs
| than mine were long, and there was a lot of crying and despair
| in the lobby. I can only imagine the situation in larger
| cities.
| dylan604 wrote:
| Was this a first time prescription? If not, was there something
| that prevented the script from being filled earlier?
| norgie wrote:
| Is that relevant?
| dylan604 wrote:
| If there's something that says you can't fill the script
| until X days before previous runs out because of some
| "regulation", then yes, it is relevant. It's just another
| example of short sighted JIT style expectations that only
| compound situations like this where the supply chain is
| interrupted.
| whalesalad wrote:
| Siblings miss critical life-extending treatment because the
| hospital IT department didn't architect their endpoint update
| strategy correctly. This should have rolled out in small,
| incremental steps to verify no failures. A mass "select * from
| hosts" global update with no testing (even if the vendor says
| it's good) is entirely foolish.
|
| Hopefully folks learn from this.
| bongodongobob wrote:
| In the 20 years I've been in IT, in a variety of industries,
| with all the legacy manufacturing systems I've dealt with, I've
| never seen a software patch that blue screens all computers.
| Firmware, yes, rare, but never software. This is way outside of
| the norm.
| whalesalad wrote:
| Anything operating that low in the stack is going to be a
| risk if shit goes wrong. It's essentially a untrusted third
| party kernel update. Any sysadmin worth a dime knows not to
| blindly upgrade something like a kernel without staging it
| first - this is no different.
|
| But honestly windows admins don't understand the systems
| they're maintaining.. so this was inevitable.
| bongodongobob wrote:
| You have no idea what you're yapping about. Bye.
| CAP_NET_ADMIN wrote:
| Maybe try reading what happened during the Crowdstrike fiasco
| and then comment about it. Crowdstrike auto updates itself, the
| agent allows you to select update cadence, but this was an
| update to the "channel" files which auto update themselves a
| few times per day and you don't have any control over it.
| whalesalad wrote:
| Doesn't matter. You had control over instituting it in your
| environment. The onus is on you.
| schroeding wrote:
| No you didn't. External audits and compliance forced you to
| install an EDR, said EDR had no option to defer this kind
| of update. Crowdstrike sold itself to the suits as said
| EDR. What do you do, ignore both your management and the
| security checklist given by the government?
| CAP_NET_ADMIN wrote:
| I've personally trialed Crowdstrike for my company (around
| 30 people) and found it a buggy mess, especially on non-
| Windows platforms - after the trial was over we decided to
| not use it. However this is not the case for most
| companies, feds, auditors or both will force you to use
| this BS and there's absolutely nothing you can do.
| russdill wrote:
| Just to be a little fair here, healthcare providers are a major
| target of ransomware. How many ransomware attacks has crowd
| strike thwarted?
| hypercube33 wrote:
| Why isn't this stuff air gapped?
| ikekkdcjkfke wrote:
| Should be, or with a clearly defined interface against the
| open web
| ctxc wrote:
| It's not "fair" that a product can brick your appliances if
| they have previously protected against it.
|
| In my mind, that is because the protection is what they're paid
| bags of cash to do.
|
| If Crowdstrike was a charity I might find myself agreeable on
| your description of fair in favor of Crowdstrike.
| akira2501 wrote:
| Why don't EMA/EHR systems have write only encrypted journal
| storage that they use to guarantee data safety against this
| problem?
|
| I mean, just for _basic_ audits, you would hope to have that.
| If ransomeware can destroy your entire facility, than an angry
| insider can do much worse.
| blackeyeblitzar wrote:
| There's are lots of reports like this. In Boston, Mass General
| and Brigham both shut down normal operations from what I heard.
| ugh123 wrote:
| Article is void of any information about _why_ they missed the
| treatment that day, just that their appointments were canceled,
| and thats it. What terrible reporting.
| justinclift wrote:
| Ugh: Error 451 It appears you are
| attempting to access this website from a country outside of the
| United States, therefore access cannot be granted at this time.
|
| Fortunately the archive.today link works.
| 1vuio0pswjnm7 wrote:
| Works where archive.ph is blocked:
|
| https://web.archive.org/web/20240720155219/https://www.kiro7...
___________________________________________________________________
(page generated 2024-07-20 23:07 UTC)