[HN Gopher] Software bugs that cause real-world harm
___________________________________________________________________
Software bugs that cause real-world harm
Author : thepbone
Score : 127 points
Date : 2023-05-29 14:59 UTC (8 hours ago)
(HTM) web link (pointersgonewild.com)
(TXT) w3m dump (pointersgonewild.com)
| RobotToaster wrote:
| The infamous bug(s) in the Fujitsu system provided to the British
| post office caused several suicides.
| https://en.wikipedia.org/wiki/British_Post_Office_scandal#Ho...
| goda90 wrote:
| >The Post Office resisted the SPMs' reports of faults in the
| system, insisted that the SPMs make up any shortfall of money
| and, when asked by an SPM, denied that other SPMs had reported
| problems.
|
| Seems more like a very toxic culture of shifting blame led to
| those suicides, with the bugs just being a catalyst.
| yifanl wrote:
| The bugs were causing very real financial distress it sounds
| like, if the software was partially responsible for
| determining their continued employment
| jmclnx wrote:
| >The screen on my phone showed that the alarm I had set was in
| the process of ringing, but for some reason, the phone wasn't
| vibrating or making any sound.
|
| And this is kind of an example of why it will be impossible to
| mitigate Climate Change.
|
| When I was young, clocks did not requite any power from the grid.
| It wound them, set the alarm and you heard it. Now we have
| faucets, toilets that use power to turn on/off and flush. Almost
| everything that was manual (mechanical) is now powered. That
| power needs to come from somewhere. In these examples, there is
| no real reason to have faucets, toilets and alarm clocks
| connected to the grid.
|
| Yes, I know in the case of the Cell Phone Alarm, it is probably a
| wash, but a mechanical clock in your house is probably a good
| thing to have.
| tnorthcutt wrote:
| > And this is kind of an example of why it will be impossible
| to mitigate Climate Change.
|
| I find it very interesting (genuinely!) that your reaction to
| the linked article was to conclude (I think?) that it will be
| impossible to mitigate climate change.
|
| Is your position, broadly stated, something like "because these
| days many more things require electricity than at some point in
| the past, our demand for electricity will be too high, and this
| will prevent us from mitigating climate change"?
| jrockway wrote:
| It's most likely to me that many more people have overslept
| because they forgot to wind their clock than have overslept
| because their iPhone hit a weird bug where the alarm sound
| didn't play. Simply because mechanical clocks have been around
| much longer.
| SoftTalker wrote:
| Also at a certain time in my teenage years I could sleep
| through a ringing alarm clock with absolutely no awareness of
| it. The spring for the bell-ringing mechanism would unwind
| after maybe 30 seconds or so, and the ringing would stop, and
| I'd sleep completely through it.
|
| Not sure the powered alarms would have done any better, but
| maybe after 10 minutes of ringing it might have woken me up.
| alpaca128 wrote:
| > a mechanical clock in your house is probably a good thing to
| have.
|
| There are much better and more reliable options. Solar powered
| watches that can run months without recharging, some digital
| watches run a decade with a single battery. A mechanical watch,
| on the other hand, has to be wound regularly and is usually far
| less accurate.
|
| I like automatic mechanical watches but I don't think they can
| compete in any technical aspect, that's not the reason one
| would use them nowadays.
| sitkack wrote:
| Everything is mission critical to someone.
| joezydeco wrote:
| Almost any and every story you'll see in this thread has already
| been collected and discussed in the RISKS mailing list and
| archive.
|
| There's nearly 38 years of articles waiting for you to peruse,
| all lovingly curated by Peter Neumann.
|
| https://catless.ncl.ac.uk/risks/
| Zealotux wrote:
| Sometimes, even perfectly functional code can do harm as well:
| https://www.freecodecamp.org/news/the-code-im-still-ashamed-...
|
| https://news.ycombinator.com/item?id=12965589
| lostlogin wrote:
| Philips sell MRI scanners. I used to do angiograms with them.
| My workflow was to start a cheap and cheerful scan that showed
| the location of the injected contrast. As it got close to the
| vessel of interest I'd click and hold the button that started
| the high resolution angio run. When it was at the vessel of
| interest I'd lift my finger off the button and the scan would
| start.
|
| This all happened over about 10-20 seconds after the injection.
| Blood moves quickly. Start the scan too early and you miss
| everything, too late and you get veins rather than arteries.
|
| In a point release update with no mention in the release notes
| they changed the behaviour of the scanner - it then started the
| scan on clicking down in the button, not on lifting off.
|
| A time critical scan on a patient due for theatre was messed up
| by me. Their renal function was poor and the scan couldn't be
| repeated for a few days.
| renewiltord wrote:
| Great story. It's interesting that regulatory barriers create
| these large companies that have no user interest at heart.
| Bundled software and hardware from a hardware company:
| guaranteed disaster. Perhaps only Apple has succeeded at
| this.
| sitkack wrote:
| That sounds like a Boeing playbook. Changing action on button
| down vs up is a HUGE deal.
| levihaku wrote:
| Only if the function was to harm in the first place.
|
| Every single antivirus program is more like malware than actual
| helpful software, because malware is in fact a spook and best
| antimalware we currently have is a functioning human brain.
|
| It's really sad that pharmaceuticals, akin to antimalware
| companies are allowed to get away with any of this.
|
| When brain doesn't work, such punyware like antivirus does
| nothing. Study done in around 2007 found that 35% of IRS
| workers will fall victim to social engineering... Ending up
| directly giving up their password, imagine the fun you can do
| if you have a password like that and you needed to run no
| malicious code to steal such information.
| kernal wrote:
| >Unfortunately, Android [9.0] suffers from poor user interface
| design in a few areas, and one of the most annoying flaws in its
| user interface is simply that the stock Android OS doesn't have
| flexible enough options when it comes to controlling when the
| phone rings, which is one of the most important aspects of a
| phone.
|
| Do Not Disturb, at the time of Android 9, did support controlling
| when the phone rings.
|
| >Bedtime mode is quite useful, but I still have the other problem
| that my mom could decide to randomly call me in the daytime as
| well, and unfortunately I rarely want to take her phone calls.
| However, I also don't want her to end up homeless or in jail
| (which has happened before, but that's a story for another time),
| and so I don't want to block her and completely lose the ability
| to receive her calls. This results in me having to almost always
| have my phone set to "do not disturb", so that I don't have to be
| disturbed at random times by unwanted phone calls
|
| Could he not have used a custom ringtone for his mother?
| pw6hv wrote:
| *she
| soult wrote:
| It's interesting that the author picked the home depot delivery
| as an example, because I don't think it was caused by software
| bugs at all.
|
| The delivery notifications right at the end of the delivery slot
| happened because the driver has an unrealistic schedule, and to
| hit their metrics, they will mark deliveries as "delivered"
| rather than missing an assigned delivery slot.
|
| And that the expensive two-man-handling and delivery to the
| apartment service got "lost" between seller and subcontractor is
| more likely cost optimization and banking on most customers not
| complaining hard enough to actually force them to provide the
| paid-for service. "Software bug" being a convenient excuse the
| subcontractor can give to the rightfully enraged customer.
| maxime_cb wrote:
| Author here.
|
| The reason I tend to think it's a software bug is that it seems
| that the system to dispatch deliveries is automated. The
| subcontractors get their orders from some kind of computerized
| system, it seems. That system seems to systematically fail to
| specify when they are to carry items indoors/upstairs. Whether
| that's due to negligence or intentional malfeasance, don't
| know.
|
| What I do know from experience is that there are numerous bugs
| on their website, besides the "unknown error" problem I've
| listed. It just seems like really shittily built software... So
| I would tend to think there's an issue with really poor
| software engineering practices at that company.
| acenes wrote:
| A recent example:
|
| https://www.nytimes.com/2023/05/26/science/moon-crash-japan-...
| pards wrote:
| Knight Capital [0] should serve as a warning to devs and ops
| alike. They lost about $450 million in about 30 mins due to some
| poor coding, a botched deployment, and botched rollbacks.
|
| [0]: https://en.wikipedia.org/wiki/Knight_Capital_Group
| superice wrote:
| Story time! When I was a much more junior programmer I once
| received a call from a client, who asked me if I could double
| check the code sending the container weights from a container
| terminal to the ship planning system. She explained a captain of
| one of the ships had called the planners and remarked he was both
| deeper in the water and tilted quite a bit more than he
| anticipated.
|
| So I do my research, and it turns out we were sending weights
| including the metal of the container itself from the one system,
| but interpreting them as net weights, so excluding container
| weight itself. So we were off by about 3000kg per container,
| which is bad enough if the container is 25000kg total, but even
| worse when transporting empties, where we were off by 100%.
|
| Thank goodness it is drilled into captains that there are strict
| limits when it comes to stability, and I have never met a captain
| who will depart if they are not absolutely positive about the
| stability of their ship, but man I have spent a good few nights
| lying awake thinking about what could have happened.
|
| I thought I was just working on some boring logistics software at
| the time, where the worst that could happen was losing a
| container for a day or so. It was a rude awakening.
| SoftTalker wrote:
| > I have never met a captain who will depart if they are not
| absolutely positive about the stability of their ship
|
| Guess you have not dealt with air freight pilots in the third
| world or bush country?
| 6D794163636F756 wrote:
| Or that's why they wanted the check. They felt like it was a
| noticable problem so they got someone to look
| WalterBright wrote:
| I doubt air freight pilots would live long if they didn't pay
| attention to weight distribution and the weight being
| properly secured.
| BalinKing wrote:
| I had interpreted the GP as specifically talking about
| maritime captains, which I imagine could be a very different
| boat (pardon the pun).
| throwawaymaths wrote:
| > I have never met a captain who will depart if they are not
| absolutely positive about the stability of their ship
|
| A lesson learned in 17th century Sweden.
| melx wrote:
| Care to share what happened in 17th century in Sweden?
| alpaca128 wrote:
| They built a ship called _Vasa_ which was so loaded with
| cannons that it sank immediately after leaving the harbor
| on its maiden voyage:
| https://en.wikipedia.org/wiki/Vasa_(ship)
| daemin wrote:
| Thankfully now it makes an interesting museum.
| throwawaymaths wrote:
| IIRC it was really top heavy and would have sunk anyways
| without the cannons, as the geometry and ballast space
| ratios were hopelessly f'ed... I remember something about
| testing it by running sailors from one side of the ship
| to the other and the captain? Admiral? wanted to nope out
| of it but the orders to launch stood.
| fbdab103 wrote:
| Wow, even with restoration, those pictures make it look
| like it is in shockingly good shape. I would expect
| hundreds of years in seawater to leave essentially
| nothing remaining.
| rippercushions wrote:
| The Baltic Sea is cold and too low in salt for sea worms,
| so wooden wrecks survive much longer than anywhere else
| in the world.
| pcdevils wrote:
| Til. Cool
| WalterBright wrote:
| Not entirely learned. There was a french ocean liner built in
| the last century that installed far too much marble in the
| first class areas near the top of the ship.
|
| They had to rip it all out.
|
| Sorry, I forgot the name of that liner.
| anonymousDan wrote:
| If you want to see a really sad example, check out the Royal
| Mail/Fujitsu postmaster scandal in the UK, where postmasters
| (franchisees) ended up being thrown in jail, losing everything
| they owned, families destroyed, suicides, etc. All because of
| bugs in Fujitsu accounting software
| (https://www.bbc.co.uk/news/business-56718036)
| meghan_rain wrote:
| > Thankfully, Android now has "bedtime mode" feature, which
| allows me to make it so that phone calls won't cause my phone to
| ring between 10AM and 8:30AM. If my mom happens to die in a
| hospital in the middle of the night, I'll just have to find out
| and be sad the next day.
|
| wtf
| usui wrote:
| Is this a comedic "wtf"? If nt, I think you should recognize
| the flippant tone, not to be taken too seriously.
| kevin_nisbet wrote:
| I think my message is, don't lose sight on the mission of the
| software that you're shipping.
|
| My story is, I found a bug in some new equipment that would've
| broken 911 calls. While other engineers were just trying to
| reboot random equipment to make the problem go away, I insisted
| we pause to figure it out. Turns out, due to a couple of bugs,
| the new network equipment could only handle a couple of 911 calls
| before failing.
|
| This was for a national cellular network... so 911 was kind of
| important.
| Aperocky wrote:
| Did you get recognition for recognizing the problem and dealing
| with it?
| SoftTalker wrote:
| Good post, but the second story about the Android phone was more
| a complaint about desired features that the phone didn't provide,
| more than a bug. I'd put that in a different category. Software
| can't be everything to everyone.
| TeMPOraL wrote:
| On the other hand, this technology has been around for 30
| years. The feature in question reaches back to the earliest
| mobile phones. Not _smartphones_ - think earlier than Nokia
| 3310.
|
| On the gripping hand, "silence.mp3" has been the solution for
| the author's "mom problem" for a good 20 years too. I hope they
| don't mean to imply you can't assign individual ringtones to
| contacts on stock Android? Even the K800i I had as a teenager
| supported that.
|
| On the _yet another_ hand (we 're moving fast into octopod
| territory here), for a moment here I was wondering what the
| author was smoking with the "ringtone & notifications"
| complaint. I don't remember seeing an Android phone that did
| not have separate volume slider for each. Then I realized, I
| never actually owned a _stock_ Android phone. Still, to support
| the author 's point against Google specifically, Samsung
| managed to not have this problem for at least 10 years now.
| maxime_cb wrote:
| > I was wondering what the author was smoking with the
| "ringtone & notifications" complaint. I don't remember seeing
| an Android phone that did not have separate volume slider for
| each.
|
| This is why I included a screenshot of the volume sliders my
| Google Pixel displays. Samsung doesn't use stock android OS,
| and some Samsung users tend to assume every android phone
| works the same.
| SoftTalker wrote:
| My silence.mp3 story on Android was this: I was using an
| android device to play background music through a PA during
| breaks in activity. Every time I'd start a song there was a
| "pop" I assume by the audio output amplifier switching on or
| waking up.
|
| My solution to avoid this was to play "silence.mp3" in a
| repeat loop in another app, and then my music app would play
| over that and there were no pops when I started a song. IDK
| if the pop problem has been fixed, or whether it was a
| software or hardware issue.
| josephcsible wrote:
| > I hope they don't mean to imply you can't assign individual
| ringtones to contacts on stock Android?
|
| You can do that.
| hoosieree wrote:
| I teach engineering ethics and while the example NSPE cases[1]
| skew more toward civil engineering, there are some cases relevant
| to software and computer engineers. But I find it helpful to draw
| examples from current events.
|
| For example, use of facial recognition tech in policing. TV and
| movies give the impression that a satellite can identify the perp
| with 99.4% accuracy if you yell "enhance" enough times.
| Meanwhile, in reality we're happy when our classifier can tell a
| "3" from a "B" in a real image.
|
| [1] https://www.nspe.org/resources/ethics/ethics-
| resources/board...
| JdeBP wrote:
| There's also the decades-long tale of the software bug that put
| some U.K. sub-postmasters in prison, bankrupted others, and
| caused a few to commit suicide.
| dm_me_dogs wrote:
| For anyone wanting more context:
| https://en.wikipedia.org/wiki/British_Post_Office_scandal
| bigbacaloa wrote:
| There are engineering ethics courses?
| is_true wrote:
| Engineering and society. that was the name in my college
| WalterBright wrote:
| There was an article 3 days ago about rampant cheating in an
| ethics class.
|
| https://news.ycombinator.com/item?id=36082650
| 0xffff2 wrote:
| It's a single one-term course in most universities I believe,
| but engineering ethics is a required subject for ABET-
| accredited (the standard in the US) engineering degrees.
| dgoldstein0 wrote:
| I was not required to take such a course, but it did exist
| in our catalog.
|
| I suspect the main requirement was actually fulfilled by a
| ~week or two in the intro course, where we talked about
| Therac-25 and maybe another disaster or two.
| tomwojcik wrote:
| Is your syllabus available online? Engineering ethics sounds
| very interesting and it's the first time I hear about such
| thing.
| hoosieree wrote:
| Nothing public-facing, but here's the gist:
|
| It's a 6-week course. Each week, students respond to a prompt
| about an aspect of engineering ethics. The main task is to
| link real life examples to concepts from a professional code
| of ethics (we use the NSPE code[1]). NSPE also has anonymized
| and searchable "case studies"[2] with examples of how to
| apply this approach of "summarize the case and find the
| relevant parts of the Code of Ethics which apply here".
|
| In a typical course, I ask them to use real-life examples
| such as these: - Snowden leaks (privacy and
| consent of the governed) - Henrietta Lacks (ethics in
| biotechnology and patient rights) - Roger Boisjoly
| (whistleblowing) - Volkswagen emissions scandal
| (environmental consequences)
|
| After 4-5 of these drills, they research a case on their own
| and present about it for a group project. Class discussions
| dive into questions like "what would you do differently in
| this situation?" or "what if it was your family?" and "how
| would the consequences have been different if X instead of
| Y?"
|
| I try to emphasize that most situations don't have a single
| obvious ethically correct choice. It's more important for
| students to learn that codes of ethics exist, and that they
| can treat them as a decision-making framework.
|
| [1] https://www.nspe.org/resources/ethics/code-ethics
|
| [2] https://www.nspe.org/resources/ethics/ethics-
| resources/board...
| [deleted]
| anonymousiam wrote:
| Well written article, but most of the "harms" described (aside
| from the Therac-25) are a direct result of attempting to replace
| human-to-human contact with technology.
|
| 1) Instead of relying on a phone alarm to wake you up, use a
| device designed for that purpose, or request a wake-up call (on a
| landline, if you have one).
|
| 2) Overcome the conflict between having a phone and not wanting
| to answer it.
|
| 3) When unusual and specific instructions are required, make sure
| you speak with a person to confirm them.
| taneq wrote:
| I don't think "software is unreliable and you should never
| trust it" is the outcome we should be hoping for here. If I
| burn to death in a Ford Pinto, would you tell me I should have
| ridden a horse?
| intelVISA wrote:
| Depends, was the horse sold to you with a potentially fatal,
| albeit financially acceptable, defect?
| groestl wrote:
| And Therac-25 was trying to remove hardware interlocks from
| Therac-6 and Therac-20, to fully rely on software, AFAIK.
| tczMUFlmoNk wrote:
| I agree with you that the trajectory of removing human-to-human
| contact, even as a fallback, is causing harm. And I agree with
| your interpretation of your point (3): having an option to
| speak to a human would have gone a long way here.
|
| I'm not sure that I agree with your point (1): a modern
| smartphone _is_ a device designed for that purpose, among other
| things, and it is virtuous to use a multi-purpose device for
| this instead of building, transporting, and disposing of a
| dozen bespoke devices for each such purpose.
|
| But reducing the complex interpersonal relationship between the
| author and her mom to just "overcome the conflict" is a really
| low-bar take. It feels callous to suggest that it would be so
| easy, or to assume that she's not trying to do this, or even
| that this is the right path forward for her. Technology should
| serve us and adapt to the diversity of our needs, as we change
| as people and those needs change with us.
| grepLeigh wrote:
| Great name for this kind of blog (pointersgonewild)
|
| > Long story short: a software bug caused the machine to
| occasionally give radiation doses that were sometimes hundreds of
| times greater than normal
|
| Oh my god.
|
| I have 15+ years working with "mission critical (software)
| infrastructure" and thought my job was important/hard because I
| played shepherd for production database fleets. Certain kinds of
| mistakes could bankrupt the business, so I had to make systems
| resilient to human errors.
|
| Today, I run a 3D printer software startup. Thinking deeply about
| the safety mechanisms I need to control machines running at
| 250-300degC.
|
| Precision errors when shooting radiation at a person is a whole
| different level. Wow.
| masklinn wrote:
| The Therac-25 story is pretty wild, especially as it was a
| reuse (with modification) of an existing codebase, but for cost
| savings the -25 was designed without hardware interlocks, under
| the assumption that the software would run fine, as it had run
| fine on the -20.
|
| Turns out the software had run fine on the -6 -20 _because_ of
| the mechanical interlocks which prevented it from critically
| fucking up, but there was no reporting built into the software
| interlocks, so no way to know when they 'd triggered. And that
| was before additional modifications were added for the -25.
| anonymousiam wrote:
| Not all that different from the Ariane-5 maiden launch,
| except for human lives being at stake vs. billions of Euros
| (adjusted for inflation).
|
| https://en.wikipedia.org/wiki/Ariane_flight_V88
| jrockway wrote:
| I think your 3D printing company basically exists because of
| Therac-25 indifference in the 3D printing community. We could
| put motor position encoders on the axes to actually detect
| crashes. But those things are like $20 per axis, and your
| microcontroller needs a quadrature encoder, so we say "fuck it,
| we'll do it with OpenCV". It's good when it's good, I guess!
| Palomides wrote:
| 3d printers are very safe and reliable, mine detects crashes
| via the current through the motor controller with no extra
| hardware (not that it's ever crashed)
|
| cost/benefit analysis is the heart of engineering, and that
| applies to redundant safety and error detection hardware as
| well
| grepLeigh wrote:
| Defects in 3D print jobs are usually usually related to model
| design, slicing parameter decisions, and _occasionally_ a
| printer part needs replacement /servicing (like belts).
|
| So far, firmware crashing hasn't been an issue. I'm
| supporting open source firmwares though (Marlin, Klipper, and
| soon RepRap/Duet) so they're VERY battle-tested. Maybe this
| is an issue for proprietary closed-source firmware?
| deanCommie wrote:
| > I wish that Android had an option to set a specific person to
| never cause the phone to ring, and it seems like that should be
| an easy feature to implement that would have a real positive
| impact on the quality of lives of many people, but I digress.
|
| FWIW, Android does have that feature - at least for sure the
| Pixel Launcher (that the author uses) does: You can set some
| contacts as "Favourites" and have it configured that Favourites
| get to bypass Do Not Distrurb.
|
| Good blog nonetheless!
| awesome_dude wrote:
| I'm surprised that the Metric <-> Imperial bug that caused NASA
| to lose 125 million wasn't mentioned
|
| https://en.wikipedia.org/wiki/Mars_Climate_Orbiter
| contingencies wrote:
| It's certain there is >$10B loss per annum on metric-imperial.
| Think about it: human error, conversion process losses,
| training materials, advertising materials, two physical
| versions of everything being produced, two physical versions in
| the supply chain...
|
| I for one plan to operate a metric shop in the US. Using
| imperial outside of supplier interface will be a formal
| warning. Will see how that goes...
| OnlyMortal wrote:
| A story related to me from a friend...
|
| A torpedo system was designed not to hit the submarine that fired
| it. It would detonate if it was aiming itself back.
|
| So, testing came along, the torpedoes armed and... the sub turned
| itself around.
|
| Bang.
| meghan_rain wrote:
| Sorry, I don't understand what happened?
| orbz wrote:
| I think the implication is that the torpedos had not yet left
| their tubes, so when the sub turned around with them still in
| it they registered that as a situation where they should
| destruct.
| isidor3 wrote:
| Sounds like the torpedoes were armed, but not yet fired. So
| the sub itself turning around in a circle triggered the self-
| destruct mechanism while the weapons were still in the tubes.
| hlieberman wrote:
| I talked about this several years ago[1], but I strongly believe
| that we, as a profession, don't invest nearly enough into
| thinking through all the possible consequences of the things that
| we design. It's easy to write off that what you're doing "doesn't
| matter" or "can't hurt", but the world is far too interconnected
| for us to be so nonchalant about our work.
|
| [1]: https://blog.setec.io/2015/11/01/ethics.html
| giantrobot wrote:
| There's also the issue that code _designed_ for one purpose
| often gets used for different purposes. You literally can 't
| think through all of the consequences for a design because the
| total number of combinations is enormous. Even when you do
| think through all of the possible combinations you can't know
| the runtime state of all of those combinations.
|
| Formal verification of software exists but can only really be
| trusted if running on hardware with some multiple redundancy
| and formal verification of its own.
| renewiltord wrote:
| I think we do actually. In terms of economic value / accidental
| death this is probably the most rigorous engineering
| discipline.
|
| The big difference with software engineering that other
| engineering disciplines fail at is asking the question "Is it
| worth the risk?" and then answering it well.
|
| For instance, historically and presently, bridge builders
| accept a much higher risk of killing someone than software
| engineers.
|
| Given the risk that most bridge engineers take, most software
| engineers would instead just do something else.
| keyringlight wrote:
| Mike Monteiro did a talk [1] along these lines - "How Designers
| Destroyed the World".
|
| [1] https://www.youtube.com/watch?v=qIcM21l61TE
___________________________________________________________________
(page generated 2023-05-29 23:01 UTC)