[HN Gopher] Alarm overload is undermining safety at sea as crews...
___________________________________________________________________
Alarm overload is undermining safety at sea as crews face thousands
of alerts
Author : geox
Score : 133 points
Date : 2026-01-25 12:40 UTC (10 hours ago)
(HTM) web link (www.lr.org)
(TXT) w3m dump (www.lr.org)
| sklargh wrote:
| When everything is important nothing can be important. I was
| particularly struck by the reduction of crew rest to service
| alarms. I'd sort of thought this was a problem for watch standers
| only but it makes sense that some alarms require a specialist or
| the crews are too thin. Heck, I bet the captain, engineer and
| first mate all have monitors in their staterooms.
| Hizonner wrote:
| Ah, but you see, the alarms from the device _I_ am building are
| _most_ important, because somebody might sue _me_ if I can 't
| shift responsibility onto the user.
| abanana wrote:
| Same sort of problem we have in modern cars? Speed, lane
| assistance, blind spot, etc, sometimes apparently beeping for the
| hell of it.
|
| For some it's distracting and frustrating, even increasing
| aggression and thereby increasing the risk. For others it breeds
| complacency, a "boy who cried wolf" scenario such that the alarms
| become meaningless. Either way, it doesn't work as intended.
|
| Interesting to know ships have followed the same pattern,
| apparently to a worse extent. I wonder how many more walks of
| life, and industries, are suffering in the same way.
| nsavage wrote:
| My Volkswagen has assistance features which routinely fail on
| snowy days and can't seem to be disabled. The best you can do
| is disable them for a minute (!) at which point they start
| blaring again. Its ironic because the time you need the most
| focus is the time the car lets you focus the least.
| DoingIsLearning wrote:
| It can absolutely be disabled they just wouldn't get the same
| brownies points in EuroNCAP by allowing _you_ to disable it.
|
| If I spent more than 50k on a car like that, I would
| absolutely return it and file a complaint.
|
| Car companies care a great deal about after sales stats. This
| trend will continue because we as users on average tolerate
| it.
| Nextgrid wrote:
| Get yourself a VCDS cable & software and disable that shit
| for good.
| t0mas88 wrote:
| BMW has the same issue but luckily still buttons to disable
| them. Snow will quickly result in "Forward collision warning
| failure" and "Blind spot detection failure" and if more snow
| "Lane assist failure" because the sensors get covered in
| snow.
|
| Oh and before you even start driving let us "bing!" you with
| a message that the temperature is below 4C. As if you didn't
| know that already.
| B1FIDO wrote:
| I rented a car last July, and I specifically picked out a small
| one because I wouldn't need to carry any cargo or passengers
| around.
|
| As soon as I drove off the lot, 3 warning indicator lamps lit
| up, including "Tire Pressure" so I stopped at a service
| station, thought for a moment, then drove back to the rental
| lot.
|
| The other indicator had something to do with crash protection,
| and I think we worked out how to disable the system. After
| putting air into my tires, I was good to go.
|
| So I'm thankful that those lamps indicated some actual
| conditions. I always kind of make a point of taking out the
| Owner's Manual and leafing through it, however briefly, just to
| see that it covers everything. They're still fairly
| comprehensive. I really appreciate that.
| bluGill wrote:
| Any alarm that causes to take the correct action is good.
| However if it causes you to take the wrong action it is bad.
| If the engine lost all oil but you choose to fill the tires
| because that is the alarm you choose to pay attention to that
| is bad. Normally engines don't lose oil and so going to fill
| the low tire is the right call. (If the tire is very low stop
| now is the right answer)
| Litost wrote:
| I had the misfortunate of needing to hire a car in the UK last
| year. Ended up with an entry level 2025 Kia (Ceed Estate).
| Compared with the 2012 Audi A4 I'm used to driving it was a
| nightmare.
|
| Similar experience, lots of flashing and beeping which is just
| distracting whilst also being wrong often enough to be really
| annoying (this is a known problem with speed limits).
|
| Exceeding the speed limit, needing to change gear and by far
| the worst, active lane assist which pushes you back into your
| lane if you cross the white line without indicating (I only
| found this out afterwards as the hire place didn't mention it
| or leave a manual) and something which can happen frequently if
| you're driving down narrow country roads where indicating
| wouldn't have seemed appropriate and may just confuse others.
|
| I spoke to one of the mechanics at my local garage who said you
| can't permanently turn these features off as they turn back on
| when you start the car.
|
| I wonder if anyone has who's had an accident caused by being
| distracted by all these alarms has successfully sued?
| Nextgrid wrote:
| > you can't permanently turn these features off
|
| You most likely can if you have the factory diagnostics tool
| or a good aftermarket reimplementation.
|
| These features are often mandated on a per-region basis, so
| the configuration bits are definitely there.
| maccard wrote:
| My car will beep every 15 minutes when the washer fluid is low.
| "Low" being some arbitrary value, and not even remotely close
| to empty. Last time it happened. I had just started on a long
| motorway drive on a clear day and it just pinged and pinged and
| pinged. The beep is actually louder and harsher than the
| emergency anti collision break alert.
|
| I've also never used as much windscreen wash in any other car
| as I have in this one
| Nextgrid wrote:
| Wait until you see an actual failure; you will get _individual_
| alerts for every dependent system as if the dependent system
| failed directly. A single communication glitch with the ABS for
| example will trigger 5+ alerts for ESP, lane keeping assist,
| and so on... and even things like the check engine light,
| despite the engine being totally fine and maybe just operating
| in a degraded mode.
| michaelt wrote:
| Excessive alarms aren't just a problem at sea.
|
| There have also been fatal aviation accidents where there's a
| problem with a common system (a dip in the power supply or
| hydraulic pressure, or a problem with a critical sensor) and
| dozens of systems sound alarms at the same time [1].
|
| And for a technology example, a database server disappearing
| might raise a single alarm, but the applications that rely on
| that database might raise countless alarms as attempts to connect
| fail over and over again.
|
| [1]
| https://en.wikipedia.org/w/index.php?title=Air_France_Flight...
| ApolloFortyNine wrote:
| This is the flight where one pilot tried to pull up to recover
| from the stall, and the warning for dual input (which Airbus
| just averages together) was snoozed by the system yelling about
| the other errors and was reduced to a light they didn't notice.
| The captain commented towards the end"no don't climb". The
| stall alarm was the one the system chose to display over all
| others and was mishandled (by the pilot who didn't know how to
| recover from a stall).
|
| Boeing there's physical feed back, when one control moves so
| does the other.
|
| This was not the first time pilots were having conflicting
| input without noticing.
|
| >https://bea.aero/uploads/tx_elyextendttnews/annexe.01.en.pdf
| kortilla wrote:
| IMO that's the wrong take about that crash. The stall warning
| stopped once the attitude was above a certain amount, which
| was an insane decision on airbus's part.
|
| You can see in the CVR that the stall indicator stopped many
| times despite them being in a stall the entire time. The
| pilot (like every other pilot) knew how to recover from a
| stall on paper. But he had the plane telling him his airspeed
| was good (frozen tube) and that bringing the pitch down was
| causing a stall.
| tetha wrote:
| Good alert deduplication and dependency rules are worth so
| much. "Dear alerting, don't start throwing a fit about those
| 600 systems over there if you can't even reach the firewall all
| traffic to those systems goes through". Suddenly you don't get
| throttled by your SMS provider for the volume of alerts it
| tries to send, and instead just get one very spicy message.
|
| Snark aside, this also impacts resolution time, because done
| well, this instantly points out the most critical problem,
| instead of all the consequences of one big breaking. "Dear
| operator, don't worry about the hundreds of apps, the database
| cluster is down".
| bruce343434 wrote:
| This is why compilers for e.g. c++ should just halt after the
| first error, instead of spewing pages of template failures
| because of a typo elsewhere
| kyralis wrote:
| They used to, but I wouldn't want to go back to that.
| Believe me, compilers that continue and try their best are
| a massive improvement in many cases, allowing you to fix
| more issues between compilation attempts.
| somat wrote:
| Perhaps, I don't really program much c/c++. but in my
| experience most of the subsequent errors are due to the
| first error. So even where there might be several places
| I could fix the code my standard operating practice is to
| find the first error, fix that and see what it cleans up.
|
| But like I said, I am not much of a C programmer. The
| compiler authors feel strongly about pushing past all
| possible errors and keep doing it so perhaps there is
| merit to this practice. but it bugs the heck out of me.
| parados wrote:
| I think that you are confusing ACARS messages (those
| transmitted from the aircraft via airband radio or satellite)
| with ECAM messages (those that are displayed to the pilots).
| The former are mainly for operational reasons, for example so
| that ground engineers get forewarning of a failing component.
| ECAM messages are flight safety critical messages requiring
| pilot action and they are organised by priority.
|
| ECAM messages are recorded by the DFDR/CVR but are not normally
| transmitted via ACARS. Pilots were not normally aware of ACARS
| messages (see MH370).
|
| Your point is a fair one though. In the case of AF447 the crew,
| against their training, made no attempt at all to work their
| way through the multiple ECAM messages with the appropriate
| checklist so they died. The final report on QF32 shows what a
| high workload this can be.
|
| Since AF447 was lost for nearly two years the ACARS messages
| were all that the investigators had to go on until they found
| the wreckage and the DFDR and CVR. Incidentally, one problem
| with those ACARS messages was that they are only timestamped to
| the minute and may arrive out of order which makes the
| interpretation of them more difficult.
| KernalSanders wrote:
| It's hitting cars too. Our 2026 Chevy equinox has so many
| alarms and chimes that we have zero control over, the worst one
| has no notification of what its alerting us of.
|
| As a pretty decent driver, this terrifies me, because I first
| think, "What am I missing?" But then it hits me - these alarms
| and chimes are breeding generations of drivers, not just young
| ones, who are grossly incompetent and should not be driving.
|
| The fact that I cannot control what alarms go off is asinine.
| And they put a lock on how low you can turn the chime volume.
| So, basically, you're telling me that I have to harass my
| neighbors at 5 am when I load the car for work, because you
| want to chime nonstop when the door is open and I have zero
| control to turn it off or lower than the locked minimum. Oh,
| and don't forget the threats of voiding the warrantee if you
| dig deeper to disable anything. My favorite alarm and warning
| pops up randomly when you're driving, sometimes blocking the
| map, and it says something to the effect of, "Remember to stay
| focused on driving!"
|
| I see this slipping into not just alerts and notifications, but
| also ads. Waze does not care if they block my directions by
| blasting an ad on part of my screen, which has absolutely
| caused me to miss exits since they don't want to tell you the
| exit ahead of time on longer strips, only, "drive for 45
| miles"...
|
| I see this like popups, and if industry can't handle themselves
| they need to be forced to stop doing this altogether and find a
| rework, since it's clearly affecting private and public
| machines that could burnout or kill people.
| __turbobrew__ wrote:
| On my 2025 rav4 the dealer can turn off a bunch of the
| chimes. I basically told them to turn off everything.
| bluGill wrote:
| The Magnuson-Moss Warranty Act makes it clear that digging
| deeper cannot void any warrantee except unless they can show
| you caused the issue via digging deeper. You may needa lawyer
| to enforce it.
| t0mas88 wrote:
| This is a big topic in aviation. But the example you link to is
| about ACARS messages which are sent to the ground via data
| link, not visible to the crew.
|
| The ECAM system is what displays issues to the crew. There is a
| lot of logic there on what to show and what to inhibit. For
| example some errors aren't shown during takeoff since it's more
| important to focus on safely starting the climb, they would
| only be shown once past 400ft altitude.
|
| There is also a priority order based on criticality, they don't
| just show up in chronological order. Engine fire for example is
| a red warning and would always come above a yellow caution
| message. It's designed in a way where the results of the major
| problem (e.g. engine on fire, and then loosing electric power
| from that side) are lower priority than the problem itself.
| lll-o-lll wrote:
| > And for a technology example, a database server disappearing
| might raise a single alarm, but the applications that rely on
| that database might raise countless alarms as attempts to
| connect fail over and over again.
|
| Right. The lingo for this is "cascading alarms", and there are
| various mechanisms to suppress consequential alarms if you
| design well. If an "upstream" alarm results in further
| alarms/events downstream; these should be suppressed (still
| recorded, just not alarms), until the root alarm cause is
| resolved.
|
| I thought this was well understood in the industry, but perhaps
| not.
| 1e1a wrote:
| Direct link to the report PDF: https://maritime.lr.org/AM-
| report-2026
| paulnpace wrote:
| I didn't look at the PDF, but the sample size reported in the
| article is 11 ships(!), which makes me wonder how this might look
| across a larger population of ships.
|
| I wonder how much labor expense has to be saved to make up for a
| future catastrophic event?
| seb1204 wrote:
| Internalize gains, externalizing cost of events or accidents.
| ryukoposting wrote:
| 11 might be a larger share of ships worldwide than you'd
| expect. The complete list of US-flagged privately-owned
| merchant ships is less than 200:
|
| https://www.maritime.dot.gov/sites/marad.dot.gov/files/2025-...
| yorwba wrote:
| US-flagged privately-owned merchant ships might be a smaller
| share of ships worldwide than you'd expect: https://en.wikipe
| dia.org/wiki/List_of_merchant_navy_capacity... Liberia rules
| the waves.
| chriskanan wrote:
| This has been a major problem in hospitals where there are many
| false alarms and often concurrent alarms for hospital beds:
| https://pmc.ncbi.nlm.nih.gov/articles/PMC3928208/
| throwaway173738 wrote:
| We talked about alarms and alarm fatigue often when considering
| whether to show an alarm and at what priority when I was
| involved in a medical device.
| bob1029 wrote:
| I learned about the consequences of overloading the human
| operator when working on the primary UI for a manufacturing
| business. A natural inclination is to put things like
| confirmation dialogs around dangerous activities. I had managers
| telling me that one confirmation wasn't enough and that we had to
| add additional because people were still missing.
|
| Eventually, we tried removing the dialogs altogether and the
| incident rate approached zero. If you take away the guardrails
| completely, it radically alters the psychology and game theory
| around user interaction. Imagine climbing a tall building with
| multiple layers of protection vs having none at all.
|
| I strongly believe in ideas like "safety 3rd". It's not that I
| want the humans to be maimed by the machines. Quite the opposite.
| The difficulty is in understanding higher order consequences of
| "safety" and avoiding the immediate knee-jerk satisfaction that
| first order resolutions may provide.
| user_7832 wrote:
| > Eventually, we tried removing the dialogs altogether and the
| incident rate approached zero. If you take away the guardrails
| completely, it radically alters the psychology and game theory
| around user interaction. Imagine climbing a tall building with
| multiple layers of protection vs having none at all.
|
| I think there's evidence and studies on this. IIRC removing
| traffic lights forces people to be much more alert, _reducing_
| accidents.
|
| Fun fact: Bhutan is perhaps the only country in the world
| without traffic lights!
| funkyfiddler69 wrote:
| > IIRC removing traffic lights forces people to be much more
| alert, reducing accidents.
|
| No way this would work long-term in Germany. Maybe there
| wouldn't be _that_ many more accidents but traffic would
| stutter, all the time, everywhere. Some safety-first drivers
| still don 't get how roundabouts work ...
| ioanaci wrote:
| > Fun fact: Bhutan is perhaps the only country in the world
| without traffic lights!
|
| Afaict they have police officers regulating traffic instead.
| Not much difference in this particular discussion.
| rurp wrote:
| Years ago I did a lot of driving around rural Latin America
| and it could not have been more different from a US city.
| Official traffic rules were almost non-existent in many areas
| but the informal ones that had evolved worked shockingly
| well. Like a cramped two way street might only have room for
| one car in spots, but there would be a pattern for pulling
| over and letting opposing traffic pass.
|
| Things like that would probably break down at a certain level
| of crowded-ness, but it did somewhat change my view of
| regulation in general. I think there are a lot of cases where
| people will figure things out just fine if you leave them
| alone and count on them to be responsible, versus having a
| million detailed rules that are poorly enforced.
| kenrick95 wrote:
| My gripe is with multi-layer approvals for permission request
| tickets. If it's only 1 layer, the only layer will make sure
| the person it's correct. However once there are >1 layer, each
| layer will think the other n-1 layers will check and turn out
| no one will check and blindly approve things...
| Esophagus4 wrote:
| > If you take away the guardrails completely, it radically
| alters the psychology and game theory around user interaction
|
| Cool! Did workers expect consequences for incidents? Did they
| get rewarded for lack of incidents?
|
| Meaning, I imagine a world where there are no consequences for
| incidents and removing guardrails doesn't lower incident rates
| because people aren't incentivized to care?
|
| Or you're saying they naturally cared and removing guardrails
| allowed them to take ownership?
| bob1029 wrote:
| It was definitely more of a stick than carrot situation.
|
| The issue with multiple dialogs is that the operator could
| claim that they were confused with conflicting wording and
| the implications of things like "Confirm" vs "Cancel" in
| certain contexts of use. This provides some degree of cover
| for moving with less care. With no dialog at all, the
| operator has nothing to point to but their own actions. There
| is nothing to hide behind.
|
| The fact that this was also a heavily multi-lingual/cultural
| environment amplified the effect of poorly designed safety
| mechanisms dramatically.
| kyralis wrote:
| Confirm/Cancel (like Yes/No) for dialog buttons has been
| known to be confusing and detrimental for _decades_ now.
| The button names should always describe action to be taken,
| _not_ a response to the text above.
|
| My point is that the operator may be _genuinely confused_
| by a poor interaction model. Removing that interaction
| model entirely is certainly an option, but it 's not clear
| that comparing "no dialog" vs "bad dialog" is a strong
| argument for "dialogs bad, better to have none" - you don't
| have data for the "good dialog" case, which may be better
| still.
| bananaflag wrote:
| I remember being confused as a kid by "Yes/No/Cancel"
| when the computer asked "Do you want to save the
| changes...?" because I couldn't figure out whether
| "Cancel" meant "Yes" or "No" and why on earth would one
| have a third option. I then realized it meant to "cancel"
| my intention to close the file. I had been confused
| because I thought it meant to "cancel" the computer's
| intention to ask me.
|
| Also, I was that obnoxious kid who, after asking someone
| a yes/no question, used to add "Yes/No/Cancel" (probably
| to highlight my perceived absurdity of that button).
| fhars wrote:
| Like the "Cancel subscription" dialog with options
| "Cancel" and "Cancel"...
|
| UX Design is hard...
| Esophagus4 wrote:
| Really interesting, thanks.
|
| Kind of reminds me of parallels to not wearing a seatbelt
| making someone drive with more care.
| lll-o-lll wrote:
| Having worked in this space for over a decade (but not in
| the last 7 years), your comments surprise me. Is this a
| smaller operation perhaps?
|
| In general, operator UX (HMI, human machine interface), is
| a an area that's well researched and more or less
| standardised in recent times (ISA101). Abnormal Situation
| Management (ASM), Situational Awareness, automatic alarm
| suppression for "consequential alarms", High Performance
| Graphics (basically everything grey except the stuff that
| matters). If your Engineers do a good job; the operators
| can do a good job.
|
| Removing all interlocks sounds like a bit of a cop-out to
| me. Interlocks are there to prevent the mis-click pouring
| molten steel on peoples heads. If you have a nice boring
| standardized ASM HMI, operators can't "hide behind them".
| Every operation is the same.
| TrainedMonkey wrote:
| In litigious countries confirmations and alerts also serve as a
| mitigation against lawsuits.
| applied_heat wrote:
| I've had to add more confirmations as people connect remotely
| using VNC and accidentally click buttons which then have real
| world actions.
|
| To catch them all it logs the user out so buttons which affect
| the process are disabled, but now it can be annoying to be
| logged out after x minutes and always logging in.
| dinkblam wrote:
| on around 300 days per year i see a "severe weather alarm" on the
| iPhone Weather app, although _nothing_ at all is happening,
| completely ridiculous.
| SoftTalker wrote:
| We've gone so over the top on weather fearcasting. Just look
| out the window if you want to know what the weather is. Save
| the "the world is ending" messages for truly life-threatening,
| property-damaging weather (and no, temperature alone doesn't
| qualify---it's easy to know it's cold or hot by just stepping
| outside).
| phil21 wrote:
| Timely. I'm about to turn off severe weather alerts from my
| local city because they insist on spamming - multiple times
| per day - cold weather alerts.
|
| And they start at pretty ridiculous temperatures in the
| double digits. The only way those would be dangerous to you
| is if you were homeless and lacked any form of winter
| clothing, at which point you either already know or are too
| far mentally gone for a text alert to help you.
| The_President wrote:
| Excessive alarms from machinery could result in the alarm being
| bypassed. This is an insidious issue as both the alarm condition
| and the jumper clips could go unnnoticed and result in an event.
| general1465 wrote:
| That was one of the reasons why Three Mile Island melted down -
| Alarm overloading operators who were then distracted by mundane
| issues and did not noticed that nuclear core is melting down.
|
| https://en.wikipedia.org/wiki/Three_Mile_Island_accident
| havaloc wrote:
| If you want a real life simulation of this, just go into the
| lobby of a local McDonald's near the kitchen. It's a cacophony
| and the employees have tuned it out.
| SoftTalker wrote:
| And those are all just timers.
| bluGill wrote:
| As an employee years ago - we learned to tune the alarms from a
| different station out - but someone handled each one in a few
| seconds. Because non were ingored they were all good - despite
| the vast quantity.
| funkyfiddler69 wrote:
| > Improvements were delivered through traditional marine
| engineering interventions, including correcting valve
| installations, replacing faulty sensors and tuning existing
| systems.
|
| This sounds like engineers wanted to go traditional routes in the
| first place but the chain of command wanted to reduce
| production/constuction time and save money.
| nik282000 wrote:
| I work at a plant with a site wide SCADA/HMI (Siemens WinCC)
| system, every alarm is displayed on every HMI regardless of its
| proximity to the machine or even its ability to address the
| issue. And any given minute a hundred or more alarms can be
| generated, the majority being nuisance messages like "air
| pressure almost low" or my favorite " " (no message set) but
| scattered among those is the occasional "no cooling water -
| explosion risk".
|
| This plant is operated and deigned to the spec of an
| international corp with more than 20 factories, it's not a mom-
| and-pop operation. No one seems to think the excessive, useless,
| alarms are an issue and that any damage caused by missed warnings
| is the fault of the operator. When approaching management and
| engineering about this the responses range from "it's not in the
| budget" to " you're maintenance, fix all the problems and the
| alarms will go away".
|
| The only way for this kind of issue to be resolved is with
| regulation and safety standards. An operator can't safely operate
| equipment when alarms are not filtered or sorted in some way.
| It's like forcing your IT guy to watch web server access logs
| live to spot vulnerabilities being exploited.
| varjag wrote:
| I think it's regulated in places, as it was certainly an HMI
| concern ever since Three Mile Island. Our customer is really
| grilling vendors for generating excessive alarms. Generally for
| a system to pass commissioning it has to be all green, and if
| it starts event bombing after you're going to be chewed.
| nik282000 wrote:
| I have never seen a piece of new equipment that ever gets to
| an All Green state, before, during or after commissioning. I
| frequently recommend that we do not allow the commissioning
| team to leave until they can get it to that state but it has
| yet to happen.
| varjag wrote:
| I guess it's the matter of setting the expectations, both
| on SCADA and equipment side. Spent this weekend getting rid
| of that last sporadic alert...
| CamperBob2 wrote:
| _The only way for this kind of issue to be resolved is with
| regulation and safety standards._
|
| Are you sure that's not what caused the problem in the first
| place? Unqualified and/or captured regulators who come up with
| safety standards that are out of touch with how the system
| needs to work in the real world?
| AlotOfReading wrote:
| Do regulators come up with SCADA safety standards? I would
| have assumed it was IEC.
|
| Regulators coming up with engineering standards is pretty
| rare in general. Usually they incorporate existing
| professional standards from organizations like SAE, IEEE,
| IEC, or ISO.
| anonymousiam wrote:
| The criticality of the alerts should be classified, and
| presented with the alert. Users should have the ability to
| filter non-critical messages on certain platforms.
|
| Unfortunately, some systems either don't track criticality, or
| some of the alerts are tagged with the wrong level.
|
| (One example of the latter is the Ruckus WAP, which has a
| warning message tagged at the highest level of criticality, so
| about two or three times a month, I see the critical alert:
| "wmi_unified_mgmt_rx_event_handler-1864 : MGMT frame, ia_action
| 0x0 ia_catageory 0x3 status 0x0", which should be just an
| informational level alert, with nothing to be done about it.
| I've reported this bug to Ruckus a few times over the past five
| years, but they don't seem to care.)
| varjag wrote:
| In reality users will keep everything on default.
| lostdog wrote:
| I wonder if you could calculate a "probability of response to
| major alert" and make it the inverse of the total or irrelevant
| alerts. Then you get to ask "our probability of major alert
| saliency is onlt 6%. Why have the providers set it at this
| level, and what can we do to raise it?"
| terminalshort wrote:
| This is a fundamental organizational and societal problem. An
| engineer would look at the situation and think "what is the
| best way to get the failure rate below a tolerable limit?" But
| a lawyer looks at the situation and thinks "how do I minimize
| liability and bad PR?" and a bureaucrat thinks "how can I be
| sure the blame doesn't land on me when something goes wrong?"
| And the answer to both of those questions is to throw an alarm
| on absolutely everything. So if there is a problem they can
| always say "our system detected the anomaly in advance and
| threw an alarm." Overall the system will be less safe and more
| expensive, but the lawyer's and bureaucrat's problems are
| solved. Our society is run by lawyers and bureaucrats, so their
| approach will win out over the engineer's. (And China's society
| is run by engineers, so it will win out over ours.)
| mmooss wrote:
| The first step in problem solving is to look in the mirror.
| It's not surprising that in an engineering community, the
| instinct is to blame outsiders - lawyers, bureaucrats,
| managers, finance, etc. - because those priorities are more
| likely to conflict with engineering, because it is harder to
| understand such different perspectives, and because it is
| easier to believe caricatures of people we don't know
| personally.
|
| Those people have valuable input on issues the engineer may
| not understand and have little experience with. And engineers
| are just as likely to take the easy way out, like the
| caricature in the parent comment:
|
| For example, for the manufacturer's engineering team it's
| much easier, faster and cheaper to slap an alarm on
| everything than to learn attention management and to think
| through and create an attention management system that is
| effective and reliable (and it had better be reliable -
| imagine if it omits the wrong alarms!). I think anyone with
| experience can imagine the decision to not delay the project
| and increase costs for that involved subproject - one that
| involves every component team, which is a priority for almost
| none of them, and which many engineers, such as the
| mechanical engineer working on the robotic arm, won't even
| understand the need for.
|
| > And China's society is run by engineers, so it will win out
| over ours.
|
| History has not been kind to engineers who do non-
| engineering, such as US President Herbert Hoover who built
| dams and but also had significant responsibility for the
| Great Depression. It's not that engineers can't acquire other
| skills and do well in those fields, but that other skills are
| needed - they aren't engineering. Those who accept as truth
| their natural egocentric bias and their professional
| community's bias toward engineering are unlikely to learn
| those skills.
| terminalshort wrote:
| Your own answer circles right back to the problem I'm
| talking about:
|
| > and it had better be reliable - imagine if it omits the
| wrong alarms!
|
| This is entirely based on the premise that an error due to
| omitting the wrong alarm is worse than an error based on
| including too many alarms. That right there is lawyerthink.
| Also, these priorities don't conflict as you say, they just
| take different sides of a tradeoff. Managers and finance
| people are balancing a tradeoff of delivery speed, cost,
| and quality to maximize business value. And the bureaucrats
| and lawyers are choosing more expensive and less reliable
| systems because they better manage the emotions of panicky
| anxious people looking for a scapegoat in a crisis. This
| has a cost.
|
| Besides having bad luck in timing to be president when the
| stock market crashed, and therefore scapegoated for it,
| Herbert Hoover was well regarded in everything he did
| before and after his term, including many non engineering
| related things. So I think he is a particularly poor
| example of this. Public blame for things like that tends to
| be exactly as rational as thinking a hangover has nothing
| to do with last night.
| mmooss wrote:
| I don't see how it's 'lawyerthink' at all; engineers also
| want to prevent bad outcomes, especially from their own
| work, as does everyone else.
|
| Also, I think this ignores the rest of my point to
| nitpick one part of a complex system, which was part of a
| larger point.
| terminalshort wrote:
| To your larger point I don't think engineering logic is
| necessarily superior to financial logic, or manager
| logic. The problem is that because of the way we have
| built our society, engineering (and all other fields)
| must comply with and be subservient to bureaucrat and
| lawyer logic. The legal defense against an engineering
| failure is not to prove that your overall failure rate is
| low and within acceptable limits, but rather to come up
| with as long a list as possible of safety measures and
| policies that you followed without any regard to whether
| they actually have any effect at all.
| pstuart wrote:
| > This is a fundamental organizational and societal problem
|
| Absolutely, and we'd collectively be better served if we had
| tools to deal with it.
|
| I think of it as "incentive ecology" -- as noted, everybody
| has their own incentives which shapes their behavior, which
| causes downstream issues that begin the process anew.
|
| Obviously there's no simple one-shot solution to this, but
| what if we had ways to simplify and model this "web of
| responsibility" (some sort of game theory exposed as an
| easily consumed presentation, with computed outcomes that
| show the cost/ROI/risk/reward) that could be shared by all
| stakeholders?
|
| Obscurity and deniability are the weapons wielded in most of
| these scenarios, so what if we could render them obsolete?
|
| Sure, those in power would not want to yield their
| advantages, but the overall outcomes should reward everybody
| by minimizing risks and maximizing rewards for the enterprise
| and everybody wins.
|
| Yes, I'm looking at it as a an engineer and a dreamer, but I
| think if such a tool existed that was open source and easily
| accessible that this work could be done by rogue participants
| that could put it out there so it's undeniable.
| renewiltord wrote:
| Is it though? Engineer can optimize on different manifold.
| Company can succeed/fail for different reasons. Getting
| destroyed for legal suit because didn't place alarm is small
| peace when you did better engineering.
|
| After all, read any post-mortem comments on HN. Many of those
| people can be hired as expert if you like. They will say "I
| would have put an alert on it and had testing". You will lose
| the case.
|
| "Oh but we are trying to keep error rate low". Yes, but now
| your company is dead when high error rate company is alive.
|
| In revealed preferences, most engineers prefer vendors who
| have CYA. This is obvious from online comments. This is not
| because they are engineer. It's because most people want to
| believe that event is freak accident.
|
| Building system for error budget is not actually easy. Even
| for engineer who say they want it. Because when error
| happens, they immediately say it should not have happened.
| Counterfactual other errors blocked, and business existing
| are not considered. Every engineer is genius in hindsight.
| Every person is genius in hindsight.
|
| Why these genius never make failure proof company? They do
| not. Who would not pay same price for 100% reliable tech?
| terminalshort wrote:
| > Getting destroyed for legal suit because didn't place
| alarm is small peace when you did better engineering.
|
| Indeed it is. That's why I said it's a larger societal
| problem in how we manage risk and react to failures.
|
| > Why these genius never make failure proof company?
|
| Because this is mostly a matter of unknown unknowns and
| predicting the future, so even a founder who makes zero
| mistakes is more likely than not to fail.
| gopher_space wrote:
| Up to a certain point society is run by actuaries. Finding
| someone at your insurance company who both understands the
| problem with excess errors _and_ appreciates how easily
| enumerable they are would be an interesting "whistleblowing"
| target.
| terminalshort wrote:
| But the actuaries too are constrained by the same societal
| constructs. Let's say you work for a large car company and
| invent a self driving system that is 10x safer than the
| average human driver, and the cost is minimal. This system
| would likely save 36,000 lives annually in the US. The
| actuary will calculate that with human drivers, the
| accident liability for your car company is $0, but then
| with the self driving system, the liability is potentially
| in the billions, and it doesn't make sense to include this
| system in your vehicles. You can argue that since the error
| rate is 10x safer than the average human driver, that
| clearly this is an acceptable safety level. But your
| argument will fall on deaf ears because there is no such
| legal concept of an acceptable level of death. You could
| also say, well then the company can buy insurance. The cost
| will be minimal since there will only be 10% the rate of
| accidents as a normal car. The societal calculation for
| liability in a death is based mainly on:
|
| 1. How much money does the responsible party have
|
| 2. How much media attention and outrage does it generate
|
| And since now instead of a single human driver, the
| liability will rest on a massive corporation, and the media
| attention will be massive vs nonexistent for a normal human
| crash, the liability will be massive. The accident rate may
| be 10x less, but the cost per death may be 1000x of a
| normal human driver crash.
| lazyasciiart wrote:
| Ah, so that's why nobody has built self driving cars.
| terminalshort wrote:
| Go to dictionary.com and look up "hypothetical"
| bluGill wrote:
| Courts do accept alarm fatigue and if there is an
| injury/death and there were many alarms you can bet that
| whatever lawyers' side benefits will bring in experts to
| explain the issue.
|
| if there are a lot of issues the lawyers will also ask why
| they were not corrected first: using that to establish a
| pattern of bad maintenance.
| miki123211 wrote:
| Useless warnings are a great CYA tactic.
|
| THe more of them you have, the more likely it is that there's a
| warning if something happens. Whether the warning is ever
| noticed is secondary, what matters is the fact that there was a
| warning and the operator didn't react to it appropriately,
| which makes the situation the fault of the operator.
| cucumber3732842 wrote:
| This is partly a problem with our workplace laws.
|
| In the eyes of the regulators and courts individual low level
| employees can not take responsibility. This is the logic by
| which they fine the company when someone does something you
| shouldn't need to be told not to do on a step ladder or
| whatever.
|
| What this means is that low level employees become liability
| sinks. Show them all the warnings and make them figure it
| out. Give them all sorts of conflicting rules and let them
| sort out which ones to follow. Etc, etc.
| acheron wrote:
| Follow the incentives. If I'm in charge of setting alarms, and
| something bad happens without an alarm going off, it's my fault.
| If the alarm goes off and the operator ignores it, it's their
| fault.
| barrkel wrote:
| The main purpose of alarms is to relieve automation of liability.
| fallingfrog wrote:
| Any equipment eventually fails. Designing it to fail safe- in a
| way that does not threaten the human operator- costs money. But a
| little warning message costs nothing and ensures that the human
| can be blamed when something goes wrong. The company can point to
| the message and say, "see, you should have known."
|
| The message was never intended to help the human operator. It was
| intended to allow the company to avoid responsibility for cutting
| corners.
|
| If the goal of the message was to communicate something important
| to the human operator, extraneous messages would be a serious
| problem. But if the goal is simply to cover the ass of the
| company, then extra error messages are not a problem at all.
| Thats why they never get fixed or pruned.
| dlcarrier wrote:
| It's a problem in every field. In aviation, notices about life-
| threatening conditions can be buried in pages about taxiways
| closing during certain hours and other nonsense.
| __turbobrew__ wrote:
| When I have over 10 alarms firing on systems I run I find that
| overwhelming, I cant imagine thousands. There is no way you can
| operate a system with that many alarms.
| pwjtnlskshc wrote:
| No need to solve the underlying problem, let's just send all the
| alerts to AI to deduplicate and prioritize them! /s
| uejfiweun wrote:
| This was my issue with platforms like Sentry. Turns out, in a
| decently used prod web app with a lot of dependencies, there's a
| lot of noise. After spending hours trying to debug some of these
| only to conclude that they were not debuggable, I ended up kind
| of letting the Sentry reports fade into the background, and
| eventually I turned it off entirely to reduce payload sizes.
| AngryData wrote:
| This is how oil pipeline owners get away with leaking millions of
| petroleum products all over our natural landscape. They
| legislated to be able to keep using their 60 year old pipeline
| leak detection methods, which are a false alarm hell, despite
| better methods existing and being cheaper to maintain. Hire a guy
| whos job is to stand around and press a button if things go
| wrong, give them 99 false signals out of 100, then fire anybody
| who presses the button too often. And pretty soon, nobody ever
| sounds the alarm when bad shit happens, because ignoring the
| problem and letting people die is more profitable.
___________________________________________________________________
(page generated 2026-01-25 23:01 UTC)