[HN Gopher] How Complex Systems Fail (1998)
___________________________________________________________________
How Complex Systems Fail (1998)
Author : kiyanwang
Score : 126 points
Date : 2022-09-19 08:40 UTC (14 hours ago)
(HTM) web link (how.complexsystems.fail)
(TXT) w3m dump (how.complexsystems.fail)
| bob1029 wrote:
| The root cause thing I would push back on.
|
| Management of complex systems is never a done deal, so there is
| always the possibility you missed some tiny gap in your process
| that can still take you out entirely.
|
| A good example of this being the Texas grid in 2021.
| aliasxneo wrote:
| I believe the point is to not look for a _single_ root cause.
| redtexture wrote:
| Not a tiny gap at all.
|
| For which there had been numerous warnings for years by
| industry observers.
|
| That system does not draw upon outside of network utilities, to
| avoid Federal regulation, hence has limited reserve resources.
| And the Texas system did not pay providers to have standby
| reserves. Thus a fragile system, easy to be subject to failure.
|
| Edit:
|
| Texas Was Warned a Decade Ago Its Grid Was Unready for Cold
| (Bloomberg)
|
| https://www.bloomberg.com/news/articles/2021-02-17/texas-was...
| kwhitefoot wrote:
| Yes, but that tiny gap is no more the root cause than any other
| one of many other decisions that preceded the catastrophe.
| dqpb wrote:
| This does not seem very rigorous. Can someone point to a better
| coverage of this topic?
| thraxil wrote:
| It's not intended to be rigorous. The context here is that
| Richard I. Cook, one of the main figures in safety and
| resilience engineering, who's published many, many papers on
| these topics died recently. The "How Complex Systems Fail"
| paper is intended to be a bit pithy and light; more an attempt
| at summarizing years of wisdom. See:
| https://www.adaptivecapacitylabs.com/blog/2022/09/12/richard...
| dqpb wrote:
| Well, this sounds wrong to me:
|
| > Catastrophe requires multiple failures - single point
| failures are not enough
|
| My experience is that a single failure causes a cascade of
| subsequent failures. This topic is very interesting, but this
| post is more of a teaser of topics than a real explanation.
| advisedwang wrote:
| Places where a single failure in an otherwise perfectly
| functioning system can cause catastrophic outcomes are
| relatively easy to identify, relatively easy to argue need
| to be fixed and relatively easy to fix. As a result mature,
| complex systems have generally developed safety mechanisms
| for such issues. Once you have done that you need at least
| two failures (underlying issue + safety, hot+cold, or two
| interacting systems).
|
| I would suspect that your experience of single modes of
| failure being present are one of the following
|
| * Immature system (e.g. a startup) * One where failure is
| acceptable and so engineering isn't invested in solving
| these issues (i.e. the author is talking about disasters
| that kill people, not causing a few mins of ads not
| getthing shown) * Extreme organizational dysfunction
| (talking criminal negligence type stuff)
| dqpb wrote:
| > Once you have done that you need at least two failures
| (underlying issue + safety, hot+cold, or two interacting
| systems).
|
| Ah, I missed the part where he said - except for
| distributed systems. The thing is, effectively all
| systems are distributed systems with two or more
| interacting subsystems.
|
| And no, I'm not talking about immature systems or ones
| where failure is acceptable. Queuing issues, for example,
| are well known to cause to cascading effects, and are not
| trivial to identify or solve.
|
| Even basic correctness issues can be very difficult to
| identify if you have a large permutation space and no
| model checking, and will also cascade.
| novixyz wrote:
| I agree.
|
| I always thought that late Paul Ciliers' did a great summary on
| complexity (sorry no online link):
|
| "Complexity in a Nutshell:
|
| I will not provide a detailed description of complexity here,
| but only summarise the general characteristics of complex
| systems as I see them.
|
| -Complex systems consist of a large number of elements that in
| themselves can be simple.
|
| - The elements interact dynamically by exchanging energy or
| information. These interactions are rich. Even if specific
| elements only interact with a few others, the effects of these
| interactions are propagated throughout the system. The
| interactions are nonlinear.
|
| - There are many direct and indirect feedback loops.
|
| - Complex systems are open systems - they exchange energy or
| information with their environment - and operate at conditions
| far from equilibrium. Complex systems have memory, not located
| at a specific place, but distributed throughout the system. Any
| complex system thus has a history, and the history is of
| cardinal importance to the behaviour of the system.
|
| - The behaviour of the system is determined by the nature of
| the interactions, not by what is contained within the
| components. Since the interactions are rich, dynamic, fed back,
| and, above all, nonlinear, the behaviour of the system as a
| whole cannot be predicted from an inspection of its components.
| The notion of emergence is used to describe this aspect. The
| presence of emergent properties does not provide an argument
| against causality, only against deterministic forms of
| prediction.
|
| - Complex systems are adaptive. They can (re)organise their
| internal structure without the intervention of an external
| agent.
|
| Certain systems may display some of these characteristics more
| prominently than others. These characteristics are not offered
| as a definition of complexity, but rather as a general, low-
| level, qualitative description. If we accept this description
| (which from the literature on complexity theory appears to be
| reasonable), we can investigate the implications it would have
| for social or organisational systems."
|
| Ciliers, P. (2016). Critical Complexity Collected Essays,
| Walter de Gruyter GmbH. 67
|
| Also if you look up any Dave Snowden's video on YT you'll find
| plenty of useful info.
| greyface- wrote:
| 2020, 84 comments https://news.ycombinator.com/item?id=25550685
|
| 2019, 33 comments https://news.ycombinator.com/item?id=20380055
|
| 2017, 21 comments https://news.ycombinator.com/item?id=15002683
|
| 2014, 16 comments https://news.ycombinator.com/item?id=8282923
| dang wrote:
| Thanks! Macroexpanded:
|
| _How Complex Systems Fail (1998)_ -
| https://news.ycombinator.com/item?id=25550685 - Dec 2020 (83
| comments)
|
| _How Complex Systems Fail (1998) [pdf]_ -
| https://news.ycombinator.com/item?id=20380055 - July 2019 (33
| comments)
|
| _How complex systems fail (2002) [pdf]_ -
| https://news.ycombinator.com/item?id=15002683 - Aug 2017 (19
| comments)
|
| _How Complex Systems Fail (1998) [pdf]_ -
| https://news.ycombinator.com/item?id=14127543 - April 2017 (7
| comments)
|
| _How Complex Systems Fail (1998)_ -
| https://news.ycombinator.com/item?id=8282923 - Sept 2014 (16
| comments)
|
| _How Complex Systems Fail [pdf]_ -
| https://news.ycombinator.com/item?id=926735 - Nov 2009 (1
| comment)
| mellavora wrote:
| So it looks like 2020 is still in the lead...
| xbar wrote:
| There were some spectacular systems failures in 2020 that
| earned reminders.
|
| I am of the mind that this document is undertaught.
| tempie_deleteme wrote:
| why read about it when we can experience it by being alive today?
|
| to better understand the times, of course...
| eternityforest wrote:
| The real URL is how.complexsystems.fail
|
| Point 2 is "Complex systems are heavily and successfully defended
| against failure"
|
| Complex systems do fail. But airplanes are still extremely safe.
| Because people stacked on even more complex systems, often
| involving worldwide change in response to an accident that
| happened once.
|
| You constantly hear about how safe it is to fly. And yet hardly
| anyone seems to learn fron their successes. When you stop
| accepting failure and are willing to disrupt everything if it
| saves even one life, you can do a lot.
|
| Complex systems may be unreliable, but with enough work, it seems
| we can sometimes make the overall picture safer than not having
| them.
|
| I can't firmware update all of mankind to never leave a baby in a
| hot car. But they can put sensors on seats and continually do
| studies to be sure it's working. Complex systems are sometimes
| more controllable than people or simple systems.
|
| The choice sometimes seems to be "Add complexity, do nothing, or
| do something that nobody will accept"
| hutzlibu wrote:
| "I can't firmware update all of mankind to never leave a baby
| in a hot car. But they can put sensors on seats and continually
| do studies to be sure it's working."
|
| We can also put mandatory sensors in peoples bodies, to make
| sure they act and live allright.
|
| But I think this would be overcomplicating things.
| dijit wrote:
| I really see your point here; but I have to caution: Airplanes
| are "exactly as simple" as they need to be.
|
| There is a lot that goes into their design to simplify things
| greatly; you're probably thinking of complicated computer
| systems that are used in planes.
|
| But those computer systems are incredibly simple compared to
| what we use or build atop of: as simple as they have to be in
| order to be fully understood.
| teddyh wrote:
| s/^/How /
| Zealotux wrote:
| Simple systems also fails!
| 0xbadcafebee wrote:
| Numbers 1, 4-8, and 11-18, are all Truisms. The rest are not:
|
| _" 2. Complex systems are heavily and successfully defended
| against failure"_
|
| Many complex systems are weakly defended, sometimes not at all.
| Sometimes the defense is accidental or incidental. Sometimes they
| are heavily yet unsuccessfully defended. Never attribute to
| defense that which can be attributed to purely random chance,
| ignorance, convenience, and avoidance of responsibility.
|
| _" 3. Catastrophe requires multiple failures - single point
| failures are not enough."_
|
| Catastrophe definitely can and does happen from single points of
| failure. It's just that in highly defended systems, multiple
| failures are common.
|
| _" 9. Human operators have dual roles: as producers & as
| defenders against failure."_
|
| These can be distinct roles, but in practice that requires extra
| money, staffing, etc which makes it rare. However, there are
| systems in which defense becomes its own role, often because the
| producers suck at it or don't want to do it, or are just really
| busy.
|
| _" 10. All practitioner actions are gambles."_
|
| On the fence about this one. I would say all practitioner
| _changes_ are gambles. A practitioner looking at a pressure gauge
| dial is an action, but it isn 't a gamble. Unless the gauge
| needle sticks, and reading it was a critical action... I suppose
| you could say all actions are gambles, and changes are much more
| risky gambles, and non-change actions are likely to be seen as
| non-risky.
| eternalban wrote:
| _Engineering a safer world - Systems thinking applied to
| thinking_ , by Nancy Leveson (MIT - 2009) was recommended in a
| previous discussion as a more comprehensive and systemic
| treatment: https://news.ycombinator.com/item?id=14131981
|
| http://sunnyday.mit.edu/safer-world.pdf
|
| STAMP ("System-Theoretic Accident Model and Processes") is
| reviewed here:
| https://www.sciencedirect.com/science/article/abs/pii/S09504...
|
| And there is a course (lecture notes look great):
| https://ocw.mit.edu/courses/16-63j-system-safety-spring-2016...
| _plg_ wrote:
| The Career, Accomplishments, and Impact of Richard I. Cook: A
| Life in Many Acts (September 12, 2022)
| https://www.adaptivecapacitylabs.com/blog/2022/09/12/richard...
| x32n23nr wrote:
| I highly recommend the book Normal Accidents by Charles Perrow.
| Perrow argues that multiple and unexpected failures are built
| into society's complex and tightly coupled systems
___________________________________________________________________
(page generated 2022-09-19 23:01 UTC)