[HN Gopher] A Brief History of High Availability
___________________________________________________________________
A Brief History of High Availability
Author : EntICOnc
Score : 49 points
Date : 2022-04-26 11:21 UTC (1 days ago)
(HTM) web link (www.cockroachlabs.com)
(TXT) w3m dump (www.cockroachlabs.com)
| jve wrote:
| > second, by placing computers under constant demand, they were
| more likely to fail
|
| Really? In what sense? Hardware fails or what? The complexity
| usually brings everything down.
|
| Anyway, looking forward to cockroachdb, it seems it solves some
| high-availability / multi-site problems.
| notacoward wrote:
| I think what the authors are getting at is that many failure
| conditions are more likely to manifest under load. This
| includes various kinds of resource exhaustion, including those
| induced by leaks. It also includes things like timeouts due to
| long queues, and even race conditions. The higher the load, or
| the longer it's sustained, the more likely it becomes that such
| bugs will manifest. In every such system I've seen, from very
| early days of the internet or old-school HA to modern multi-
| thousand-machine clusters, there have been bugs that only
| showed up under heavy and/or sustained load. Both Erlang's
| supervisor trees and Recovery Oriented Computing
| (Berkeley/Stanford) are based on the observation that such
| failures will always be with us, and have influenced many
| subsequent systems - arguably including CockroachDB.
| marcosdumay wrote:
| Heat caused failures also increase with load, as does
| electromigration, disk stress, power failures...
|
| The GP has a very digital model of computers. It's not really
| useful for predicting failure.
| zamadatix wrote:
| One concrete example involving DBs is the PostgreSQL TXID
| wraparound. It's basically 32 bit counter where if things go
| wrong or were built wrong it just keeps going up until it
| becomes a problem. If you're only doing <=1 request per second
| this won't ever be a problem since even if you ran it your
| whole life without freezing any old records it wouldn't
| approach max value. On the other hand if it's under a constant
| 1000 request per second load it can be the source of problems
| in about a month.
|
| You could argue to just call the TXID "complexity" but that'd
| come with a wide enough definition of what complexity is that
| it just becomes a replacement for saying "buggy code" which
| isn't very useful. Most leave complexity as being more like
| "grossly overengineered" to leave room to avoid that.
| throwaway787544 wrote:
| Fun facts about availability:
|
| 1) the more nodes, the higher probability of failure, 2) the more
| load, the higher probability of failure, 3) complex logic and/or
| nodes that aren't tested frequently have a higher probability of
| failure
|
| Basically, every person who has claimed "X is easy, I do it for
| my personal use and it's fine with one giant node, don't need HA"
| is true (for them) because it's literally less likely to fail.
| But if they had more load they'd see more failures, and
| probability doesn't mean they won't encounter the random failures
| that are much harder to recover when there's no backup system.
|
| HA means you're going to have more failure, which means more work
| to keep it running. But it's necessary work if total system
| failure is unacceptable. Two engines in an airplane is twice the
| maintenance, but it's better than the only engine conking out
| mid-flight.
| marcosdumay wrote:
| HA has a very sweet spot at the place where "the system is down
| at 3AM and must be fixed RIGHT NOW!" do not happen, but the
| complexity isn't increased enough for it to generate much
| ongoing work.
|
| Anybody bellow that spot is just begging for being taken out by
| random at the worst possible times.
| rospaya wrote:
| > 1) the more nodes, the higher probability of failure
|
| I may be misreading your comment, but how is this true?
| notacoward wrote:
| This is one of the classic problems of talking about HA:
| making it clear which "failure" or "availability" you mean at
| any given moment. In the particular quoted sentence,
| "failure" refers to _component_ failure even though the
| "availability" in HA means _system_ availability. The goal of
| HA design is to make a system that becomes _more_ available
| as components are added instead of less (as you seem to
| expect). Redundancy, internal monitoring, and failover are
| the tools of the trade. What you should end up with is a
| system more like two chains holding up an object instead of
| one chain twice as long.
| lhoff wrote:
| If you have one node with the probability of x% that it fails
| within a year and you add a second one with the same
| probability to fail the total pobability that one of the two
| nodes fail within a year goes up.
| macintux wrote:
| Not OP, but two things I've found to be true:
|
| * the more complex the configuration required to provide high
| availability, the more likely something will go wrong and
| reduce availability
|
| * the more nodes you have, the more likely one of them will
| fail. Whether that reduces availability typically depends on
| the failure mode and control layer.
|
| For an example of the latter, with the distributed database
| Riak (and I imagine others as well) if a server is starting
| to fail (corrupt disk, bad network card, etc) you're in a
| much better place if it stops working entirely. A gradual
| node failure is much much worse.
| notacoward wrote:
| I was very unsurprised to see that this history starts a bit
| late. In 1992 I was working on HACMP/6000, which was a high-
| availability clustering product for IBM's AIX (though I worked at
| a company that was on contract to them). Databases were a primary
| use case, relying on shared storage instead of replication. We
| were hardly the first into that space BTW. DEC in particular had
| developed a lot of the concepts (e.g. locking models) that we and
| others at the time emulated, and the history in this space goes
| back even further than them. Were those systems as scalable as
| what we have now, or even as robust? Nowhere even close. But the
| erasure of that history still irks me. Modern distributed
| databases (and distributed systems in general) in their turn
| relied heavily on experience and concepts from those earlier
| systems, but rarely acknowledge the fact.
| cbm-vic-20 wrote:
| Here's a link to a PDF of the September 1987 issue of the
| Digital Technical Journal, which was all about clustering on
| the VAX/VMS machines.
|
| https://vmssoftware.com/docs/dtj-v01-05-sep1987.pdf
| jlokier wrote:
| High-availability database systems go back to 1974 or so, with
| Tandem Computers.
|
| https://en.wikipedia.org/wiki/Tandem_Computers
|
| "Tandem's NonStop systems use a number of independent identical
| processors and redundant storage devices and controllers to
| provide automatic high-speed "failover" in the case of a
| hardware or software failure. To contain the scope of failures
| and of corrupted data, these multi-computer systems have no
| shared central components, not even main memory. Conventional
| multi-computer systems all use shared memories and work
| directly on shared data objects. Instead, NonStop processors
| cooperate by exchanging messages across a reliable fabric, and
| software takes periodic snapshots for possible rollback of
| program memory state."
|
| The general ideas of failover for high-availbility go back much
| further.
|
| https://en.wikipedia.org/wiki/Failover
|
| "The term "failover", although probably in use by engineers
| much earlier, can be found in a 1962 declassified NASA report.
| The term "switchover" can be found in the 1950s when describing
| '"Hot" and "Cold" Standby Systems', with the current meaning of
| immediate switchover to a running system (hot) and delayed
| switchover to a system that needs starting (cold). A conference
| proceedings from 1957 describes computer systems with both
| Emergency Switchover (i.e. failover) and Scheduled Failover
| (for maintenance)."
|
| Also worth a mention that the well known Paxos distributed
| consensus algorithm was described in a 1989 paper by Leslie
| Lamport, and the almost identical (but less well known)
| predecessor Viewstamped Replication in a 1988 thesis by Brian
| Oki.
| jorangreef wrote:
| I had the recent privilege of interviewing Brian Oki [1] of
| Viewstamped Replication in September last year, since VSR is
| the consensus protocol that we use in TigerBeetle [2] to
| process a million financial transactions per second.
|
| One of the first things that Brian Oki mentioned was Tandem
| NonStop, and I understand that Brian's VSR was then the first
| consensus protocol to deal with network partitions, where the
| network is not reliable, for high availability.
|
| What was also special was that James Cowling joined me
| afterwards--we covered Viewstamped Replication Revisited, the
| 2012 revision, with a focus in both interviews on Barbara
| Liskov's influence on both James and Brian.
|
| [1] https://www.youtube.com/watch?v=ps106zjmjhw
|
| [2] https://www.tigerbeetle.com
| pinewurst wrote:
| Not only late, but strangely Google-centric as though we lived
| in caves before the Google monolith appeared.
| macintux wrote:
| Tandem started selling their NonStop systems in the mid-70s, so
| clearly high availability was important.
|
| At Basho we worked hard to acknowledge our debt to distributed
| systems pioneers. Not everyone is ignorant of that history.
| notacoward wrote:
| Yup, the Basho folks were/are very good about that. BTW hi,
| we've quite likely met, though I don't know which Basho alum
| you are. ;) And yes, even I was a pretty late entrant to the
| HA game. I never worked on Tandem systems, but I worked with
| Tandem people at the job I mentioned, after having worked
| with both Stratus and Sequoia systems previously. Those were
| all FT rather than HA, but the point remains that work on
| making systems survive failures through redundancy goes back
| _much_ further than the OP mentions.
| macintux wrote:
| Oh, definitely, was trying to add to rather than detract
| from your point.
|
| I was mainly on the periphery at Basho, started as tech
| evangelist, migrated to engineering to help get 2.0 out the
| door. John Daily.
| bogomipz wrote:
| I always thought Basho was a great company with a great
| product. I felt like I learned a lot through Basho
| documentation. I never really understood what happened to the
| company though as it seemed highly regarded amongst the dev
| community as well and Riak was always mentioned in the same
| sentence as the other distributed data stores at that time.
| macintux wrote:
| Tough market, limited potential customer base for such a
| specialized tool, some product/release planning issues with
| Riak 2.0, and ownership structure that didn't make anything
| easy at the end (but I'm far from an expert on this topic,
| especially the last one, just parroting distant memories).
|
| Engineering was very talented, and the customer support
| team was truly top notch. Such a shame, best job I've ever
| had.
| icedchai wrote:
| Yes, there wasn't much "history" there. I worked on an AIX
| system that used HACMP. Before that, I used Veritas clustering
| and file system on Suns. Prior to that, I was at a VMS shop
| that made use of DEC's built in clustering.
___________________________________________________________________
(page generated 2022-04-27 23:03 UTC)