[HN Gopher] Cold restart whole system after total outage
___________________________________________________________________
Cold restart whole system after total outage
Author : dmazin
Score : 25 points
Date : 2023-07-19 20:06 UTC (2 hours ago)
(HTM) web link (www.evalapply.org)
(TXT) w3m dump (www.evalapply.org)
| jacquesm wrote:
| This is very interesting in the context of power infrastructure
| as well. As we found out the hard way during the 2003 power
| blackout in North America.
| rdhatt wrote:
| Practical Engineering did a video on the complexity of bringing
| a power grid back online, called "black start" (not cold
| start).
|
| https://practical.engineering/blog/2022/12/5/what-is-a-black...
| lantry wrote:
| In the anecdote about Bill and the DISASTER script, I'm not so
| sure that deleting the script would be such a big deal. If this
| script hasn't been touched since the 1980s and nobody knows what
| it does, presumably nobody has tested it recently.
|
| It seems like if there really was a disaster, first of all nobody
| would know that script existed, and second of all if they tried
| to run the script, it would fail because of all the changes to
| the system since the script was initially developed.
|
| Isn't there some saying like "if you don't test your backups, you
| don't have backups" or something like that?
| LeoPanthera wrote:
| I bet it still would have been a useful template for a human to
| read to get a general idea of what things to do and in what
| order.
| wkdneidbwf wrote:
| good luck reading 10k lines of shell written decades ago. it
| would likely be an incredible waste of time.
| wkdneidbwf wrote:
| right? that whole bit reads like some lame parable. like who in
| there right mind is going to run a 10k line shell script named
| DISASTER they've never read and cannot read because it's 10k
| lines of shell? there is apparently no documentation (and
| positively no tests)? one guy close to retirement remembers
| what it's for and says "don't delete this critical but of
| code!"
|
| it's just utter bullshit.
| anotherhue wrote:
| IMO if you can't cold start it you probably can't develop against
| it very quickly.
|
| Then again we couldn't cold start a supply chain or a semi fab or
| humanity itself so maybe that's the default.
| bamfly wrote:
| You may enjoy the first episode of James Burke's _Connections_
| ( "The Trigger Effect"), if you've not seen it.
|
| https://www.youtube.com/watch?v=NcOb3Dilzjc
| anotherhue wrote:
| I enjoyed the one in the Witness but hadn't gotten around to
| the rest, thanks for the excellent recommendation!
| https://archive.org/details/james-burke-connections_s01e10
| JohnFen wrote:
| Every new semi fab that comes online was cold-started.
| anotherhue wrote:
| With the output of the prior generations was my point.
| johngalt wrote:
| At a certain point, you aren't doing a cold restart, but a high
| speed recreation of the system based on prioritized needs.
| tivert wrote:
| > Another colleague in the chat remarked up-thread (apropos cold
| reboot thinking):
|
| > I have seen this at <Indian eCommerce Giant> and at <a FAANG>.
| Most of it is related to cached data. Cold starts with empty
| caches causes too much load on databases. And then the failures
| cascade.
|
| > -- Another M'colleague in the Slackroom.
|
| Isn't that not really a problem with cold restart per se, but
| more the restart procedure? If caches are so critical, wouldn't
| you need a feature to throttle the load to what the databases can
| handle, as the caches populate? E.g., if you're cold-rebooting
| Facebook, start by blocking all connections except those
| geolocated to North Dakota, then add other regions as your caches
| fill.
| nickdothutton wrote:
| It's been a few years but I used to run DR exercises for
| corporates. Cold start means your only possessions are the fire
| proof suitcase full of LTO-5s and the street address of the DR
| data center. 1 day to bootstrap essential infra services, after
| the end of the 2nd day you'd have most customer facing systems
| up, day 3 would be the non-essential stuff. Personally I'd do it
| without sleep, but most of the youngsters would need a break.
| Pretty exhilarating, as IT work goes. Always use the feature that
| generates multiple index tapes of what backup set is on what
| numbered tape :-)
| gumby wrote:
| The full telephone system, which the author starts with, may not
| be restartable. Sure, you could restart the SS7 databases and
| computers, but the control plane runs over the data plane, which
| is configured via...the control plane. Originally the network
| controls were literally operators (humans), but bit by bit parts
| were incrementally automated, pulling the system slowly (over
| decades) by its bootstraps, which were gradually decommissioned
| as they weren't needed any more.
|
| I have a friend who knows a _lot_ about the phone system (he has
| a security clearance for some of his telephone work) one time we
| had a long conversation about this topic, until at one point he
| said "and let's talk about something else" -- I guess from that
| point some of the details are classified. So maybe there is a
| plan, or maybe they just designed the system in such a way that
| they could convince themselves that it would not go down unless
| things were so severe that loss of the phone system would not be
| your chief worry.
|
| ---
|
| In September 2001 there was a full standdown of US airspace. That
| was accomplished pretty quickly: "you are ordered to land
| immediately on the closest airport that can handle your aircraft,
| or be shot down". Undoing that, however, took some careful
| planning! Fortunately the standdown lasted several days so there
| was time to work it out. Even if you had a plan for this (and I
| assume FAA had one), figuring out what the realities on the
| ground were and matching them up with the plan was nontrivial.
|
| Apparently some of the planes landed where they could not take of
| again unless they were empty with a small amount of fuel to get
| to an airport designed for them. I don't believe I heard that any
| planes landed where they could _never_ leave.
| protastus wrote:
| > So maybe there is a plan, or maybe they just designed the
| system in such a way that they could convince themselves that
| it would not go down unless things were so severe that loss of
| the phone system would not be your chief worry.
|
| My belief from working in very large companies, and
| (previously) in mission critical systems is that a clean
| bootstrap and recovery process is extremely unlikely, almost
| impossible. Because in complex systems full of legacy parts and
| people who have long retired, the stars won't align.
|
| The only way to truly know is to design and periodically test
| for disaster scenarios (emphasis on the plural). But due to the
| scale in time and space, cost and bureaucracy, this planning
| and rehearsing is not going to happen with the desired detail
| and intensity. People do not seriously plan for things that
| have never happened.
|
| If it does happen, there will be a small group of extremely
| capable people that will find a way to bootstrap the system. It
| won't be according to some previously laid out plans -- they
| will make the plan in real time. They're not famous and
| probably never will be.
| drbawb wrote:
| I'm reminded of Bryan Cantrill's talk "Debugging Under Fire"[1],
| which includes a retrospective of sorts about an entire
| datacenter rebooting.[2] That is a pretty large-scale disaster,
| but even that is a rung below a continent-wide outage. Poor
| "Bill" must have saw the proverbial light when he heard some
| folks wanted to trash the DIASTER script.
|
| [1]: https://www.youtube.com/watch?v=30jNsCVLpAE
|
| [2]: https://www.tritondatacenter.com/blog/postmortem-for-
| outage-...
___________________________________________________________________
(page generated 2023-07-19 23:01 UTC)