[HN Gopher] Southwest Airlines pauses departures after reporting...
___________________________________________________________________
Southwest Airlines pauses departures after reporting technical
problems
Author : jbredeche
Score : 64 points
Date : 2023-04-18 15:00 UTC (8 hours ago)
(HTM) web link (www.cnbc.com)
(TXT) w3m dump (www.cnbc.com)
| nothingneko wrote:
| This is their second time if I'm not mistaken. It's sad that
| there's no changes to prevent this.
| Paul-Craft wrote:
| > Southwest had a meltdown over the year-end holidays, which
| drove thousands of flight cancellations and resulted in an $800
| million pretax hit for the fourth quarter. The airline
| _upgraded scheduling software_ that couldn't handle numerous
| changes to crew assignments during severe winter weather late
| last year.
|
| > The issues Tuesday _did not appear to be related to that
| software._
|
| From the article, emphasis added.
| borner791 wrote:
| https://news.ycombinator.com/item?id=34149340
| 0zemp1c wrote:
| this is why I walk everywhere - no tech debt!
| xeromal wrote:
| An earthquake is a form of tech debt.
|
| I think.
| bastardoperator wrote:
| I love how they immediately place blame on the vendor. You would
| think an operation of this scale and importance would have some
| redundancy built in or after the previous disaster would have
| attacked single points of failure.
| Paul-Craft wrote:
| Where do they "place blame on the vendor?" What SPoF should
| they have addressed? They blame "a vendor-supplied firewall,"
| which is different from the scheduling software that caused the
| incident over the holidays, and different from blaming the
| vendor itself.
| [deleted]
| bastardoperator wrote:
| Are we reading the same article?
|
| "Early this morning, a vendor-supplied firewall went down and
| connection to some operational data was unexpectedly lost,"
|
| They're blaming the vendor and absolving themselves of any
| responsibility. If losing a connection causes catastrophic
| failure, who's really at fault?
| Paul-Craft wrote:
| That's a description of _what_ happened. It doesn 't say
| _why_ it happened. Nobody is being blamed for anything. For
| all we know, somebody at Southwest tripped on a power cord
| and shut down the machine running the firewall. Facetious,
| yes, but I have once had someone literally trip on a power
| cord for a machine running test software I cared about,
| shutting it down. (Yes, we fixed the issue and got UPS 's
| for these machines after that lol)
| AnimalMuppet wrote:
| At a minimum, that firewall appears to have been a single
| point of failure.
| jrockway wrote:
| That's how security stuff intrinsically works. It sits in
| the critical path and fails closed. If it failed open,
| then it would be "malware installed on Southwest's
| production network due to unpatched node.js module" or
| something.
| unethical_ban wrote:
| Network security devices are often deployed with high
| availability baked in. If one firewall appliance dies on
| a production system, it should have a peer to failover
| to.
|
| We don't know what happened, but at least for hardware,
| HA is the standard.
| m348e912 wrote:
| >> [security stuff] sits in the critical path and fails
| closed.
|
| Not entirely accurate. Some inline IPS appliances fail
| open, and networks can be architected to fail around a
| single firewall in a high availability pair.
| [deleted]
| bink wrote:
| That's a dramatic over-simplification. You have redundant
| / clustered firewalls that allow for testing
| configuration changes before they're deployed everywhere
| so you never end up in a situation where you have to
| "fail closed". You have redundant links into critical
| data centers, either hot/hot or hot/warm. You have
| redundant power supplies in the racks running the
| firewalls and routers. You have redundant data centers
| (where necessary).
| dilyevsky wrote:
| Nothing stopping you from running redundant links with
| redundant appliances, except beancounters ofc
| jrockway wrote:
| I guess I've accepted that it's something intrinsic to
| the software that's going to make it fail. Bad config
| push, bad packet segfaults the packet inspection
| software, etc. (Yup, people still write parsers for
| untrusted user input in C, and don't fuzz it. That's the
| state of the industry.) In those cases, all of your
| replicas fail identically at the same time.
|
| For mere hardware failure, we seem to have moved past
| that as an industry. Cloud providers will "transparently"
| migrate your app to a new computer if the one you're
| running on blows up. (Transparently is in quotes because
| you'll see the P99.9 latency increase when it happens.)
| ano-ther wrote:
| Interesting approach. What does it even mean?
|
| Wouldn't most of an airline run on "vendor supplied
| $thing"?
|
| "Today, a vendor-supplied plane went down [...]"
| unethical_ban wrote:
| True, but being technically accurate isn't bad. If a vendor
| owns it, a vendor owns it.
|
| Not having HA on a network segment is bad, though.
| bastardoperator wrote:
| Can we validate the accuracy? Probably not, but I can say
| in my own postmortem or RCA analysis taking zero
| responsibility or deflecting is a sign of a greater
| problem. There is always room for learning and growing.
| unethical_ban wrote:
| Saying a fact isn't deflection. I have been in post-
| mortems too, and accuracy is important.
|
| If I work at a call center and there is an outage on a
| partner's PBX, then the summary will say "vendor PBX
| caused an outage". Part of that conversation will be "how
| to we mitigate the risks of a vendor PBX outage in the
| future", but the technical fact stands: Someone else's
| equipment failed.
| Paul-Craft wrote:
| You can't claim they aren't taking responsibility unless
| you've seen the internal postmortem. External messaging
| is PR. It's designed to not affect the share price. You
| can't extrapolate anything about internal process from
| it.
| bastardoperator wrote:
| You seem to be pretty invested here, everything okay?
| It's shit PR, devoid of any responsibility, and they're
| already in the dog house when it comes to stock value.
| For me personally, I just won't fly this airline, I can't
| trust them on the ground why would I put my life or
| valuable time in their hands or reward them with
| business?
|
| We don't need to a postmortem, the blame has been solely
| laid on someone else's shoulders. I think the only thing
| we can extrapolate from this conversation is that you
| possibly work for SW, or that you're upside down on SW
| shares. Have a great day and welcome to HN.
| javawizard wrote:
| It reminds me of the saying: "when you owe the bank $100 that's
| your problem; when you owe the bank $100 million that's the
| bank's problem"
|
| It doesn't matter if fault lies with an external vendor, people
| can't fly and Southwest is still going to take reputational
| damage as a result.
| TylerE wrote:
| The way I've heard it, which I think is a bit punchier is "If
| you owe the bank $10,000 dollars, the bank owns you. If you
| owe the bank $100 million dollars, you own the bank."
___________________________________________________________________
(page generated 2023-04-18 23:02 UTC)