[HN Gopher] Wolf Incident Postmortem
___________________________________________________________________
Wolf Incident Postmortem
Author : zzq
Score : 91 points
Date : 2023-01-19 17:43 UTC (5 hours ago)
(HTM) web link (www.jefftk.com)
(TXT) w3m dump (www.jefftk.com)
| dvh wrote:
| Reminds me of an admin story where each incident ended with
| updating /etc/motd
| Communitivity wrote:
| This is awesome. Job well done, and hats off sir.
| barbazoo wrote:
| I don't get the joke but I browsed a bit, got to the story about
| "MA RMV Overloaded" [0] where they tried to renew their driver
| license. I was baffled by the fact that it was possible at all
| and even faster to renew your official driver license through a
| private company (AAA) than through their DMV office.
|
| [0] https://www.jefftk.com/p/ma-rmv-overloaded
| kayodelycaon wrote:
| It's a reference to this story:
| https://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf
|
| :)
| mlyle wrote:
| It's about the "boy who cried wolf" vs how incident management
| works.
|
| False positives can make real issues get an insufficient
| response when they occur (e.g. sentinel getting eaten).
| jandrese wrote:
| False positives can be a far more serious issue than people
| are willing to admit. There are plenty of loss of life
| incident reports that include wording like "the operator
| disabled the alert system due to numerous false positives"
| near the beginning.
|
| An example: https://www.youtube.com/watch?v=1zDcsjHyxr8
| thebetatester wrote:
| It took me longer than I care to admit to figure out this wasn't
| talking about a real incident. I figured flock was about some
| Kubernetes/VM thing.
| michaelnoguera wrote:
| Context for those wondering: The Boy Who Cried Wolf
|
| https://en.wikipedia.org/wiki/The_Boy_Who_Cried_Wolf#The_fab...
| msla wrote:
| Here I thought Jeff K was a sly fox, not a wolf.
| pixiemaster wrote:
| Cruel.
| cialowicz wrote:
| Well done. Seems like a good prototype to show someone who is
| unfamiliar with how a postmortem should be written up.
| paulgb wrote:
| Indeed. Bookmarked it for exactly this.
| h2odragon wrote:
| Sensor was faulty, sensor replaced, system functioning normally.
| 0xbadcafebee wrote:
| The only wolf evidence are alerts by a now-missing sentry saying
| "wolf" twice and "real wolf" once. Yet the conclusion due to
| missing sentry is 'wolf', and not only that, that there must have
| been only one wolf rather than a pack. If you don't have
| supporting evidence, this conclusion may be faulty, and the real
| fault may return.
|
| My initial reaction is that there is no wolf. The sentinel began
| exhibiting a mental bug that reoccurred for 2 days until the
| sentinel broke and wandered off. I think the resolution should
| have an action item to continue closely monitoring sentinels for
| further bugs, and collect metrics to show proof of wolf. And, you
| know, maybe come up with an additional wolf countermeasure (if
| the fault was a wolf, it will probably come back)
| layer8 wrote:
| A very good observation: https://www.jefftk.com/p/wolf-incident-
| postmortem#1096589888...
| abeppu wrote:
| I really prefer when the "postmortem" label is metaphorical.
___________________________________________________________________
(page generated 2023-01-19 23:02 UTC)