Post AUpHpmOCqvt4BS3tgm by mcg@social.lol
(DIR) More posts by mcg@social.lol
(DIR) Post #AUojIlMMGPqCzZHYxc by tek@freeradical.zone
2023-04-19T17:48:42Z
0 likes, 0 repeats
Last night's AWS hardware maintenance on our production DB instance took 11 hours 49 minutes to complete.Just in case, you know, that wasn't a scenario you'd previously considered.
(DIR) Post #AUokgTpVBudhWnms9g by A0bMJlR1vXmshMJTm4.pagrus@tears.intherain.club
2023-04-19T18:03:55Z
0 likes, 0 repeats
@tek egads, it's things like that that'd drive a person to cockroachdb
(DIR) Post #AUokgUsNIiX0m08gGu by A0bMJlR1vXmshMJTm4.pagrus@tears.intherain.club
2023-04-19T18:04:08Z
0 likes, 0 repeats
@tek (if that person was me, I guess)
(DIR) Post #AUomXuh88dVU9NukFM by tek@freeradical.zone
2023-04-19T18:24:58Z
0 likes, 0 repeats
@pagrus It was definitely driving me toward *something*.
(DIR) Post #AUomizRynUai5L6qVk by tek@freeradical.zone
2023-04-19T18:27:03Z
0 likes, 0 repeats
I got the highest praise of my career this morning. I was talking about the outage with our CISO, and she said watching our engineering team respond to the outage was like watching an episode of Star Trek. I told her Scotty was my role model, and she said that was exactly how it looked from the outside.That's the nicest thing someone's said to me in ages.
(DIR) Post #AUomqGg590qdXgAybg by case@freeradical.zone
2023-04-19T18:28:04Z
0 likes, 0 repeats
@tek đź‘Š
(DIR) Post #AUomysB1bjIa9zkiKu by nomad@hackers.town
2023-04-19T18:28:56Z
0 likes, 0 repeats
@tek When was the last time you exclaimed she canna take it anymore?
(DIR) Post #AUon5VkhNd2kdebVVA by r343l@freeradical.zone
2023-04-19T18:29:45Z
0 likes, 0 repeats
@tek *blinks*That is way too long. That's long enough that if you had a hot standby you'd flip to it and make it the primary and just get rid of the under maintenance one. If you can.
(DIR) Post #AUonVLg9SHHovSnUpM by cinja@hackers.town
2023-04-19T18:33:07Z
0 likes, 0 repeats
@nomad @tek Oh god.. Current Status:(This was posted by @mikemathia earlier today, totally stealing it though as it's the current thought process)
(DIR) Post #AUonVMJr4f2subNCiG by tek@freeradical.zone
2023-04-19T18:35:41Z
0 likes, 0 repeats
@cinja @nomad @mikemathia I literally LOLed. Thank you.
(DIR) Post #AUonb1m0NP7YvYF8oy by tek@freeradical.zone
2023-04-19T18:36:13Z
0 likes, 0 repeats
@nomad I didn't use those exact words, but the sentiment was there.
(DIR) Post #AUonpTs0i4JQ7dwaZs by tek@freeradical.zone
2023-04-19T18:39:12Z
0 likes, 0 repeats
@r343l That's what we ended up doing, sort of. We didn't have a hot standby for the whole cluster, so we blocked writes to the old, stood up a new one from a point in time snapshot, then pointed services at the new DB.For giggles, we left the old cluster running to see if/when it'd finish finish. I clicked the “apply changes" button at 7:00PM. It was done at 6:49AM this morning. I was annoyed at the idea that it might finish like 2 minutes after we'd completed the cutover, but no.
(DIR) Post #AUoo9abTNx1snSnUQa by dennix@octodon.social
2023-04-19T18:43:06Z
0 likes, 0 repeats
@tek * Exceeds SLAs* Hides alcohol at work* Gets into fights A LOT* Utilzes undocumented and unsafe features👍​
(DIR) Post #AUooRah3mybeHxzi4m by tek@freeradical.zone
2023-04-19T18:45:55Z
0 likes, 0 repeats
@dennix Are you on my team?!(Minus the alcohol bit, unless you count beer in the home fridge, and replace fights with “discussions about strongly held opinions borne from experience”.)
(DIR) Post #AUoot8l1FSdXkqyzNQ by mav@hackers.town
2023-04-19T18:50:55Z
0 likes, 0 repeats
@tek That is indeed the highest of praise.
(DIR) Post #AUopSvvMRIOsNVt54S by dennix@octodon.social
2023-04-19T18:56:27Z
0 likes, 0 repeats
@tek Unfortunatly not, I joined a Pakled engineering crew :/
(DIR) Post #AUoqDsMbUlGMwh0vXk by r343l@freeradical.zone
2023-04-19T19:05:54Z
0 likes, 0 repeats
@tek Oh phew. Your original toot I feared you were in a situation of having to be down for the entire time. But what an annoying scramble.
(DIR) Post #AUotdxE4L6LFPx9HHM by mikemathia@ioc.exchange
2023-04-19T19:44:34Z
0 likes, 0 repeats
@tek @cinja @nomad My pleasure.
(DIR) Post #AUoyNcfvThuConD6xs by tek@freeradical.zone
2023-04-19T20:37:39Z
0 likes, 0 repeats
@r343l Ah, nah, we wouldn't have put up with that. But yeah, the unplanned scramble did not spark joy.
(DIR) Post #AUoyZmkxp1wtYlMmUy by tek@freeradical.zone
2023-04-19T20:39:40Z
0 likes, 0 repeats
@dennix Oh dear.
(DIR) Post #AUpHpmOCqvt4BS3tgm by mcg@social.lol
2023-04-20T00:15:32Z
0 likes, 0 repeats
@tek Was this running on EC2 or a managed service like RDS or Aurora?
(DIR) Post #AUpJ1OcBAiufsRhV6u by tek@freeradical.zone
2023-04-20T00:28:56Z
0 likes, 0 repeats
@mcg RDS.
(DIR) Post #AUpJBiWI1KTkVVrpmi by mcg@social.lol
2023-04-20T00:30:49Z
0 likes, 0 repeats
@tek Ok that’s scary.