Post AWqteVS0cuBI35Wvg0 by tachan@mastodon.social
(DIR) More posts by tachan@mastodon.social
(DIR) Post #AWqSn7V4LWuPbwqPCK by Codeberg@social.anoxinon.de
2023-06-19T09:36:45Z
1 likes, 2 repeats
Hi folks, we are sorry, but this downtime could last a moment. It looks like one of our root SSDs failed. While the RAID seems to work, the performance has dropped to a level where the whole service is impacted, and even our shell on that server is near to unresponsive.
(DIR) Post #AWqSuu9TrHd8WrSU6a by n0toose@chaos.social
2023-06-19T09:38:11Z
0 likes, 0 repeats
@Codeberg "While the RAID seems to work" presumably means that no data loss happened.
(DIR) Post #AWqSzYJiT2wUG0Sj2m by vintprox@techhub.social
2023-06-19T09:38:26Z
0 likes, 0 repeats
@Codeberg Get well soon.
(DIR) Post #AWqTIkmEMkpQP2xIhc by bytter@fosstodon.org
2023-06-19T09:42:35Z
0 likes, 0 repeats
@Codeberg Hope nothing serious happened, do you know how much itll take to pull it up?
(DIR) Post #AWqU04Ba51UD3CSxUG by uncomfyhalomacro@julialang.social
2023-06-19T09:50:08Z
0 likes, 0 repeats
@Codeberg Thanks for letting the community know! 🥰
(DIR) Post #AWqVGjCxQbPorlcQEK by Codeberg@social.anoxinon.de
2023-06-19T10:04:42Z
0 likes, 0 repeats
@n0toose Damn, would have been a great stresstest for data recovery. No, we can confirm at this point, no data loss happened.
(DIR) Post #AWqVNIWX2PuC8KuIkK by n0toose@chaos.social
2023-06-19T10:05:49Z
0 likes, 0 repeats
@Codeberg Thanks for working on this.
(DIR) Post #AWqVSEWnJzw3BlNpdg by foxy@mastodon.online
2023-06-19T10:06:09Z
0 likes, 0 repeats
@Codeberg Good work and get well soon❤️
(DIR) Post #AWqVYl1FQuLRhCjrCC by ltlnx@g0v.social
2023-06-19T10:06:56Z
0 likes, 0 repeats
@Codeberg Take care & get well soon!
(DIR) Post #AWqVe0qtzdlyFWPCKW by daviwil@fosstodon.org
2023-06-19T10:08:06Z
0 likes, 0 repeats
@Codeberg Thanks for your hard work on getting things back online! I am very appreciative of what Codeberg does for all of us.
(DIR) Post #AWqXSzjK1cxRUXjYYK by EredYasibu@mastodon.ml
2023-06-19T10:29:09Z
0 likes, 0 repeats
@Codeberg @n0toose this is good
(DIR) Post #AWqeMOvGf0ktHG7lei by jssfr@zombofant.net
2023-06-19T11:46:20Z
0 likes, 0 repeats
@CodebergGood luck with the recovery! Your service is much appreciated.
(DIR) Post #AWqgZ52M6dWlHAc5Bo by xavier@sunny.garden
2023-06-19T12:11:04Z
0 likes, 0 repeats
@Codeberg Thanks for the work you put into this 😊
(DIR) Post #AWqglspin5CbBuoCjQ by Codeberg@social.anoxinon.de
2023-06-19T12:13:38Z
0 likes, 0 repeats
We're back for now. We'll try to keep everything running until we are able to install the spare hardware.
(DIR) Post #AWqgzwTC4BJEWYyWgK by gothnbass@linuxrocks.online
2023-06-19T12:15:37Z
0 likes, 0 repeats
@Codeberg Welcome back!
(DIR) Post #AWqh59a6kHlEdXh6no by claudius@imd.social
2023-06-19T12:16:54Z
0 likes, 0 repeats
@CodebergGlad to have you back ❤️
(DIR) Post #AWqhJyPEZs7IJwftwW by n0toose@chaos.social
2023-06-19T12:19:39Z
0 likes, 0 repeats
@Codeberg Awesome!
(DIR) Post #AWqhfULp0C1hTfL0pU by pax@mstdn.social
2023-06-19T12:23:25Z
0 likes, 0 repeats
@Codeberg what a bad failure.high hopes that my projects didn't disappeared due to the failure.
(DIR) Post #AWqiPnG52MKdRFjSyW by louis@emacs.ch
2023-06-19T12:32:38Z
1 likes, 0 repeats
@Codeberg Thumbs up for all your work!
(DIR) Post #AWqjsqgrSKukR0g3wO by communistcapy@c.im
2023-06-19T12:48:18Z
0 likes, 0 repeats
@Codeberg What sort of precautions can you take that might prevent this from happening again?
(DIR) Post #AWqjwIDrY2sCh2kvS4 by Codeberg@social.anoxinon.de
2023-06-19T12:48:49Z
0 likes, 0 repeats
@communistcapy Replace the root SSDs. Prepare more servers (WIP).
(DIR) Post #AWqkGXdnvsK6IvXrcm by Codeberg@social.anoxinon.de
2023-06-19T12:52:42Z
0 likes, 1 repeats
We'll have another downtime. We want to fix some things before the CEST-evening with the highest traffic amount starts.Our root SSDs are dying, probably because of a firmware bug. We'll try to update the firmware now.
(DIR) Post #AWqoUnPqTiZ56G4JJw by quazaromega@mastodon.green
2023-06-19T13:39:56Z
0 likes, 0 repeats
@Codeberg Thanks for keeping us updated!
(DIR) Post #AWqok5DuYKIsflmAKm by Codeberg@social.anoxinon.de
2023-06-19T13:42:57Z
0 likes, 1 repeats
Firmware update succeeded, everything looks OK for now. We'll monitor, but expect Codeberg to stay online for the time being.We'll have another scheduled downtime once the replacement SSDs arrive.
(DIR) Post #AWqrAlB5OCteEIIovQ by datenritter@digitalcourage.social
2023-06-19T14:09:57Z
0 likes, 0 repeats
@Codeberg Let me guess: Samsung 980 Pro?
(DIR) Post #AWqrZegeevyLL1wJqy by Codeberg@social.anoxinon.de
2023-06-19T14:14:41Z
0 likes, 0 repeats
@datenritter exactly
(DIR) Post #AWqrhUcXy9SWR1ktvM by Codeberg@social.anoxinon.de
2023-06-19T14:16:04Z
0 likes, 0 repeats
@pax We do not expect any data loss.
(DIR) Post #AWqteVS0cuBI35Wvg0 by tachan@mastodon.social
2023-06-19T14:36:51Z
0 likes, 0 repeats
@Codeberg Thanks for the updates and for all the hard work!
(DIR) Post #AWqu54LDWkMnmy1GFc by datenritter@digitalcourage.social
2023-06-19T14:42:34Z
0 likes, 0 repeats
@Codeberg Das ist hinter der Aufregung um die 990er untergegangen: https://blog.datenritter.de/archives/716-Backups!-14-SSD-Samsung-980-Pro-2TB-ein-Streifschuss.html
(DIR) Post #AWqunlhhfPpj18qqVk by joel@functional.cafe
2023-06-19T14:50:26Z
0 likes, 0 repeats
@Codeberg what storage strategy do you use in your services?Just curious. And, I’m not a specialist either.
(DIR) Post #AWqus1Yye1iXMWnbYu by aral@mastodon.ar.al
2023-06-19T14:50:29Z
0 likes, 0 repeats
@Codeberg Thank you.💕
(DIR) Post #AWqvbURg6jsZyE453o by Codeberg@social.anoxinon.de
2023-06-19T14:59:45Z
0 likes, 0 repeats
@joel For the root filesystem, we went for a RAID-1 nvme SSD system based on btrfs. Btrfs has built-in checksumming, so you don't need a third drive to decide which version of the data is correct.This is used as root filesystem for the containers and for databases.Other data (Git repos, LFS, attachments, packages) are stored in a tiny but growing Ceph cluster, currently using a mixture of SSDs (performance) and HDDs( cheap redundancy).
(DIR) Post #AWqwncIipJZtlNDpBY by pixelcode@social.tchncs.de
2023-06-19T15:12:45Z
0 likes, 0 repeats
@Codeberg Thank you, as always! :)
(DIR) Post #AWr24V7pG69uCYznSy by moonglum@social.yakshed.org
2023-06-19T16:11:58Z
0 likes, 0 repeats
@Codeberg ❤️
(DIR) Post #AWr2rQJ1xuHFRcZ4W8 by GreyKraken@floss.social
2023-06-19T16:20:56Z
0 likes, 0 repeats
@Codeberg Thanks for the work and dedication guys!!!
(DIR) Post #AWr3FfQ5QLyfGkQQgS by pjbrunet@noagendasocial.com
2023-06-19T16:26:09Z
0 likes, 0 repeats
@datenritter @Codeberg I would have guessed Sandisk :-/
(DIR) Post #AWr8Fk9qJ98tjbgPxo by ronix@sueden.social
2023-06-19T17:21:19Z
0 likes, 0 repeats
@Codeberg Ah, that sounds so familiar. Good faith and good work!
(DIR) Post #AWsARzFMQXwX4dBq1Q by vazub@mastodon.online
2023-06-20T05:20:30Z
0 likes, 0 repeats
@Codeberg Thank you!
(DIR) Post #AWsoUljB8xUmBF1CcK by pax@mstdn.social
2023-06-20T12:49:12Z
0 likes, 0 repeats
@Codeberg and there's none, unispeak is there, npprw is there, as private repo, so all is ok.
(DIR) Post #AWvuYonaqE7mqSDw36 by melroy@mastodon.melroy.org
2023-06-22T00:41:17Z
0 likes, 0 repeats
@Codeberg That doesn't sound good.. I think you are still impacted by this performance hit.
(DIR) Post #AXBvma8Pqv6lxk7i5I by communistcapy@c.im
2023-06-29T18:10:19Z
0 likes, 0 repeats
@Codeberg Well in your defense, github is down right now, so it can happen to anyone! :bd17: