Post Axr4bDpPBZA3M0DWzY by matrix@mastodon.matrix.org
(DIR) More posts by matrix@mastodon.matrix.org
(DIR) Post #AxoD9rvGpzTZi5738i by matrix@mastodon.matrix.org
2025-09-02T17:57:00Z
0 likes, 1 repeats
the matrix.org homeserver is having problems: https://status.matrix.org/incidents/mm9hdm78svgv apologies for the inconvenience…
(DIR) Post #AxoD9syUvTeSyNd8oC by matrix@mastodon.matrix.org
2025-09-02T19:01:27Z
1 likes, 0 repeats
So: the matrix.org database secondary lost its FS due to a RAID failure earlier today (11:17 UTC). Then, we lost the primary at 17:26. We're trying to restore the primary DB FS (which could be fastish), while also doing a point-in-time backup restore from last night (which takes >10h). We believe the incremental DB traffic since last night is intact however. Apologies for the downtime; folks on their own homeserver are of course not impacted.
(DIR) Post #AxoD9uDQJTBWoxcZiC by vincep@piaille.fr
2025-09-02T19:43:32Z
0 likes, 0 repeats
@matrix jokes aside, RAID failures are NOT fun. Props for the quick reaction and godpseed!
(DIR) Post #AxoD9zoTV8nGBJzjeq by matrix@mastodon.matrix.org
2025-09-02T21:39:25Z
4 likes, 3 repeats
Sorry, but it's bad news: we haven't been able to restore the DB primary filesystem to a state we're confident in running as a primary (especially given our experiences with slow-burning postgres db corruption). So we're having to do a full 55TB DB snapshot restore from last night, which will take >10h to recover the data, and then >4h to actually restore, and then >3h to catch up on missing traffic. Huge apologies for the outage. Again, folks using their own homeservers are not impacted.
(DIR) Post #Axooft7Jd4x4z6sUCG by hisold@toot.io
2025-09-02T19:35:41Z
1 likes, 0 repeats
@matrix This is why we need more decentralization which happens to be the goal of matrix.
(DIR) Post #AxooftiXOgj4qYIDDM by tyil@fedi.tyil.nl
2025-09-03T05:00:01.287Z
0 likes, 0 repeats
@hisold@toot.io @matrix@mastodon.matrix.org The Matrix homeserver is so unwieldy and such a massive resource hog that very few people are willing to host it themselves.
(DIR) Post #AxowWGGrUJA3X6GP8i by AJCxZ0@fosstodon.org
2025-09-03T01:08:26Z
0 likes, 1 repeats
@matrix Godspeed, admins!
(DIR) Post #AxowYOxe9Joj41IR9c by crispycat@mastodon.calitabby.net
2025-09-02T22:29:13Z
0 likes, 1 repeats
@matrix
(DIR) Post #AxownGuKNGzyQipnzE by mrclon@mastodon.ml
2025-09-03T06:31:04Z
0 likes, 0 repeats
@matrix it's remainder that Matrix network to concentrated on matrix.org.Use another homeservers, my dudes
(DIR) Post #Axr4bDpPBZA3M0DWzY by matrix@mastodon.matrix.org
2025-09-03T07:09:49Z
0 likes, 0 repeats
Status update: we’re 47TB through restoring the 55TB db snapshot of the matrix.org DB, but then have to rebuild the DB and replay the subsequent 17h of DB traffic, which will take several hours. Thank you for your patience, and apologies once again for the outage.
(DIR) Post #Axr4bF5OVbXrFshoYK by matrix@mastodon.matrix.org
2025-09-03T10:57:00Z
0 likes, 0 repeats
Status update: we've restored the 55TB snapshot and subsequent incremental backups, and are about to replay the remaining traffic since the backup. There are still several unknowns, but if things go well the matrix.org instance should be back in 3-4 hours.
(DIR) Post #Axr4bFqtezXhdCvkau by matrix@mastodon.matrix.org
2025-09-03T17:42:21Z
0 likes, 1 repeats
Right, matrix.org is back online as of 17:00 UTC. The server is struggling a bit as it catches up. Huge apologies again for the outage; postmortem + ways to avoid a repeat will be forthcoming. See also https://www.theregister.com/2025/09/03/matrixorg_raid_failure/ & https://www.heise.de/en/news/Matrix-main-server-down-millions-of-users-affected-10630524.html. Thanks all for your patience.