Post APipNvC8igUUaX9y1A by grapheneos@infosec.exchange
 (DIR) More posts by grapheneos@infosec.exchange
 (DIR) Post #APipNvC8igUUaX9y1A by grapheneos@infosec.exchange
       2022-11-18T01:13:38Z
       
       0 likes, 0 repeats
       
       There have been major federation issues on Matrix with matrix.org today and many users in #grapheneos:grapheneos.org were dropped from the room and unable to rejoin. We think we've fixed the joining issue by setting it to invite-only and back to public.We aren't able to dedicate resources to identifying what went wrong with the Matrix protocol/federation. This could be a serious Matrix state reset bug. We'd appreciate help with figuring out the member list before it occurred so that we can reinvite 1.5k people who were dropped.We've filed an issue about this at https://github.com/matrix-org/synapse/issues/14481.We used to regularly experience these serious state reset / corruption issues with our previous rooms. We haven't experienced it since we created new Matrix rooms after the freenode takeover to replace our IRC channels.If you're having trouble (re)joining our main Matrix room, please let us know. Consider joining our discussion forum at https://discuss.grapheneos.org in the meantime. Matrix room tends to be nicer for getting quick support and short form answers but our forum is often a better fit now.#grapheneos #matrix #matrixchat #matrixdotorg
       
 (DIR) Post #APipNvhKqhRc9Hksds by deepbluev7@fosstodon.org
       2022-11-18T02:03:21Z
       
       0 likes, 0 repeats
       
       @grapheneos @matrix Maybe a good thing to look into, since this is not the first community hit by this.
       
 (DIR) Post #APipNwJGZfmm2vVAlU by grapheneos@infosec.exchange
       2022-11-18T02:55:46Z
       
       0 likes, 0 repeats
       
       @deepbluev7 @matrix It's strange since our room is version 6 and part of the reason we recreated it was to escape the completely broken early version(s) we were stuck with for our previous generation rooms. As far as we know, we shouldn't be hitting this issue with room version 6 but we just lost over 10% of users in the room to a similar problem.
       
 (DIR) Post #APipNwkYwBcVPaGyJM by deepbluev7@fosstodon.org
       2022-11-18T02:58:07Z
       
       0 likes, 0 repeats
       
       @grapheneos @matrix I have seen them before. Techlore had them a lot and a few other large communities. I think it is an implementation bug in some homeservers. I recently heard they found some reason for it, but not sure if they could fix it yet and it might not be your issue. Having some cases that happened recently to look at logs might help though.
       
 (DIR) Post #APipNxG72srCzR2AUK by grapheneos@infosec.exchange
       2022-11-18T04:31:33Z
       
       0 likes, 0 repeats
       
       @deepbluev7 @matrix It was a major issue with room version 1 and potentially some of the other early versions but as far as we know this is supposed to be solved at this point. On the positives side, the massive rollback in the number of users in our rooms was partially corrected after some users joined and seemingly triggered partial recovery of the state. It initially dropped from ~14900 to ~13300 across servers but has now gotten back to ~14400 with only a couple users joining. It's a weird situation. Still missing a bunch of users who may be unable to rejoin.
       
 (DIR) Post #APipNxlJAtoKYBd572 by matrix@mastodon.matrix.org
       2022-11-18T08:06:07Z
       
       0 likes, 0 repeats
       
       @grapheneos @deepbluev7 ugh, sorry. thanks for filing the bug; we’ll dig into it asap.