Posts by dean@uwu.social
(DIR) Post #AYz01LAuKbJqJeCuSO by dean@uwu.social
2023-08-22T08:11:50Z
0 likes, 0 repeats
(DIR) Post #AZ15I0nOF5jY7WtbCS by dean@uwu.social
2023-08-23T08:20:17Z
0 likes, 0 repeats
lofi hip-hop beats to 3d print a keyboard case to
(DIR) Post #Ab26ZSiGgjrGoacEQi by dean@uwu.social
2023-10-22T15:53:33Z
0 likes, 0 repeats
Cert renewal was borked on uwu.social again, fixed now! Sorry about that! When I get back from my vacation I’ll fix the bug in the renewal script
(DIR) Post #AiNVPCADEdIw7N6y0m by dean@uwu.social
2024-05-29T08:45:27Z
0 likes, 0 repeats
We had to take mastodon down for about 2 hours due to hitting a kernel bug in devmapper on the storage host. We had to force shutdown and then e2fsck the disks while they were unmounted. Sorry for the inconvenience.
(DIR) Post #AiNuJjdgMLO73YOAPA by dean@uwu.social
2024-05-29T13:24:36Z
0 likes, 0 repeats
We're struggling to figure out a good way to migrate Mastodon media storage from one server to another... Mastodon stores files in millions of directories by hash (kinda like aaa/bbb/ccc/ddd) which means doing something simple like an initial rsync and then taking Mastodon down for a quick resync is impossible. We initially were going to go this route but after the initial file listing took more than 24h we cancelled it and gave up.So now we're looking at just copying the raw filesystem over, but if we want to do it without taking mastodon down for the entire sync we need to come up with a way of copying it and resyncing the changed blocks afterwards.One way could be to use overlayfs. Remount the old volume R/O, create a temporary upperdir and create an overlay between them. Then, copy the R/O image to it's new home, expand it or whatever, and apply the upperdir onto it. This way we only need to list the directories that actually had writes. Special care will need to be taken to ensure we delete any files that have overlayfs tombstones. IDK if anyone has ever done this before.Another way could be to use devmapper snapshots to create a new COW-backed volume, rsync the R/O underlying block device over and then apply the COW to the new volume with snapshot-merge. We tried testing this out and caused devmapper to die horribly and spit out kernel bug log lines, so we had to reboot and e2fsck for 2 hours.At this point it might be better to just take everything down for as long as it takes. I'm extremely annoyed at Mastodon's file structure making it impossible to move without major downtime. Their solution just seems to be "use S3 lol". It would probably take 24 hours (8TB at 1Gbps is roughly 17 hours). We could shrink it first since we don't use all the space, but resize2fs will take a while as well.If anyone has any tips or ideas for doing it with minimal downtime I'd like to hear them. Or if you're an uwu.social user and don't care about extended downtime I'd also like to hear your thoughts too.
(DIR) Post #AiOAFrHHcDT2r75L4C by dean@uwu.social
2024-05-29T16:23:11Z
0 likes, 0 repeats
@Hanuwu I think it would have the same problems as rsync since it has to recursively list files in millions of directories :(
(DIR) Post #AiODHe8fjhe8Y1bMbA by dean@uwu.social
2024-05-29T16:57:07Z
0 likes, 0 repeats
@sulian I don't think you can turn off media without causing a lot of issues. It's (probably) easy to block for remote posts, but blocking for local posts would be difficult. I suppose the upload endpoints could be disabled.Also, any deleted posts from local or remote instances that have media need to have their corresponding media deleted which would fail... and then we'd have orphaned files in the media storage directory forever :(
(DIR) Post #AiPgUeakSIQnXtFTEG by dean@uwu.social
2024-05-30T09:59:08Z
0 likes, 0 repeats
@Jain yeah we purged media before we started, I don't think it speeds up the file listing though because the hash directories get left behind AFAIK :(
(DIR) Post #AiPh5ng1dk3SRAQhgu by dean@uwu.social
2024-05-30T10:05:50Z
1 likes, 0 repeats
@Jain the 8TB is the raw volume size if we were to copy the ext4 partition over instead of rsyncing each file over. Sorry if that wasn't clear 🙏
(DIR) Post #AiPhebLlSk5sLFXziS by dean@uwu.social
2024-05-30T10:12:07Z
1 likes, 0 repeats
@izaya If we were gonna move the FS we'd definitely do it with compression. The other admin is gonna benchmark zstd today on the source and destination servers to see what settings we can use. We could also probably make it more efficient by running e4defrag online first (but it's kinda scary lol)
(DIR) Post #AiPiM2TgThYuvexqRU by dean@uwu.social
2024-05-30T10:19:58Z
1 likes, 0 repeats
@Jain that's a good idea! Might look into this and see how feasible it is for us
(DIR) Post #AiQJKH2J4lHlo2BEKe by dean@uwu.social
2024-05-30T17:14:14Z
0 likes, 0 repeats
@crashdoom @RavenLuni Thanks for the tips! Is there a good way to clean up orphaned files? Maybe some tootctl command or something?
(DIR) Post #AiUVk4LYWcdFd3z8sa by dean@uwu.social
2024-06-01T17:52:12Z
0 likes, 0 repeats
uwu.social will be down for up to 2 hours later today (unsure when we'll start, but probably in 10 or so hours). We will be beginning the server data migration. Once the migration starts, we will restore service until the data migration finishes, then we will go down again for up to 6 hours to finalize the move.We've decided to go with the overlayfs route, and we will be copying the raw partition over in the background rather than listing each file individually. We've tested this procedure on smaller disks and it seems to be durable.Thanks to everyone who gave suggestions about the Linux moving thing. We'll be combining some of the tips we received to make the process smoother.
(DIR) Post #AieQNTcKreQTD6CqIq by dean@uwu.social
2024-06-06T12:39:10Z
1 likes, 0 repeats
We successfully migrated all mastodon media to it's new home. Please let me know if you see any serious issues with media (like no media loading at all, no incoming posts have media for more than a few minutes, uploads failing etc.). Please don't notify me about random posts not having media - this happens all the time.For the curious, we remounted the old data disk RO and setup an overlayfs for it into a new directory on the new server (with the help of NFS). Then, in the background we copied the raw block device to it's new home. After that was completed, we deleted tombstones present in overlayfs upperdir from the volume and copied every other file over with rsync.Total downtime was less than an hour for the initial setup, and about 1h30m for the resync and finalization.
(DIR) Post #AjIC2I5tbzjSqgdtyq by dean@uwu.social
2024-06-25T17:06:59Z
0 likes, 0 repeats
@admin idk what to set for the profile logo and banner so if anyone has any funny images or ideas
(DIR) Post #B083xCp2J29qStWcfA by dean@uwu.social
2025-11-11T07:34:57Z
0 likes, 0 repeats
@kura I'll let aurieh know
(DIR) Post #B085Mrm7HUCifixXBA by dean@uwu.social
2025-11-11T07:50:49Z
0 likes, 0 repeats
@kura it's fixed, thanks!
(DIR) Post #B0DdkCiBl5bmBUWIe8 by dean@uwu.social
2025-11-14T00:09:33Z
1 likes, 0 repeats
these stupid russian telegram spam accounts are so fucking annoying
(DIR) Post #B22vWYltZcMZQMB3h2 by dean@uwu.social
2026-01-07T15:51:09Z
0 likes, 0 repeats
@kura told aurieh and he says it's fixed, thanks!
(DIR) Post #B2G6TkgWRrLG3S0UVc by dean@uwu.social
2026-01-14T00:25:09Z
0 likes, 0 repeats
@kura it's up for me