Post ARwP2qtKEYTOQNcgK0 by stux@mstdn.social
(DIR) More posts by stux@mstdn.social
(DIR) Post #ARwOKFVHsZKXj86wTY by stux@mstdn.social
2023-01-23T16:55:10Z
0 likes, 0 repeats
Cleaning external media older than 90 days from mstdn.social to safe some costs2499/2269485 ETA ~ 17 hours :flan_laugh:
(DIR) Post #ARwOevqu8aMcrQ5J9k by realcaseyrollins@social.freetalklive.com
2023-01-23T16:59:03Z
0 likes, 0 repeats
@stux Is this something that can be automated?I feel like smaller instances especially could benefit from implementing something like this.
(DIR) Post #ARwOfpxvVfFstPnhsO by feld@bikeshed.party
2023-01-23T16:58:49.235222Z
0 likes, 0 repeats
why does it take so long? Is Mastodon not using batch requests to S3?
(DIR) Post #ARwOi2t2RLkRXvZEmG by stux@mstdn.social
2023-01-23T16:59:33Z
0 likes, 0 repeats
@feld It's per 5 and i don't wanna stretch it since all other operations are also still going :blobcatgiggle: Just take it's sweeeeet timee
(DIR) Post #ARwOl2uCpJCSTPXnQ8 by stux@mstdn.social
2023-01-23T17:00:05Z
0 likes, 0 repeats
@realcaseyrollins Yess! From the latest version this can be set from the admin panel :ablobwink: But i like to chose my moments :blobcatgiggle:
(DIR) Post #ARwOpcIRBbjjOW3dyq by stux@mstdn.social
2023-01-23T17:00:40Z
1 likes, 1 repeats
@realcaseyrollins This is tottally new :blobcatgiggle:
(DIR) Post #ARwOwPEm8bxSv4aADo by freemo@qoto.org
2023-01-23T17:02:10Z
0 likes, 0 repeats
@stux One of the features at QOTO is complete media and cache retention from the start of QOTO, some 5 years now... Basically means QOTO has one of the most complete search histories around :)@realcaseyrollins
(DIR) Post #ARwP2qtKEYTOQNcgK0 by stux@mstdn.social
2023-01-23T17:03:13Z
0 likes, 0 repeats
@freemo @realcaseyrollins But what's the total storage cap by now? :blobcatgiggle:
(DIR) Post #ARwP493iAmJd3l66cK by Jain@blob.cat
2023-01-23T17:03:35.905705Z
0 likes, 0 repeats
@stux @realcaseyrollins :blobcateyes: nice, so finally masto wont fill the media space with data from other servers :blobcat3c: i actually would prefer if they would implement how pleroma handles that... by using the caching ability from webservers like nginx or others
(DIR) Post #ARwP5XIvVxaOXqsTGy by freemo@qoto.org
2023-01-23T17:03:50Z
0 likes, 0 repeats
@stux I need to check but its absolutely massive. Our storage costs are as much as the server costs right now.@realcaseyrollins
(DIR) Post #ARwP9pu0JFITNsGpzk by stux@mstdn.social
2023-01-23T17:04:34Z
1 likes, 0 repeats
@Jain @realcaseyrollins Well, it kinda still does :blobcatgiggle: also nginx comes with default 5GB i think but we have 5 times that now lol I do think it's better though, more relayable for external content :blobcatgiggle: but unfort not with media larger than config
(DIR) Post #ARwPCeDAEU0jzgZb16 by stux@mstdn.social
2023-01-23T17:05:06Z
0 likes, 0 repeats
@freemo Woahhh :blobcatgiggle: it's growing freaking fast though.. we are also at 300-400GB database now wtf @realcaseyrollins
(DIR) Post #ARwPRJWPREW8AbZIiO by Jain@blob.cat
2023-01-23T17:07:48.473810Z
0 likes, 0 repeats
@stux @realcaseyrollins how much space did the media require before the cleanup? back when busshi.moe existed, it took around ~ 3TB
(DIR) Post #ARwPVhFp8wrsvvttcu by untitaker@woodland.cafe
2023-01-23T17:08:30Z
0 likes, 0 repeats
@stux @realcaseyrollins don't set content cache... It will delete peoples bookmarks.
(DIR) Post #ARwPYyyViMESlJMwxk by wild1145@mastodonapp.uk
2023-01-23T17:09:06Z
0 likes, 0 repeats
@stux We're reducing ours to 30 days soon and I'm both excited and dreading the time it'll take for that job to run for us 😂
(DIR) Post #ARwPbuDfqquOskr7E8 by freemo@qoto.org
2023-01-23T17:09:40Z
0 likes, 0 repeats
@stux Yea im well into the 10s of terrabytes range by now.@realcaseyrollins
(DIR) Post #ARwQCXDsJdSSH1PqKm by supernovae@universeodon.com
2023-01-23T17:16:16Z
0 likes, 0 repeats
@freemo @stux Are y'all talking about postgresdb being 300+ gig or terrabytes in size or your S3 object storage being that big?My S3 is a few TB and growing, but DB is "only" 50ish gb and i'm curious how much could be cleaned up by culling dead accounts and things like that - do you preserve the internet for sake of preservation, or do you keep things around only if they're still active. :thinkSpin:
(DIR) Post #ARwQHkezmc4ZOfaYwy by realcaseyrollins@social.freetalklive.com
2023-01-23T17:17:16Z
1 likes, 0 repeats
@Jain @stux I do like how #Mastodon hosts caches of other instances' content, it's literally the only reason I use it in some cases over #rebased + #Soapbox
(DIR) Post #ARwRCHDBwbRSZQXLjE by pavel@mastodon.arch-linux.cz
2023-01-23T17:27:20Z
0 likes, 0 repeats
@stux That reminds me good old SQL joke: 38 563 rows affected. Uh - oh!
(DIR) Post #ARwRxmSxYivF79Z1fs by adnan@1210.nl
2023-01-23T17:35:56Z
0 likes, 0 repeats
@stux How has the fediverse settled on 90 days for the cache? I am thinking of making it even shorter - 30 days - for myself.
(DIR) Post #ARwUC0OglBeMHdTcpc by mvrenselaar@mastodon.nl
2023-01-23T18:00:58Z
0 likes, 0 repeats
@stux is that something that you need to start up automatically?
(DIR) Post #ARwiHweR01QhlFZI00 by cwizprod1@mstdn.social
2023-01-23T20:38:58Z
0 likes, 0 repeats
@stux how much space do our posts take up now?
(DIR) Post #ARx5AqoQbiqYZSpS9Q by juliefaithmcmurrayart@wandering.shop
2023-01-24T00:55:22Z
0 likes, 0 repeats
@stux does that mean deleting posts?
(DIR) Post #ARx5InL3PHHY77sSJM by stux@mstdn.social
2023-01-24T00:56:48Z
0 likes, 0 repeats
@juliefaithmcmurrayart No, no, just remove the media that's attachted to external posts :ablobwink: it will automatically be redownloaded when someone looks up that post again