Post AnfcNm4cNwOagOYkCm by Chita@poa.st
 (DIR) More posts by Chita@poa.st
 (DIR) Post #AnfWNYeTgeXJKbqW4O by Chita@poa.st
       2024-11-03T14:41:42.464083Z
       
       0 likes, 1 repeats
       
       Hey @graf quick question. I tend to use your Nitter site to see long threads on Twitter without logging in. (It’s better than that Thread Reader thing). But I noticed if I try to make an archive of that nitter Poast page on Archive Today, it seems to give me an error page about 50% of the time. Sometimes if I add “#m” it works and properly archives. Sometimes if “#m” was in the original failed url, and I take it out, it works. And sometimes it won’t archive either way. Any idea why archive today has problems pulling a page from that Nitter instance? It’s so weird.
       
 (DIR) Post #AnfWW3QX9LW55yNK08 by graf@poa.st
       2024-11-03T14:43:14.699286Z
       
       1 likes, 1 repeats
       
       @Chita because we are filtering automated requests and if it's not 100% of the time that means it's not working as intended and I need to fix it.
       
 (DIR) Post #AnfXAGCv3GvIJgB9Oa by Chita@poa.st
       2024-11-03T14:50:30.647140Z
       
       0 likes, 1 repeats
       
       @graf ah ok. thank you. I noticed some times it archives a page that says it’s rate limited. And sometimes it archives a 404 page. Not sure if any of that info helps lol. (It probably doesn’t). Do you deal with a large amount of automated requests? (I’m assuming yes since it’s one of the few Nitter instances that work still). Would having another good instance go up be helpful? I have been debating running one but unsure on how much it would cost.
       
 (DIR) Post #AnfYrQXEd5FAklpRGS by graf@poa.st
       2024-11-03T15:09:31.014652Z
       
       3 likes, 4 repeats
       
       @Chita our nitter instance uses 350 accounts older than 2017, most of which being 2013. you can't run a public instance without hundreds of accounts. people constantly trying to scrape it which rate limits the accounts. so you need time to monitor it. you need to create an anti scraping, anti bot solution. for the traffic we get, you need to run multiple twitter instances and load balance them whether on separate hardware or on one really powerful machine. tldr the nitter is one of the most expensive things I've ever set up both in terms of time -- about six working months of my life of time to be exact -- and money to procure not only the hardware but the massive amount of accounts you will need once the public finds out there is another public instance. would I recommend it? no. but if archive.is/today/whatever is able to still archive some pages than what I've got set up for security right now isn't good enough. so I will work on that when I get home.
       
 (DIR) Post #AnfcNm4cNwOagOYkCm by Chita@poa.st
       2024-11-03T15:48:58.700917Z
       
       0 likes, 1 repeats
       
       @graf wow. I had no idea how bad/hard/challenging it would be to run an instance. I figured it wouldn’t be a plug and play + set it and forget it type of thing, but I didn’t think it would be this involved or cost so much. you do a lot for all of us retards and don’t think it goes unnoticed. If this was something that could be set up and then just left to operate on its own I’d gladly make one to lighten the load. But there is now way I could spend a lot of time monitoring and making adjustments and fixes.