Post AHSgiXvT5IJSFjBBFQ by moth_ball@shitposter.club
(DIR) More posts by moth_ball@shitposter.club
(DIR) Post #AGN4gDmKGo3KsLdAOm by p@freespeechextremist.com
2022-02-11T15:27:12.284212Z
19 likes, 15 repeats
:hacker_f::hacker_s::hacker_e:I did an absurd thing to the local DNS server after it came back from the dead, this appears to be the cause of the slowness and some errors. That's fixed as of a few hours ago (sorry!), but I think *what* was broken is kind of interesting. (I wanted to describe the mechanics, but the short "what can I use" version: if you're running Pleroma or similar, run dnsmasq locally, make the cache size 10-20k, tack "nameserver 127.0.0.1" to the front of your /etc/resolv.conf, ahead of the other nameservers.)Fedi's pretty demanding on DNS! There's an easy fix for this, though, and it'll probably be useful to anyone that hasn't tuned dnsmaq and is running Pleroma (probably applies to other stuff like Misskey, everyone's delivering posts), dnsmasq will typically be a big benefit.This can actually have a big effect on performance: Pleroma holds connections to the DB while resolving, allocated from a fixed-size pool, so you can actually free up contention for the connection pool if you have it cached in memory. (This means if you have a typo in your config and connections to the DNS server time out, there will be excessive contention for the DB pool and regular reqs from people using the site will start timing out, which is what happened to FSE.) Every time a user does something that federates, it'll get delivered to that user's followers, meaning those servers will get contacted, meaning they'll Browsers typically cache DNS but this is not what you want for server software, so Pleroma doesn't (as far as I know). If you're running on a VM, they'll probably have DHCP'd you a DNS server, usually you'll be sharing it with several other servers, the cache might be big enough to handle your Balormo and the requests from other VMs on the same machine, but maybe not. In any case, it's not a lot of memory¹ and it will save a lot of time to have it cached locally.I think most of the default configs that ship with it just set it as a caching DNS resolver, which is what you usually want it for. The default cache size is 150, though. FSE's peer list, for example, has 10,246 domains in it² (on average, instances report 2,646), and according to the fedilist crawler, there are 7,544 active servers³. There are also 9,630 that don't resolve, which can be very helpful to cache. I've got it set to 15k, which is not based on anything besides eyeballing it. I don't expect it to fill up: the peer list is all the servers FSE has ever heard of. If you're running a media cache, you'll also be resolving CDN domains⁴, and there's another bump if you enable those "Twitter cards" (which FSE also does not).libc will usually use the first nameserver, dnsmasq is smart enough to not ask itself, so typically you'll just put "nameserver 127.0.0.1" ahead of the other nameservers in /etc/resolv.conf. (If you're using DHCP, you'll want to add it to /etc/resolv.conf.head so that it persists past your DHCP lease.) Once it's up and running, you can even tell it to log all DNS queries (which will flood your log files and slow the resolver down, but can be interesting to look at once in a while).¹ dnsmasq is using 4.3MB of RAM on FSE with a cache size of 15k. Pleroma itself is using about 413MB, so a very roomy DNS cache is still a drop in the bucket.² curl -s https://freespeechextremist.com/api/v1/instance/peers | tr -dc '[,' | wc -c³ Periodic fediverse vanity metrics: 31,018 servers, 7,544 active, 2,438 with open registrations. 4,786,766 accounts, 532,779,001 posts. Of the active servers, 3,063 are Mastodon, 1,079 are PeerTube, 1,024 are Pleroma, 391 WriteFreely, 321 WordPress, 300 Misskey, 293 Friendica, 222 Pixelfed, 104 Mobilizon.⁴ This is less rare than it sounds: masto.host, for example, puts all the user content on CDNs, so a lot of small fedi instances still use a CDN, though all the masto.host instances use the same domain for the CDN, I believe.
(DIR) Post #AGN5Ew2MDxYbzFgM9w by GNUxeava@mk.absturztau.be
2022-02-11T15:33:26.904Z
2 likes, 0 repeats
@p@freespeechextremist.com 4 million accounts!? I wonder how many of them are regularly active and are not bots
(DIR) Post #AGN5PtH2pzkubQmAoy by pomstan@xn--p1abe3d.xn--80asehdb
2022-02-11T15:35:26.971226Z
2 likes, 0 repeats
@p works out of the box with systemd-resolved, albeit having a smaller cache size
(DIR) Post #AGN76cFS79VcCky5IG by moth_ball@shitposter.club
2022-02-11T15:54:22.775459Z
3 likes, 1 repeats
@p Didn't get almost any of it but thanks Mr. Ancap very cool
(DIR) Post #AGN9049CBhLCc62uXI by p@freespeechextremist.com
2022-02-11T16:15:36.682882Z
2 likes, 0 repeats
@GNUxeava You can kinda extrapolate active users from post count, especially the rate of change in post count.
(DIR) Post #AGN9ILjtWIuCO0viXQ by GNUxeava@mk.absturztau.be
2022-02-11T16:18:53.199Z
2 likes, 0 repeats
@p@freespeechextremist.com does it take autistic users into account like me who burst post like a few hundred posts in a day and make 15-30 posts in the subsequent days?
(DIR) Post #AGN9OePB29eu0zFC1w by p@freespeechextremist.com
2022-02-11T16:20:03.261236Z
4 likes, 1 repeats
@pomstan Of course: if it's slow, the NSA/CIA (through IBM/Oracle, both of whom ship RedHat) wouldn't have a system worth rooting.
(DIR) Post #AGN9aq7xHD5JqV0796 by pomstan@xn--p1abe3d.xn--80asehdb
2022-02-11T16:22:13.913069Z
2 likes, 0 repeats
@p CIA can just poke your system through PSP or BMC, they don’t need userspace root
(DIR) Post #AGN9rhzGnJeVWSywpU by p@freespeechextremist.com
2022-02-11T16:25:18.348968Z
5 likes, 1 repeats
@moth_ball :ancapshades:Happy to elaborate. Looking at it, maybe it was slightly unclear: the outbound federation workers, to pick up a job, pick up a connection from the fixed-size connection pool. However long it takes to resolve a domain is tacked on to the length of time that connection stays reserved. If the DNS request is timing out, and that timeout takes longer than the maximum time a connection can be checked out of the pool, then it's doomed to repeat itself until it gives up. Worse, it will exhaust the pool: a bunch of idle workers waiting for a DNS server that will never answer are eating all the DB connections.
(DIR) Post #AGN9vV2wqjEAXvVhgm by teknomunk@poa.st
2022-02-11T16:25:59.194393Z
3 likes, 0 repeats
There is no reason for spies to not want "all the above" spying options.
(DIR) Post #AGNAHwuZvddPL4Iftw by p@freespeechextremist.com
2022-02-11T16:30:02.904376Z
1 likes, 0 repeats
@GNUxeava It's just a total of the number of posts from all of the servers that report that; the data's not really tied to individual users, it's fedi-wide.
(DIR) Post #AGNAOWAxjJZYRk60Rc by p@freespeechextremist.com
2022-02-11T16:31:14.119428Z
1 likes, 1 repeats
@pomstan One wonders why they have used and produced so many userspace rootkits if they don't need those.
(DIR) Post #AGNAavOhmYUY93RL84 by p@freespeechextremist.com
2022-02-11T16:33:28.543254Z
2 likes, 0 repeats
@teknomunk @pomstan Indeed, and plenty of reasons to want it. For one thing, not all packet filters filter the same packets.
(DIR) Post #AGNAkUxyuI3l4BVNwW by p@freespeechextremist.com
2022-02-11T16:35:12.477551Z
1 likes, 0 repeats
@moth_ball I should probably stress that this isn't a Pleroma bug: it would be kind of absurd to make the rest of the system try to account for DNS failure resolution and it's usually fine to assume that DNS resolves quickly for something like this.
(DIR) Post #AGNC8AWgeeWYZnaatE by moth_ball@shitposter.club
2022-02-11T16:50:40.944756Z
4 likes, 1 repeats
@p I always enjoy reading about things I know next to nothing about, even if I end up looking like pic related half the time. Fedi as a whole has opened my eyes a lot regarding what's possible with these silicone slave minds we take for granted.
(DIR) Post #AGNCfajmDu8nQPeIxU by p@freespeechextremist.com
2022-02-11T16:56:43.859073Z
6 likes, 2 repeats
@moth_ball > I always enjoy reading about things I know next to nothing aboutThis is a good quality to have!> Fedi as a whole has opened my eyes a lot regarding what's possibleIf I were making a fediverse hype page, I'd be asking to use that quote.> silicone slave mindsHa, I like that expression. There was, back when Twitter was tolerable, someone soliciting the shortest description of programming that someone could muster, and it was mostly really bad jokes and people trying to be clever about their personal frustrations (as you might expect) but there was a pretty good one: "meat telling sand to think".
(DIR) Post #AGNItL2pxpr8XCVE2a by moth_ball@shitposter.club
2022-02-11T18:06:26.415854Z
4 likes, 0 repeats
@p Well, we did have that whole "scientists teach brain cells in a petri dish to play pong" thing, so we're no exactly far from slave minds at this point. I can't take credit for the idea though; that goes to a Finn-turned-American physicist Hannu Rajaniemi's excellent hard sci-fi book The Quantum Thief. It featured gogols, which were described as slave minds that were harnessed for computation. There's also a very interesting concept of time as a currency, as well as Gevulot, a sort of advanced social media that allows sharing of thoughts and adjustment of one's privacy level (maximum would make others see a person as "privacy fog"). Zuck is not far off with his metaverse, I reckon.
(DIR) Post #AGNJE1tINBYlD8uDnE by bebe@poa.st
2022-02-11T18:10:07.551031Z
2 likes, 1 repeats
Yes here is an interesting video which is somewhat related. Thank you friend.youtube.com/watch?v=9MxHXO8KggE
(DIR) Post #AGNLd6oxNd7NeIxlmS by p@freespeechextremist.com
2022-02-11T18:37:07.665881Z
2 likes, 0 repeats
@moth_ball > Well, we did have that whole "scientists teach brain cells in a petri dish to play pong" thing, so we're no exactly far from slave minds at this point.Oh, have you heard that we've got rat brains grown around electrode that fly fighter jets?> Hannu Rajaniemi's excellent hard sci-fi book The Quantum ThiefI hear that's good!> It featured gogols, which were described as slave minds that were harnessed for computation.Ah, that's probably where the Matrix got it. (That was the original answer to "Why keep people in tubes?" Studio thought it was confusing so the "human battery" concept was shoved into the movie.)
(DIR) Post #AGNLyunGmH6SiaZVr6 by moth_ball@shitposter.club
2022-02-11T18:41:03.756960Z
1 likes, 0 repeats
@p >rat brains flying fighter jetsmfwDo you have a link about the subject? I'd like to share manmade horrors with my friends.All I remember is the one article that said rats loke to drive tiny cars and they'll gladly do it for free. What's next, recreational RCAVs (Rat Controlled Armored Vehicles)?
(DIR) Post #AGNMBLRHXKd6pfDL7o by p@freespeechextremist.com
2022-02-11T18:43:18.893072Z
1 likes, 1 repeats
@moth_ball October 2004: https://www.newscientist.com/article/dn6573-brain-cells-in-a-dish-fly-fighter-plane/
(DIR) Post #AGNYnApOdSjANQIXx2 by throwaway1@freespeechextremist.com
2022-02-11T21:04:36.718624Z
1 likes, 0 repeats
@moth_ball @p rat jannies
(DIR) Post #AHQ52eQYR9RxdLxYNE by coyote@pl.lain.sh
2022-03-15T00:07:29.126977Z
2 likes, 1 repeats
@p as root: # apt install dnsmasq && systemctl enable dnsmasq && echo "cache-size=15000" >> /etc/dnsmasq.conf && systemctl start dnsmasqThen edit /etc/resolv.conf.head (and for immediate effect, /etc/resolv.conf) to have 127.0.0.1 as the FIRST line
(DIR) Post #AHQ55lbYCxvIlLaYoC by coyote@pl.lain.sh
2022-03-15T00:08:03.256235Z
1 likes, 1 repeats
@p with sudo:sudo su -c 'apt install dnsmasq && systemctl enable dnsmasq && echo "cache-size=15000" >> /etc/dnsmasq.conf && systemctl start dnsmasq'
(DIR) Post #AHQ5rYo55rVcyFEuCu by neo@pl.comfysnug.space
2022-03-15T00:16:40.894736Z
2 likes, 0 repeats
@coyote @p do i need to do anything to maintain this or is this fine 9ever
(DIR) Post #AHQ62SWkmTJegGKc3E by coyote@pl.lain.sh
2022-03-15T00:18:39.351035Z
1 likes, 0 repeats
@neo @p it's fiiine, it works with your existing configuration (uses existing dns servers and just caches their responses)If you'd like to configure dns SOME OTHER WAY than /etc/resolv.conf or /etc/resolv.conf.head, then just make sure 127.0.0.1 is the first dns server in the list
(DIR) Post #AHQ689pPDUyyQKlzdI by coyote@pl.lain.sh
2022-03-15T00:19:41.443330Z
1 likes, 0 repeats
@neo @p dnsmasq warns 15000 might be high, but it's probably worth the tradeoff, on my machine I'm doing 10k so it doesn't warn me :) 15k should be no problem though, as pee said
(DIR) Post #AHQ69eOL4HafVJpBr6 by neo@pl.comfysnug.space
2022-03-15T00:19:56.977997Z
2 likes, 0 repeats
@coyote @p Nah, I just use default DNS resolution on this box. thanks for the quick answer
(DIR) Post #AHQ6Cw4XmuyBD08GK8 by coyote@pl.lain.sh
2022-03-15T00:20:33.265916Z
2 likes, 0 repeats
@neo @p It's already helping me a bit
(DIR) Post #AHQ6PfU0mH9RWT7F3o by coyote@pl.lain.sh
2022-03-15T00:22:51.000123Z
0 likes, 0 repeats
@neo also I had some issues with UFW automatically enabling rate limiting to a ridiculous degree that broke federation, I switched to IPtables
(DIR) Post #AHQ6R1QifIQfsaDnai by EdBoatConnoisseur@poa.st
2022-03-15T00:23:05.392915Z
2 likes, 0 repeats
@neo @coyote @p DNS conf at router > DNS conf per device.That is the ultimate redpill on DNS.
(DIR) Post #AHQ6VfniVbwwFJCEQC by neo@pl.comfysnug.space
2022-03-15T00:23:56.140226Z
1 likes, 0 repeats
@EdBoatConnoisseur @p @coyote I'm renting a VPS from Vultr, sadly that doesn't really come with router controls, but I can set firewall rules in their web manager and yeet the entire UFW
(DIR) Post #AHQ6fdjr2044kI8sCW by coyote@pl.lain.sh
2022-03-15T00:25:44.201422Z
1 likes, 0 repeats
@neo @p @EdBoatConnoisseur you don't use SSH?
(DIR) Post #AHQ6lvBH5u8SZyxlRo by coyote@pl.lain.sh
2022-03-15T00:26:52.622440Z
1 likes, 0 repeats
@EdBoatConnoisseur @p @neo I do both :)
(DIR) Post #AHQ6pmGWBRL65SFpIm by neo@pl.comfysnug.space
2022-03-15T00:27:34.139990Z
2 likes, 0 repeats
@coyote @p @EdBoatConnoisseur I do everything except firewall stuff via SSH. I just don't have the patience to wrangle UFW or IPTables
(DIR) Post #AHQ6x5rNjIzWsxzA9Y by EdBoatConnoisseur@poa.st
2022-03-15T00:28:53.040651Z
2 likes, 1 repeats
@neo @p @coyote I have seen some autistic VPS setups, one of them bein to set up a VPS as a VPN, pit all the good stuff on it, DNS, pihole deployment and so on, then all traffic from the other VPSs routed through that one…
(DIR) Post #AHQ756ng7ML07P4isi by neo@pl.comfysnug.space
2022-03-15T00:30:20.382589Z
2 likes, 0 repeats
@EdBoatConnoisseur @p @coyote I mean, that way, you only have to do the firewall config once
(DIR) Post #AHQ7Fdpi9oxtSo34D2 by EdBoatConnoisseur@poa.st
2022-03-15T00:32:12.900282Z
2 likes, 0 repeats
@neo @p @coyote Yeh, no one likes to mess with firewall and iptables, so it is autistic and depending on how many services u r running, you will end up paying more for bandwith, but the tradeoff is worth it to some.
(DIR) Post #AHR20zqbikIsmecoMK by p@freespeechextremist.com
2022-03-15T11:08:17.453389Z
2 likes, 0 repeats
@neo @coyote It Just Works. :smb64_t::smb64_m:
(DIR) Post #AHR28poGvNCxy9GoC0 by p@freespeechextremist.com
2022-03-15T11:09:42.447764Z
2 likes, 0 repeats
@coyote @neo Yeah, it's designed for little home networks. Because of the TTL it almost never fills up but it's really worth it.
(DIR) Post #AHR31w9J3ZlFYbNqEq by p@freespeechextremist.com
2022-03-15T11:19:39.951503Z
2 likes, 0 repeats
@EdBoatConnoisseur @neo @coyote Caching at the edges is usually worth it if you can afford it, but fedi servers interact with a much larger number of external hosts than most interactive applications, and if you're on a VM, you don't usually own the router: you're sharing any router-level DNS cache with a lot of completely unrelated machines that have completely unrelated workloads. 4MB RAM is worth saving a round-trip to the router (which will often miss and then it's a round-trip upstream) on a blocking call to the DB that most servers are making repeatedly, checking out a DB connection from the pool is the web-app equivalent of an iteration of a tight inner loop: you save time wherever you possibly can.
(DIR) Post #AHR3L3HmkLeLC3SqTQ by p@freespeechextremist.com
2022-03-15T11:23:07.227716Z
2 likes, 0 repeats
@EdBoatConnoisseur @neo @coyote Just trimming noise in the log files is worth learning enough iptables to "-j DROP" things as needed.UFW, though, I've never seen behave predictably or reliably.
(DIR) Post #AHS3kVOBagiUkfYTke by methyltheobromine@netzsphaere.xyz
2022-03-15T23:02:25.255043Z
1 likes, 0 repeats
@p amazing to know, thanks
(DIR) Post #AHS4CzCYLIjnJzh7nk by Hyolobrika@mstdn.io
2022-03-15T23:07:33Z
1 likes, 0 repeats
@moth_ball @p>Hannu Rajaniemi>hard sci-fiIdk if it's just because I wasn't knowledgeable enough to get it but I assumed it was soft sci-fi because the science bits felt made up to me
(DIR) Post #AHSgiXvT5IJSFjBBFQ by moth_ball@shitposter.club
2022-03-16T06:19:04.048964Z
1 likes, 0 repeats
@Hyolobrika @p A lot of it was based on what was then cutting-edge particle and quantum physics knowledge, and at the very least the author himself is a scientist in the field so there wouldn't be much of a reason to make shit up.In any case, the scifi crowd hasn't really challenged the "hard scifi" label that the book gave itself, so I'm content with that.
(DIR) Post #AI3ztfJVTa6vIWkmNU by twizzay@thisis.mylegendary.quest
2022-04-03T06:18:16.919793Z
2 likes, 0 repeats
@coyote @p Thanks for the tips guys. Just applied to thisis.mylegendary.quest!
(DIR) Post #AI4ITV8kZoSJLSxZMO by p@freespeechextremist.com
2022-04-03T09:46:27.834265Z
2 likes, 0 repeats
@twizzay @coyote :bigbosssalute: Glad to be of service.
(DIR) Post #AQstCKeyZK7YDJJyGu by p@freespeechextremist.com
2022-12-23T02:30:19.337734Z
3 likes, 2 repeats
> (This means if you have a typo in your config and connections to the DNS server time out, there will be excessive contention for the DB pool and regular reqs from people using the site will start timing out, which is what happened to FSE.)People are still not listening to me about running a local DNS cache. Everyone that is having trouble with timeouts getting out of the connection pool, please see the above post.> ³ Periodic fediverse vanity metrics: 31,018 servers, 7,544 active, 2,438 with open registrations. 4,786,766 accounts, 532,779,001 posts. Of the active servers, 3,063 are Mastodon, 1,079 are PeerTube, 1,024 are Pleroma, 391 WriteFreely, 321 WordPress, 300 Misskey, 293 Friendica, 222 Pixelfed, 104 Mobilizon.These numbers have gone up significantly (active servers have gone from 7,544 to 20,353, fedi's population is now 8,970,159, and there are 766,990,384 posts), but now there is http://demo.fedilist.com/ and you can get all of those numbers yourself instead of me typing them.
(DIR) Post #AQstIMDbgeS6mjOOXI by neo@pl.comfysnug.space
2022-12-23T02:31:23.506822Z
1 likes, 0 repeats
@p I may need to up my dnsmasq's cache size to 30k~ then
(DIR) Post #AQstwrRwzMuzEMPo0W by feld@bikeshed.party
2022-12-23T02:38:23.883777Z
1 likes, 1 repeats
@p you can fine tune DNS caching in the beam itself too...https://www.erlang.org/doc/apps/erts/inet_cfg.html
(DIR) Post #AQt9HL1cEl1m3pJsUi by p@freespeechextremist.com
2022-12-23T05:30:30.440400Z
0 likes, 0 repeats
@neo I set it to 65535 some time ago.
(DIR) Post #AQt9l5WYQxRqflycjI by p@freespeechextremist.com
2022-12-23T05:35:53.023978Z
0 likes, 0 repeats
@feld If Pleroma uses edns and edns is pretty good at handling TTLs appropriately (it has become fashionable to set your TTL to 1, apparently), then it might be good to actually have it resize the cache based on the number of instances it is likely to talk to, which would stop requiring me to eyeball the instances table and adjust dnsmasq accordingly.(I would file an issue on the Gitlab instance but every time I log in, I get 5xx errors and cannot even view PRs or bug reports.)
(DIR) Post #AQwE6aQdi0tRtTb3Sa by feld@bikeshed.party
2022-12-24T17:08:20.601126Z
1 likes, 0 repeats
@p do you still get those 5xx errors? I fixed that a while ago, it was a Gitlab anti-abuse feature for "too many requests from this IP" which somehow affected me too
(DIR) Post #AQwVLNEQktdAWqWTom by p@freespeechextremist.com
2022-12-24T20:21:52.839624Z
0 likes, 0 repeats
@feld awwwwwsnapz It works!That is awesome.