[HN Gopher] Does Lemmy benefit from Rust? Is code execution spee...
___________________________________________________________________
Does Lemmy benefit from Rust? Is code execution speed the
bottleneck?
Author : dragontamer
Score : 23 points
Date : 2023-06-18 18:48 UTC (4 hours ago)
(HTM) web link (programming.dev)
(TXT) w3m dump (programming.dev)
| thefilmore wrote:
| Is ActivityPub not scalable? I keep seeing criticisms of it on
| this front.
| dingusdew wrote:
| [dead]
| detaro wrote:
| A certain degree of overhead just has to be there, a
| distributed system needs to exchange information between nodes
| after all. And both the protocol and many implementations leave
| room for optimization at least. I think tackling that is one of
| the long-term challenges for the ecosystem, but I wouldn't say
| it is categorically "not scalable".
|
| EDIT: As an example, right now in Mastodon federation traffic
| is handled by sidekiq queues running Ruby. Someone makes a post
| -> tasks to send that information to all their followers
| instances get queued, which then send individual HTTP requests
| to all those instances. This to me feels like something that
| could be moved into some smaller and more-efficiently-written
| component that batches requests, makes sure to reuse
| connections, ... at least for paths that see a lot of traffic.
| notacoward wrote:
| The way that instances connect to each other is O(n^2), which
| is not to say it's _exactly_ n^2 but the curve is of that
| shape. This isn 't strictly a problem with ActivityPub as with
| how it's used, especially in Mastodon. It's why you keep
| hearing about super-long queues chewing up RAM, and causing
| long delays - hours or even days - before content appears. It's
| why Mastodon instances tend to be clusters of larger machines
| at a scale that you would think can be served by a single
| smaller machine.
|
| The usual solution is connection aggregation via a proxy layer.
| Relays (in fediverse-speak) do exist, but seem very lightly
| used and do little or no caching. One person suggested that
| it's uncacheable because ActivityPub relies heavily on POST,
| which for that reason alone seems like a mistake. I wrote about
| this a while ago and probably got some stuff wrong, but it
| might be interesting nonetheless. At least it drew some
| interesting comments at the time.
|
| https://gist.github.com/jdarcy/60107fe4e653819138396257df302...
| CharlieDigital wrote:
| It's more like that the bottleneck is any of the following:
| 1. Infrastructure 2. Architecture (not sufficiently
| decoupled, insufficient cache design, etc.) 3.
| Database/storage limitations ... ... 99.
| Rust
| dragontamer wrote:
| Lemmy has become one of the major "lifeboats" during the
| #RedditBlackout even this past week. This has lead to interesting
| performance issues and questions... as well as human-level
| scaling issues.
|
| On the technical side, this https://programming.dev thread
| (itself, a Lemmy instance) asks how Rust may have affected Lemmy.
| The ongoing discussion covers a wide variety of topics.
|
| The #1 post right now is frome snowe@programming.dev, the owner
| of this instance. He took disk-measurements of the server and
| published them, demonstrating that Lemmy's current bottleneck
| seems to be RAM (proven by the change in Swapping behavior /
| Disk-io before-and-after he upgraded RAM).
|
| --------
|
| I do think you can see the benefits of federation in this manner.
| Users from many instances: https://kbin.social,
| https://lemmy.world, and https://beehaw.org... and more (all
| different Lemmy instances) have joined in the discussion and have
| made a high-quality thread.
|
| ---------
|
| Note: I am bullish on federation / Fediverse after playing around
| with it this past week. I'm less bullish on Lemmy itself, which
| has proven to be a buggy, laggy mess. Still, there's good
| discussion happening on Lemmy now (albeit slower due to fewer
| users). Communities are springing up (and will be dying), and I'm
| sure instances will spin up and die too. But the idea of
| federation works, especially for bringing together the
| discussion.
|
| I think work (and play) needs to happen on Lemmy, kbin, Mastodon,
| and other Fediverse instances to figure out what is the best
| software, and best experience for users. Its not quite ready for
| prime-time, but the core feature set / Minimum-viable-product
| demo here is extremely exciting to me.
| dybber wrote:
| What happens with the content if posted on an instance that
| shuts down? Will it be kept alive by other instances?
| grumbel wrote:
| It might still remain in the cache of another instance, but
| there is really no permanence build into the Fediverse. If an
| instance goes down, its content and user accounts go down
| too. There is no quick&easy way to migrate content to another
| host. There are no crypt-ids (e.g. PGP keys), everything is
| refered to as user@host, even message-ids aren't even
| globally unique, e.g. these two refer to the same post:
|
| * https://beehaw.org/post/598023
|
| * https://feddit.de/post/881008
|
| Fediverse still has a lot of work to, right now it feels more
| like a quick hack than something that could significantly
| improve the Internet in the long run. If an instance goes
| malicious, you have all the same problems as you'd have with
| an unfederated server, after all federation is completely
| optional in the Fediverse.
| dragontamer wrote:
| This has already happened in practice, as https://beehaw.org
| and https://lemmy.world defederated due to some (hopefully
| temporary) issues.
|
| I don't wanna go into the drama / discussion here but lets
| just call it some nerd drama + lack of moderation tools that
| can solve the issue. (Beehaw.org is very confident that if
| Mastodon-like tools existed, they'd be able to refederate and
| get the things they want. The issue is being blamed right now
| on Lemmy's relatively new codebase that's missing some
| features that Mastodon users are familiar with)
|
| https://Lemmy.world is continues to host the copies of the
| "old Beehaw.org" material, allowing Lemmy.world users to
| talk. But the "federation" has shut down, you can only talk
| with local users (because Beehaw.org serves as the "True
| copy". So upvotes / downvotes, and federated
| messages/comments are coordinated by the Beehaw.org server)
|
| In practice, this means that all the Beehaw.org communities
| has become kinda-sorta useless on https://Lemmy.world. But,
| the copies of the conversations (up to the point of
| defederation) do exist and remain on your local (erm... more-
| local?) server.
| tialaramex wrote:
| > Lemmy's current bottleneck seems to be RAM
|
| Rust seems like a sound choice in that regard because it's not
| garbage collected, so you're not wasting some RAM overhead just
| to avoid needing to do extra work, and it doesn't have C-style
| struct packing issues. If you don't specify the layout you need
| for a user defined type, Rust will assume you are OK with it
| re-arranging your data structure to be smaller / faster.
|
| If you're willing to put in some work you can often go further
| with this in Rust than in other "systems programming" languages
| too. For example C++ Strings have the "Small String
| optimisation", exactly how this works varies but e.g libc++ (on
| a modern 64-bit server) String is a 24 byte data structure
| which can hold up to 22 bytes of textual data inline without
| yet needing a heap allocation. So if you do a lot of work
| modifying small amounts of text, Rust's built-in String isn't
| as efficient as a libc++ String, but the CompactString type
| from a popular Rust crate is also a 24 byte data structure yet
| it holds a more impressive _24_ bytes of text before needing
| heap allocation.
| cbeach wrote:
| Just wanted to pick up on this minor point:
|
| > current bottleneck seems to be RAM
|
| RAM exhaustion is often a secondary side effect of the true
| underlying bottleneck.
|
| It could be that RAM used for temporarily processing an HTTP
| request, say, would be freed quicker if the database returned
| sooner. Maybe the database is the actual bottleneck. And the
| actual root cause is an indexing / partitioning issue.
| mvdtnz wrote:
| There is no good reason for any ordinary web application to hit
| scaling limits at 1,500 users. The developer vaguely waves his
| hands at memory issues but it's hard to imagine what kind of
| reasonably-designed system could possibly be putting memory
| pressure at that scale.
|
| There's no way any modern language will be your bottleneck at
| that scale.
| TazeTSchnitzel wrote:
| I think it's perfectly ordinary for a new web application to
| hit scaling issues at that number. Accidental O(n^2) stuff,
| missing database indexes, poor use of memory, excessive I/O are
| easy mistakes to make during initial implementation and
| probably will get fixed.
| mvdtnz wrote:
| That's what I mean by "no good reason". This is the scale
| where you start to notice the stupid stuff you let through -
| N+1's, excessive unnecessary string allocations in inner
| loops, debug logging etc.
|
| What I mean is, there's nothing fundamental about their tech
| choices that could explain such low scaling limits. It'll be
| either poor design/architecture, poor coding standards or
| both.
| dragontamer wrote:
| I don't think the Lemmy developers were expecting
| #RedditBlackout to dump 150,000 users onto their federation
| in a week.
|
| They were probably trucking along with various feature
| requests when suddenly performance became an issue this
| past week.
| mvdtnz wrote:
| Where are you finding that number, 150,000? The instance
| in question has one one hundredth that figure and the
| Lemmy devs claim 27,000 accounts grand total.
| dragontamer wrote:
| Lemmy.world is 31k users alone. Beehaw.org is 12.k users
| (and each of those users at Beehaw.org were manually
| curated)
|
| Maybe 150,000 is wrong, I saw it posted somewhere but I'm
| not sure what the methodology is. But between Lemmy.world
| and Beehaw.org we're already at 43k+ users.
|
| 27k is a gross underestimate.
|
| -------------
|
| EDIT:
|
| https://kbin.social/nodeinfo/2.0
|
| https://fedidb.org/software/lemmy
|
| Lemmy-fediverse is claiming 159,000+ users today.
|
| EDIT2: kbin link must have been glitched. kbin.social's
| specific nodeinfo is "only" claiming 35k users, which is
| still substantial.
| dragontamer wrote:
| > Lemmy devs
|
| Programming.dev is _NOT_ the lemmy devs. This is a
| discussion from https://programmers.dev, a totally
| separate group of people.
|
| This is just an administrator who is sharing their Lemmy
| experiences while talking about Rust and such.
| dragontamer wrote:
| Lemmy.ml, the original developer's instance, is well known to
| be a small server.
|
| Lemmy.world is well into 31,500+ users on a single server and
| still growing strong. But growing pains have begun to creep in
| in any case. The server's software clearly needs a few
| optimization passes.
___________________________________________________________________
(page generated 2023-06-18 23:02 UTC)