[HN Gopher] When imperfect systems are good: Bluesky's lossy tim...
___________________________________________________________________
When imperfect systems are good: Bluesky's lossy timelines
Author : cyndunlop
Score : 753 points
Date : 2025-02-19 17:48 UTC (1 days ago)
(HTM) web link (jazco.dev)
(TXT) w3m dump (jazco.dev)
| nightpool wrote:
| Note that all of this reflects design decisions on Bluesky's
| closed-source "AppView" server--any federated servers interacting
| with Bluesky would need to construct their own timelines, and do
| not get the benefit of the work described here.
| xrisk wrote:
| What reason does Bluesky give for not opening up their AppView
| code?
|
| Another notable component that is closed source is the
| discovery feed generator, where at least there is _some_
| reason.
| muscomposter wrote:
| what else? profit by means of doing work that benefits first
| and foremost the private proprietors of the closed source
|
| if they gave it away (which used to be unfeasible until the
| digital era) they feel they're loosing their valuable effort
| which they're wont on concentrating, not diluting.
| verdverm wrote:
| The App View frontend is open source:
| https://github.com/bluesky-social/social-app
|
| Much of the backend is open source as well:
| https://github.com/bluesky-social/atproto/tree/main/packages
|
| What is not are the extra services they run to provide a
| better and faster UX. Even if it was open source, it likely
| costs 10s of thousands to run per month (they have moved
| largely to "onprem" hardware instead of the cloud aiui)
| nightpool wrote:
| That's the frontend code, it doesn't include the backend
| API services, which are closed source.
| verdverm wrote:
| Which is what I said in the second sentence
| nightpool wrote:
| AppView is a specific term of art within the Bluesky
| federation architecture:
| https://atproto.com/guides/glossary#app-view, you were
| incorrect in identifying the public frontend repo as the
| AppView.
| verdverm wrote:
| A frontend is (can be) part of an App View. It is quite
| literally the app you view the network through. There can
| also be headless app views and app views which have no
| backend
| half-kh-hacker wrote:
| this is not correct
| half-kh-hacker wrote:
| the backend (the AppView) can be found here:
|
| https://github.com/bluesky-
| social/atproto/tree/main/packages...
|
| there are various supporting services written in Go as
| well
|
| https://github.com/bluesky-social/indigo
| half-kh-hacker wrote:
| that's not the appview, that's the client
| verdverm wrote:
| App View is a bit fuzzy of a term. To me it seems like a
| combination of frontend, backend, custom lexicon, and
| supporting services. There isn't really another place in
| the spec or design where clients or browsers fit in,
| which do in fact provide a view of the network via an
| app.
| verdverm wrote:
| "UI" is part of the definition they give in the glossary
|
| https://atproto.com/guides/glossary#app-view
| dingnuts wrote:
| when I read the spec it seemed like the operator of an
| AppView & Relay would be most in need of compensation for
| their hosting costs due to the amount of demand on those
| components so I believe the spec allows an operator to
| implement their own AppView & monetize it as that operator
| sees fit, so that they can afford to operate the service and
| maybe even make money off of it so that they can make it
| their full time jobs.
| verdverm wrote:
| It seems this way to me as well. ATProto fundamentally
| changes how monetization works in social media by removing
| lockin. It's going to be interesting to see what emerges
| from this design decision.
|
| Another interesting way to view ATProto is that it could be
| a collection of headless features and network browsers that
| leverage those feature providers.
| iameli wrote:
| I asked this and got
|
| > We did a backend rewrite from postgres to scylla and it has
| a bunch of deployment specific stuff, but is functionally
| identical to the open source postgres version. Its not really
| a "v2" in terms of new features, we just made it make use of
| our hardware really well[1]
|
| [1]: https://bsky.app/profile/iame.li/post/3l7e3jfqit22s
| nightpool wrote:
| Thanks, so are both the Postgres and Scylla versions
| maintained in terms of new features?
|
| I wasn't aware that AppView v1 was open source, and the
| most recent info I'm aware of on the topic is
| https://alice.bsky.sh/post/3laega7icmi2q,
| https://github.com/bluesky-social/atproto/discussions/2961
| and https://docs.bsky.app/docs/advanced-guides/federation-
| archit..., and everything I've heard about Bluesky was that
| open source appview is "still coming".
| psionides wrote:
| It's not coming, it never went away... As I understand
| it, the "business layer" with all the logic is above the
| data later, shared by the Postgres and Scylla versions,
| and the data layer just makes queries to the database. I
| think they are using the Postgres version locally for
| development.
| haileyok wrote:
| This is not true. Third party PDSes are fully supported by our
| app view, and our app view generates timelines for all the
| users on those PDSes.
| nightpool wrote:
| What does this have to do with third party app views?
| psionides wrote:
| You didn't specify what kind of federated servers you were
| thinking about
| madeofpalk wrote:
| The statement "any federated servers interacting with
| Bluesky" is ambiguous, because Bluesky's federated model
| means there's many different types of servers, and one
| user's view of what a "federated server" could be vastly
| different from another.
|
| Federated PDS-s (which is probably the closest to what
| people mean when they say they want to federate on bluesky)
| would not need to reconstruct timelines if their users use
| the bsky.app appview.
| nightpool wrote:
| Thanks, that's a fair point that I was overlooking. When
| I say a "federated server", I don't just mean a self-
| hosted PDS, I mean a third party app that potentially has
| its own lexicon and design decisions. Creating a robust
| third-party app that can meaningfully interact with the
| Bluesky network is still a very difficult engineering
| challenge, which I think this article does a good job
| demonstrating--that was the tension I was trying to
| underscore in my comment. Bluesky may be solving those
| engineering challenges for those clients who are
| satisfied with Bluesky's frontend and AppView, but every
| single other app built on top of ATProto will have to
| resolve those same challenges. This is directly
| downstream from Bluesky's "global firehose" topology and
| various design decisions that stem from that.
| pfraze wrote:
| As others have noted, the appview is open source. The dataplane
| has two implementations, one in postgres and another in scylla.
| The scylla dataplane is closed, the postgres one is open.
|
| The interesting next stage for the postgres implementation is
| to create a sync engine for partial syncs of the network, so
| that an appview can run affordably. We ran some benches on the
| current state of the postgres implementation and found we could
| index 300k users on a $100/mo vps. I think with a couple of
| weeks of optimization that could reach 1mm users.
| nightpool wrote:
| This is great to hear--my current understanding of the most
| recent state of the art on the topic is
| https://alice.bsky.sh/post/3laega7icmi2q which mentions that
| the self-hosted appview is not yet open source. So I'm glad
| to hear the situation has changed in the past 3 months.
| psionides wrote:
| It was open source (except the Scylla database layer) from
| the beginning, AFAIK - that blog post just says that they
| haven't set it up yet, because that's the hardest part to
| run
| evbogue wrote:
| My thinking has evolved on this topic significantly as of late.
| My current thinking is we should create a secure gossip network
| on top of the Bluesky API, and forgot about all the DAG-CBOR
| stuff that gets stripped from the Jetstream. Hash the posts on
| the gossip layer and if posts change then diff them. This is
| all prep for when X billionaire buys out Bluesky then we just
| pop some signing key crypto on top of this gossip layer and
| wow! It's distributed!
| pfraze wrote:
| isnt that ssb?
| evbogue wrote:
| reverse-ssb
| dang wrote:
| [stub for offtopicness]
| amazingamazing wrote:
| I don't understand the infatuation with blue sky. The minute
| they need money it'll go the way of the Reddit and twitter.
| xrisk wrote:
| People want the old Twitter, and Bluesky is close to that. It
| also cosplays being decentralized to people who don't look
| too closely.
| dom96 wrote:
| What makes it not decentralised?
| xrisk wrote:
| The fact that you have to be on "the" relay to
| meaningfully participate on the network.
|
| If you instead claim that users can always choose to use
| other 3P relays, then you immediately lose all the nice
| things that Bluesky is able to do well today (search,
| discoverability, a "discover" algorithm). Indeed, you
| fall back to the same old problems that every other
| decentralized social network has.
|
| Bluesky is just a shittier version of Nostr, except that
| the people over at Nostr don't pretend.
| immibis wrote:
| The approximately a million dollars a year that it costs
| to run another copy.
| BizarroLand wrote:
| https://dustycloud.org/blog/how-decentralized-is-bluesky/
| Larrikin wrote:
| If everything good is assumed to eventually become bad, why
| not use things while they are good and then immediately move
| on when it becomes bad?
| treyd wrote:
| Not everything good becomes bad. That premise is wrong.
|
| Bluesky accepted VC money. For a social platform that means
| its death certificate has already been signed.
|
| What you're ignoring with that framing is that we can use
| social media that operates outside the VC startup pipeline
| and doesn't have enshittification baked in from the start.
| sodality2 wrote:
| Your actions' consequences are not limited to benefiting
| from the thing like it would for a product - with social
| media, you improve the networking effect for the soon-to-be
| bad. (Nothing against bluesky, I don't know or think it
| will do so)
| VectorLock wrote:
| People seem to lark on and on about how it has better
| "default moderation" than Mastodon.
| verdverm wrote:
| It's not that it is "better" but that the choice is
| individual, not up to the mastodon server. In Mastodon, you
| trade Elon for some other group of individuals, so what
| happens if they make decision on moderation or content you
| do not agree with?
|
| ATProto is designed around accounts that are independent of
| data host, application, and moderation, all in the name of
| giving users individual control over these things. It's
| like if every Mastodon user ran their own server, but
| without the overhead
| VectorLock wrote:
| >It's like if every Mastodon user ran their own server
|
| No, it's like every Mastodon user used the same server,
| and all the coordination is done by one server that
| nobody can replicate.
| verdverm wrote:
| Every user in ATProto gets their own database that
| amounts to a tar file (technically sqlite with car export
| format)
|
| This is nothing like having a single server for every
| user. Perhaps you are confusing Bluesky (one app) with
| ATProtocol the shared network? There are already
| independent servers and apps operating separate from
| Bluesky
| fc417fc802 wrote:
| Are you suggesting the "big few" can't largely censor a
| given account?
|
| I don't see how ATProto is doing noticeably better than
| the scenario where a large ActivityPub instance blocks
| your external account.
| verdverm wrote:
| Generally, yes. Currently, because Bluesky requires the
| use of their labeler if you use their app, this could
| happen.
|
| Two points of note
|
| 1. You can participate in Bluesky without the Bluesky
| app, so you can remove this requirement by using an
| alternative app
|
| 2. The most blocked account is blocked by around 0.25% of
| the full network (https://clearsky.app/)
|
| This second point does not account for users banned from
| Bluesky by Bluesky for breaking the ToS or PDS abuse.
| fc417fc802 wrote:
| > does not account for users banned from Bluesky by
| Bluesky for breaking the ToS or PDS abuse.
|
| Then you are missing the point. I am asking how much
| censorship power the largest node in the network has.
|
| If being blocked by the largest provider means 95% of
| users can't see me anymore then the situation is
| _strictly worse_ than Mastodon vs ActivityPub-at-large.
| verdverm wrote:
| What did you do to get banned? Most people who make this
| claim are unwilling to share the behavior that led to it.
| Bans are not arbitrary, nor is anyone obligated to host
| or show your speech
|
| ATProto is young compared to the alternatives, give the
| network time to build out an ecosystem and the resilience
| we need.
| fc417fc802 wrote:
| First you made incorrect and misleading statements. Then
| when I point it out you try to rhetorically shift blame
| on to me.
|
| To answer your misleading and irrelevant question, I have
| never been banned by Bluesky. What did you do to get
| banned on Mastodon? Do you understand that "moderation"
| and "banned for violating the ToS" are one and the same?
|
| ATProto is not any better than ActivityPub (ie Mastodon)
| regarding the impact of network effects on moderation.
| The "big few" retain their outsized influence with both
| protocols, at least as far as I am aware. If there is a
| technical misunderstanding about this on my part then do
| please point it out.
|
| Moreover, the _current_ situation is significantly worse
| on ATProto because the vast majority of the network
| currently relies on a single provider.
|
| To be clear, I am not opposed to ATProto. I am only
| opposed to inaccurate technical comparisons.
| verdverm wrote:
| I was using the indeterminate "you" for a hypothetical
| question, one that gets asked often
|
| > Please respond to the strongest plausible
| interpretation of what someone says, not a weaker one
| that's easier to criticize. Assume good faith.
|
| https://news.ycombinator.com/newsguidelines.html
|
| ATProto moderation is technically and fundamentally
| different from ActivityPub:
| https://bsky.social/about/blog/03-12-2024-stackable-
| moderati...
| fc417fc802 wrote:
| The question remains misleading and irrelevant whether it
| was addressed to me or the room at large.
|
| I am well aware that ATProto makes improvements to both
| account migration and end user moderation. Those are of
| course good things in my view. As previously stated I
| have noting against ATProto.
|
| This entire subthread started because I responded to your
| claim that "It's like if every Mastodon user ran their
| own server". That is highly misleading at best, outright
| wrong at worst. It is only equivalent to that scenario if
| every ATProto user manages their own account _as well as
| every single other required service_ , at which point you
| might as well just have run your own Mastodon instance as
| it likely would have been both simpler and less resource
| intensive.
|
| The reality is that this is never expected to happen.
| ATProto is expected to be like Mastodon, with the
| improvements that you can share a single account between
| multiple servers and there are tools to apply additional
| filtering so that you don't have to see (as much of)
| things you don't like. Those are certainly welcome
| improvements but they do nothing to address the outsized
| effects on censorship that large nodes have in federated
| networks.
|
| Purely in theory, you could simultaneously use a single
| account with a full set of politically correct ATProto
| services (such as Bluesky) and a competing set of
| politically incorrect ones operated by someone else.
| However, if history is anything to go off of then this
| will quickly get you banned by either one or both of the
| operators. Merely interacting with those guilty of
| wrongthink has always been an easy way to get kicked off
| of a surprisingly large number of Mastodon servers.
|
| Even supposing that the above _doesn 't_ happen to get
| you banned by either node, the potential still remains
| that it _could_. Whether or not a large node chooses to
| exercise its outsized influence doesn 't change the fact
| that it possesses it. ATProto doesn't fix that.
| immibis wrote:
| You have the opportunity to demonstrate this. I am banned
| from Bluesky. (They didn't tell me why - just a generic
| "you violated community guidelines")
|
| Tell me, concretely, how people can choose to continue
| following me, even though I am banned.
|
| Profile: immibis.bsky.social
| verdverm wrote:
| Create an account you own instead of having someone else
| run it. Maybe you can get your data, maybe you can ask
| Bluesky for a review (there were bugs and scaling issues
| against bot networks that cause false positives)
|
| I'm not seeing that handle resolve in the normal places.
| Do you have the DID? You should use a custom domain so
| that you can control the the reference and lookup.
|
| You can run your own PDS and manage complete account
| lifecycle
| immibis wrote:
| So after you're banned from Bluesky you create another
| account on a different server and hope the admins of your
| original server, which still hosts all the people you
| want to follow, don't block your new account from
| interacting with their server?
|
| You said it was different from Mastodon, but how is this
| different from Mastodon?
| verdverm wrote:
| I have my own domain already attached, I can point it at
| any server and my identity on the network remains the
| same.
|
| When you use a *.bsky.social handle, you have not made
| yourself independent and resilient to arbitrary decision
| by the org that manages that service
| immibis wrote:
| So that's the same as on Mastodon. What's the difference
| again?
| fc417fc802 wrote:
| To be fair, with ActivityPub your account is always tied
| to a given server.
|
| However as you have rightly noticed, the person you are
| responding to here is being disingenuous. Controlling the
| account doesn't fix the fundamental issue caused by
| network effects. An account that can't interact with a
| large portion of the network is no different than
| Mastoson in practice.
|
| The Mastodon equivalent would be running your own single
| user instance which gets defederated by one of the big
| players.
| anamexis wrote:
| Follow the instructions under "Self-hosting PDS" here:
| https://github.com/bluesky-social/pds
| bowsamic wrote:
| My bluesky feed is somehow even more abhorrent than my
| twitter one, except that instead of right wing hate it's
| Facebook memes about "reading banned books"
| rsynnott wrote:
| Bluesky's 'Discover' feed (the default algo feed that you
| get when you create your account) is based on _likes_, not
| follows, so if you never like anything you'll get random
| nonsense.
|
| You can try using other algo feeds from here:
| https://bsky.app/feeds and remove the discover feed, or of
| course you could just use the chronological one.
| bowsamic wrote:
| But when everything I see is awful, what do I like? I
| think that's my classic issue with feed style websites
| rsynnott wrote:
| You follow _people_, and ignore the 'Discover' feed, is
| generally my personal solution to this. YMMV.
| rsynnott wrote:
| Twitter was always... not great (there's a reason it was
| affectionately known as the Hellsite), but it had 16 years of
| being _tolerable_ for most people (the real exodus only
| really started with Musk's changes, though there had been a
| couple of smaller ones previously, mostly over Twitter
| messing with the API).
|
| Frankly, if I get 16 years out of Bluesky before having to
| move onto the next one, I can live with that. Social networks
| _die_; it has always been so. USENET, livejournal, Tumblr,
| twitter... nothing lasts forever.
| Boogie_Man wrote:
| Bluesky is the Conservative Dad Beer of "left" short form
| social media.
|
| I implore everyone to use something better like Mastodon or
| maybe minds
| zoul wrote:
| I would be so much more interested in Bluesky if it were
| technically impossible for a random super rich guy to buy and
| bend it to his whims.
| culi wrote:
| Isn't that the whole point of bs? Empowering users to take
| their data where they want. It's completely open-sourced and
| well-documented. If someone buys bluesky you can move all
| your data to a different service that follows the same
| protocol
| plagiarist wrote:
| Can I move my followers/following graph as well? Moving the
| actual content is barely a consolation prize if you lose
| your entire audience in the process.
| zoul wrote:
| You can move your data, but whoever controls the AppView
| controls what people see "on Bluesky".
| culi wrote:
| Yeah but anyone can make a GreenSky. The source code is
| all there
| zoul wrote:
| But the users are not. The app and the AppView can be
| bought and then you are in control of what people can
| see. Good luck trying to move them to the alternative.
| exabrial wrote:
| I honestly am annoyed to use websites and services like this.
| Annoys the crap out of me and everyone else, but since it's
| petty much forced down their throats, the "eventually" is
| "eventually everyone stops complaining".
| einpoklum wrote:
| Centrally-controlled social media platforms are not a good
| thing, period. Neither Twitter/X, nor BlueSky. Let's not fete
| them.
| nasso_dev wrote:
| Interesting! I wonder what value they chose for the
| `reasonable_limit`.
| Retr0id wrote:
| ought to be possible to reverse-engineer it by following a
| large number of active accounts and seeing what percentage of
| their posts actually hit your feed
| nasso_dev wrote:
| It's 4k:
| https://bsky.app/profile/jaz.bsky.social/post/3likncbqutk2y
| bitmasher9 wrote:
| It's really impressive how well Bluesky is performing. It really
| feels like a throwback to older social media platforms with its
| simplicity and lack of dark-patterns. I'm concerned that all the
| great work on the platform, protocol, etc won't shine in the long
| term as they eventually need to find a revenue source.
| autobodie wrote:
| Absolutely. The profit motive is the root of most evil. It is a
| shame that so many are trained to believe it is the only motive
| available.
| gkoberger wrote:
| I completely agree with this... but without profit, people
| can't get paid, and they'll stop building. I do hate this
| incredibly need for growth, of course, but financial growth
| is necessary to pay people and give them raises and allow
| them to have upward mobility at the company.
|
| I hope Bluesky is able to find a model that works for them
| AND for consumers. (I do know it's an open protocol, so it'll
| live on without Bluesky itself! However, as this post shows,
| it's a lot of work to build on the prototype... so if not
| them, who? And if someone else, how will they become
| sustainable?)
| jandrese wrote:
| At the same time I feel like a lot of companies grow much
| larger than they need to be simply because of bigger is
| better mentality. How many of Uber's 30,000ish employees
| are involved with making sure the app and backend database
| are working properly? Are they really doing 600 times more
| work than Craigslist at connecting sellers with buyers?
| bitmasher9 wrote:
| You cannot compare uber to Craigslist.
|
| Uber takes on so much more responsibility of the
| transaction. Setting price, handling disputes, real time
| coordination, etc.
| gkoberger wrote:
| I'm an Uber hater, but... yes.
|
| Like, sure, they don't need every single one of those
| 30,000... but they have to have ground teams in every
| city in the world. Connections with every airport.
| Connections with almost every restaurant in the world.
| Customer support and safety (okay I know they don't nail
| this, but still). They need to pay out drivers in each
| country. The app needs to work in hundreds of countries,
| all with different laws, currencies, languages and more.
| Some places let you pick up anywhere, others require
| specific locations. And that's not even including
| marketing, partnerships, HR, finance, etc.
|
| I don't think the employees are the problem with Uber,
| it's the shareholders. They need to make X back, so that
| delta is where drivers get squeezed.
| redcobra762 wrote:
| Aren't you actually arguing in _favor_ of profit-driven
| behavior? You 're not disagreeing with profit as a
| motivator, you're questioning if the 30,000 employees is
| the maximal way to achieve profit.
| tdb7893 wrote:
| It's semantics but I like to separate money from profits.
| You need money to pay people and to survive but you don't
| need to be raking in endlessly growing piles of it. This is
| something that was really demoralizing about working for a
| big company, they could be making like 50000000000 a year
| in just profits but still be ruthless in getting more. Like
| I just want to make a product I'm proud of and I'm happy
| living a simple life, I am happier now making less money
| but not feeling like I'm endlessly milking customers.
| JustExAWS wrote:
| Once you take on investors, that's not an option. VCs
| expect rapid growth and an exit - statistically through
| acquisition, but occasionally an IPO.
|
| Once you go public, then you have investor pressure and
| can be subject to activist investors unless the founder
| has controlling interests like in the case of Meta and
| Google.
| raxxorraxor wrote:
| Basically why I don't like to work for any public company
| anymore. You are there to be exploited by moneyed
| interests that most often don't even have any positive
| effect on the products itself.
| cyberax wrote:
| On the other hand, running something like BlueSky is not
| terribly expensive. A foundation with a reasonable
| endowment can do that indefinitely.
|
| Initially, it can be funded by selling tools that do
| analytics or by donations (like Wikipedia).
| bbor wrote:
| Yes! If the venture capitalists that are already involved
| stick to their stated principles and don't demand eternal
| growth (which... fingers crossed?), I think bsky has an
| extremely feasible, promising future.
|
| They've intentionally kept a low footprint to keep
| expenses down, and while income via donation is out of
| the picture (unless AT Proto grows into a full ecosystem,
| I suppose?), cosmetics are a tried-and-true model for
| supporting something that most users use for free, but
| that some power users spend all day on and want shiny
| stuff for. They'll probably end up exploring Discord-
| esque paywalled features for power users as well, which
| isn't necessarily _ideal_ but is leagues better than
| getting on the currently-dying vicious cycle of Display
| Ads, IMO.
| immibis wrote:
| This is hardly the first instance of "if only venture
| capitalists light their money on fire, we can have nice
| things."
|
| Spoiler: Venture capitalists don't light their money on
| fire, and we can't have nice things.
| jarjoura wrote:
| If Bluesky ever gets close to becoming a serious threat
| to Meta's walled garden, the effort to fight back against
| them will take a lot of capital. Just the legal battles
| alone will cost a fortune.
|
| Wikipedia isn't a threat to anyone, they just have to
| generate enough capital to exist.
| impossiblefork wrote:
| Yes, but there is a path, and it's simplicity.
|
| Lichess, is it bad? It basically solves the whole problem.
| If well-designed distributed social media site could be
| something like that. Donations are enough to support one
| guy at least.
| bbor wrote:
| I totally get/relate to your perspective, but to be the
| annoying leftie in your ear:
|
| A) Sustainable revenue is a requirement for any company,
| yes, but the unlimited (above-inflation) growth demanded by
| most large corporations is absolutely not. Lots and lots of
| companies operate for a long time without expecting massive
| growth, raises n' all. MBAs pejoratively call such
| companies "lifestyle businesses"--as in "just pays for
| people to live"--but I'd call them "normal, healthy
| companies".
|
| B) More fundamentally: the idea that a social media network
| can only be built by a single corporation owned by
| investors is an omnipresent, yet extremely toxic,
| assumption. Mastodon represents another extreme end of the
| capital<->labor spectrum where anyone can contribute to the
| network at any time with their own instance, but I think
| Bluesky is a hint of a less-pure--and therefor more
| feasible--future.
|
| To use the language of my favorite dream, Chomskian
| Anarcho-Syndicalism: imagine a social media network
| organized by a democratic non-profit entity akin to the
| Python or Linux Foundations, that then contracts out work
| to a hierarchy of smaller, purpose-built teams
| ("syndicates"), each of which may in turn contract w/ other
| teams. Each team would have to attract talent and negotiate
| enough income to pay them sufficiently still, of course,
| but there would be no team leader to make a surplus profit
| from the system -- any "surplus" would stay at the non-
| profit level, and thus necessarily be reinvested back into
| the product.
|
| In the current system, the reason Bluesky didn't do this
| off the bat is obvious: no one would loan them startup
| funds, as ownership investment is the de facto universal
| way to start up an unproven venture. But we can dream
| bigger and better, IMHO; both on a smaller scale by
| building upon already-proven open protocols like AT Proto,
| and on a larger scale by structuring the state & economy to
| support this kind of model equally, if not primarily.
| jarjoura wrote:
| All of the big tech companies today are the result of
| 100s of smaller, well intentioned tech companies that got
| acquired into these behemoths.
|
| I always look at how WhatsApp played out as the company.
| They were the good guys, and didn't want to get acquired.
| Zuckerberg, almost bankrupt FB at the time giving into
| all of the ridiculous demands WhatsApp made. No one at
| WhatsApp thought it was going to happen, until it did and
| did result in a once-in-a-lifetime transfer of wealth to
| several hundred employees.
| JustExAWS wrote:
| WhatsApp was acquired for $20 billion in cash and stock.
| Facebook was worth $170 billion at the time. The stock
| part of the acquisition wouldn't have any real financial
| impact on Facebook.
| autobodie wrote:
| > _but without profit, people can 't get paid, and they'll
| stop building_
|
| I wholeheartledly disagree. People build things all the
| time for things other than profit. In fact, most of the
| greatest things ever built were a loss for those who built
| them.
|
| Dignity is the best motivator. Profit only supercedes
| dignity when dignity is not on offer.
| krapp wrote:
| Profit supercedes dignity when one needs to eat, because
| one cannot eat dignity.
|
| Being able to spend a significant amount of time and
| effort on passion projects is a luxury most people can't
| afford.
| jarjoura wrote:
| There's no reason Bluesky has emulate what FB Newsfeed and
| Twitter/X did to solve engagement by promoting certain items
| over others.
|
| At the very least, they do have hindsight to learn from.
| immibis wrote:
| There's the profit motive. It's funded by venture capital,
| so it has to grow at all costs (check) and then cash out.
| rsynnott wrote:
| Twitter didn't start doing it (the pay-for-attention
| model) until 17 years after it was founded, so, I mean,
| bsky will probably be okay for a _while_.
| immibis wrote:
| Alternative reading: It took 17 years for venture
| capitalists to figure out the pay-for-attention model, at
| which point they immediately applied it to everything
| they could.
| pessimizer wrote:
| Bluesky is a private for-profit company that has taken $37M
| in venture capital.
|
| https://www.piratewires.com/p/interview-with-jack-dorsey-
| mik...
|
| > That was the second moment I thought, uh, nope. This is
| literally repeating all the mistakes we made as a company.
| This is not a protocol that's truly decentralized. It's
| another app. It's another app that's just kind of following
| in Twitter's footsteps, but for a different part of the
| population.
|
| > Everything we wanted around decentralization, everything we
| wanted in terms of an open source protocol, suddenly became a
| company with VCs and a board. That's not what I wanted,
| that's not what I intended to help create.
| mullingitover wrote:
| They've done an incredible job running with an extremely low
| headcount and crazy efficient use of hardware. It would be easy
| to 10x their expenses if they were blindly following the
| standard cloud deployment playbook. Hopefully this level of
| efficiency mean they don't have to work as hard and can stay
| pre-revenue, a pure play, for a very long time.
| culi wrote:
| I love Mastodon but I have to admit that BlueSky has clearly
| out-engineered them. Of course they started with much more
| expertise and resources. I hope ActivityPub compatibility soon
| to unite the two
| knallfrosch wrote:
| Anyone following hundreds of thousands of users is obviously a
| bot account scraping content. I'd ban them and call it a day.
|
| However, I do love reading about the technical challenge. I think
| Twitter has a special architecture for celebrities with millions
| of followers. Given Bluesky is a quasi-clone, I wonder why they
| did not follow in these footsteps.
| psionides wrote:
| You don't need to follow anyone (or even have an account) to
| scrape content... Someone following a huge amount of accounts
| usually wants to get a lot of followers quickly this way
| through follow-backs.
| mikemitchelldev wrote:
| Yes, and Starter Packs make this possible.
| ruined wrote:
| if you want to scrape all the content, that's what the firehose
| is for, and it's allowed.
|
| the only reason to mass-follow is for spam purposes.
| Retr0id wrote:
| This does assume that scrapers are smart, and often they're
| really not. They have infrastructure for scraping HTML from
| webpages at scale and that is the hammer they use for all
| nails. (e.g. Wikipedia has to fight off scraper traffic
| despite full archives being available as torrents, etc.)
|
| In this case I agree though, they're all spammers and/or
| "clout farmers", or trying to make an account seem more
| authentic for future scams. They want to generate follow
| notifications in the hope that some will follow them back
| (and if they don't, they unfollow again after some interval).
| sarchertech wrote:
| 100%. I ran a job board where we provided a nice machine
| readable XML feed of all of our jobs, but we had bots that
| insisted on using the standard search box. Searching by
| city using an alphabetized list.
|
| Geographic search to was the most expensive thing they
| could have done and no matter what we did we couldn't get
| them to use the XML feed.
|
| I even tried returning a link to the feed when we detected
| a bot. No dice. They just kept working around the bot
| detection.
| culi wrote:
| Maybe not hundreds of thousands but I'd follow anybody that
| looks remotely interesting and then primarily use customized
| feeds. E.g. if I wanna hear about union news, my personal irl
| network, etc I check that feed
| tshaddox wrote:
| Or just enforce a maximum number of followed accounts.
| ARandumGuy wrote:
| No matter how high you set a maximum limit for interactions
| on social media (followers, friends, posts, etc), _someone_
| will reach the limit and complain about it. I can see why
| Bluesky would prefer a "soft limit", where going above the
| limit will degrade the experience. It gives more flexibility
| to adjust things later, and prevents obnoxious complaints
| from power users with outsized influence.
| tshaddox wrote:
| I'm skeptical that the people who would complain about that
| wouldn't find something else to complain about if you
| resolved the first complaint. I'd recommend implementing
| product features that you think are reasonable and
| accepting the fact that you will get complaints from people
| who disagree.
| DeepSeaTortoise wrote:
| Potential solutions:
|
| - Make it easy to systematically unfollow people (or
| degrade them to a different tier, see below, or sort them
| automatically into a different feed; maybe even allow
| automatic following of certain people, like your cities
| mayor or local ice cream parlors). Like based on recent
| activity, last engagement with a post, type of content
| (pictures, videos, links ...), on a schedule (e.g. follow
| for 3 yeard, follow until 2028), special status (family,
| friends, member of congress, member of city council,
| mayor...), number/ratio of common followers, regex
| expressions, recommendations by certain accounts, letter-
| to-word ratio, season, planetary alignment, weather, age,
| train departure time, side-chaing based on other accounts,
| force accounts to play russian unfollow roulette, urgency
| to pee, healthcare CEO life expectancy derivative, ... or
| any combination of these.
|
| - Allow different tiers of following someone. Like friends
| (never unfollow, always fetch updates), family (never
| unfollow, rate limit high-energy uncles), news (filter
| based on urgency or current topics of interest),
| politicians (highlight as untrustworthy, attach link to
| donation and board membership disclosure, attach term-limit
| and next election countdown), local businesses (hard rate
| limit, attach opening hours), bookmark (never unfollow, no
| updates), ... maybe multiple tiers in each category and
| allow those being followed to either temporarily boost
| their tier (or tiers of certain posts) or e.g. once per
| year.
|
| - Allow people from exempting some of their posts from not
| being shown to some of their followers. E.g. two per week
| and an additional 5 per month.
|
| - Allow people to choose which followers should be given a
| higher priority when writing posts to their feeds.
| steveklabnik wrote:
| > Given Bluesky is a quasi-clone, I wonder why they did not
| follow in these footsteps.
|
| There are only six users with over a million followers, and
| none with two million yet.
|
| I'm sure they'll get there.
| mikemitchelldev wrote:
| BlueSky has starter packs that allow you to mass follow in the
| click of a button. You join 10 starter packs in one day, you
| are following over 1000 people. Sometimes following others is
| the only way to get people to engage with your content.
| ChuckMcM wrote:
| As a systems enthusiast I enjoy articles like this. It is really
| easy to get into the mindset of "this must be perfect".
|
| In the Blekko search engine back end we built an index that was
| 'eventually consistent' which allowed updates to the index to be
| propagated to the user facing index more quickly, at the expense
| that two users doing the exact same query would get slightly
| different results. If they kept doing those same queries they
| would eventually get the exact same results.
|
| Systems like this bring in a lot of control systems theory
| because they have the potential to oscillate if there is positive
| feedback (and in search engines that positive feedback comes from
| the ranker which is looking at which link you clicked and giving
| it a higher weight) and it is important that they not go crazy.
| Some of the most interesting, and most subtle, algorithm work was
| done keeping that system "critically damped" so that it would
| converge quickly.
|
| Reading this description of how user's timelines are sharded and
| the same sorts of feedback loops (in this case 'likes' or
| 'reposts') sounds like a pretty interesting problem space to
| explore.
| culi wrote:
| What became of Blekko?
| an_ko wrote:
| > It was acquired by IBM in March 2015, and the service was
| discontinued.
|
| -- https://en.wikipedia.org/wiki/Blekko
|
| Perhaps GP has a more interesting answer though.
| ChuckMcM wrote:
| That's the correct answer, IBM wanted the crawler mostly to
| feed Watson. Building a full search engine (crawler,
| indexer, ranker, API, web application) for the English
| language was a hell of an accomplishment but by the time
| Blekko was acquired Google was paying out tens of billions
| of dollars to people to send _them_ and only them their
| search queries. For a service that nominally has to live on
| advertising revenue getting humans to use it was the only
| way to be net profitable, and you can 't spend billions
| buying traffic and hope to make it back on advertising as
| the #3 search engine in the English speaking markets.
|
| There are other ways to monetize search (look at Kagi for
| example) than advertising. Blekko missed that window
| though. (too early, Google needed to get a crappy as it is
| today to make the value of a spam free search engine
| desirable)
| chrisweekly wrote:
| Not my Q but thanks for the interesting history.
|
| Also, (for other readers), I'm a huge fan of Kagi. Highly
| recommended.
| NetOpWibby wrote:
| I really thought Neeva was gonna make it. I'm glad Kagi
| swooped in when they exited.
| NetOpWibby wrote:
| Blekko was gone by the time I learned about it. Recently
| (past few years) I emailed someone who worked on Blekko
| to get his opinion on a search engine concept I still
| have yet to start. His advice was to not bother competing
| with Google (obviously) LOL!
|
| I don't know if anyone's embarked on a P2P search engine
| but that's essentially my concept. Anyhoo, thanks for the
| inspiration!
| ChuckMcM wrote:
| Peer to peer would be tough, you really need a 10G
| network connection to some tier 1 provider, and about
| 2500 machines to distribute the crawling/serving load.
| (that is if you want to do a full stack search engine).
| And while you can run that infrastructure for on the
| order of $100K/month (not counting depreciation) that
| means you need roughly $5K/day in revenue from that
| cluster. At $10 RPM ($10 revenue per thousand queries)
| you're looking at a minimum of 500,000 'real' search
| queries during 'English time' (roughly 7AM to 11PM GMT).
| That's 31,250 queries per hour or ~9 queries per second
| (average).
|
| And that just pays to keep the lights on at the
| colocation center. If you're paying off the development
| costs (30 - 50 developers over 2 - 3 years) and the cost
| of an office somewhere. You'll want at least double that
| revenue or you'll go broke before you break even.
|
| Ideally you are the 'go to' place for people looking to
| buy something as those queries make money. People
| researching Douglas Fairbanks for a high school essay
| consume queries but don't generate ad revenue.
|
| It isn't for the faint of heart.
| NetOpWibby wrote:
| When you don't know what you don't know...wow.
|
| I know "search is hard" in the general sense but context
| is lacking (not a lot of details online from ex-search
| teams). It's always been apparent to me that you must
| have some other high-grossing product if you want to get
| into search or video, if only to pay for the servers.
|
| Thank you for providing your context!
| immibis wrote:
| Darknet Lantern is a decentralized _searchable
| directory_. It 's probably not going to take off, but it
| could inspire something else. Servers spider other
| servers with the same software, and synchronized their
| data.
| NetOpWibby wrote:
| I've never heard of this before but it looks interesting.
| Thanks for the tip!
| ChuckMcM wrote:
| Yup, directory services are a lot easier to do peer-to-
| peer. Pinboard.in is a good shared directory (sort of
| Yahoo! without the editorial). They can yield excellent
| quality when you're searching for something that someone
| has 'indexed' with them, but poor recall when it comes to
| the set of all possible answers.
|
| Doing it peer to peer without editorial allows sites to
| 'get into' the index easily which has its own plusses and
| minuses.
| gregw134 wrote:
| Would you be willing to share more about how you guys did click
| ranking at Blekko? It's an interesting problem.
| snailmailman wrote:
| I guess I hadn't considered that search engines could be
| reranking pages on the fly as I click them. I've been seeing my
| DuckDuckGo results shuffle around for a while now thinking it's
| an awful bug.
|
| Like I click one page, don't find what I want, and go back
| thinking "no, I want that other result that was below" and it's
| an entirely different page with shuffled results, missing the
| one that I think might have been good.
| PaulHoule wrote:
| That's connected with a basic usability complaint about
| current web interfaces, that ads and recommended content
| aren't stable. You very well might want to engage with an ad
| after you are done engaging what you wanted to engage with
| but you might never see it again. Similarly, you might see
| two or three videos that you want to click on on the side of
| a YouTube video you're watching but you can only click on one
| (though if you are thinking ahead you can open these in
| another tab.)
|
| On top of that immediate frustration, the YouTube style
| interface here
|
| https://marvelpresentssalo.com/wp-
| content/uploads/2015/09/id...
|
| collects terrible data for recommendations because, even
| though it gives them information that you liked the thumbnail
| for a video, they can't come to any conclusion about whether
| or not you liked any of the other videos. TikTok, by focusing
| on one video at a time, collects much better information.
| 4ggr0 wrote:
| > though if you are thinking ahead you can open these in
| another tab
|
| or add it to the "Watch Later" playlist :) so you can watch
| it...later.
| cgriswald wrote:
| I don't use DDG, but in my (very limited, just now) testing
| it doesn't seem to shuffle results unless you reload the page
| in some way. Is it possible you're browser is reloading the
| page when you go back? If so, setting DDG to open links in
| new tabs might fix this problem.
| snailmailman wrote:
| Interesting. Maybe something in my configuration is
| affecting it. I'll have to look into it
| numeri wrote:
| This behavior started happening for me in the last few
| months. If I click on a result, then go back, I have
| different search results.
|
| I've found a workaround, though - click back into the DDG
| search box at the top of the page and hit enter. This then
| returns the original search results.
| gtfiorentino wrote:
| Hi - I work on search at DuckDuckGo. Do you mind sharing a
| bit more detail about this issue? What steps would allow us
| to reproduce what you're seeing?
| dwedge wrote:
| Similar to how Google images loads lower quality blurred
| thumbnails towards the bottom of the window at first so that
| the user thinks they loaded faster
| aqueueaqueue wrote:
| This is less a question of perfection and one of trade off's.
| Laws of physics put a limit on how efficiently you can keep
| data in NYC and London in perfect sync, so you choose CAP-style
| trade-offs. There are also $/SLO trade-offs. Each 9 costs more
| money.
|
| I like your example it is very interesting. If I get to work on
| (or even hear someone in my team is working on) such
| interesting problems and I can hear about it, I get happy.
|
| Interesting problems are rare because like a house you might
| talk about brick vs. Timber frame once, but you'll talk about
| cleaning the house every week!
| genewitch wrote:
| PID techniques useful?
| gopher_space wrote:
| > Some of the most interesting, and most subtle, algorithm work
| was done keeping that system "critically damped" so that it
| would converge quickly.
|
| Looking back at my early work with microservices I'm wondering
| how much time I would have saved by just manually setting a
| tongue weight.
| sphars wrote:
| When I go directly to a user's profile and see all their posts,
| sometimes one of their posts isn't in my timeline where it should
| be. I follow less than 100 users on Bluesky, but I guess this
| explains why I occasionally don't see a user's post in my
| timeline.
|
| Lossy indeed.
| Eric_WVGG wrote:
| Are you using an app, website, or combination?
|
| Various clients (I'm writing one) interpret the timeline
| differently, as a feed that shows literally everything includes
| could things that most people would find undesirable or
| irrelevant. (replies to strangers, replies to replies to
| replies, etc)
| sphars wrote:
| I'm using the official android app. There has been an update
| or two so I'll have to confirm it's still happening
| Retr0id wrote:
| If another user you follow reposted or replied to a post, it
| can affect its order in your following feed. You shouldn't be
| seeing any loss as described in the article from following only
| 100 users.
| sphars wrote:
| I've experienced it with "first-party" posts, not replies. A
| post wouldn't show in my timeline but would on the user's
| profile. This is the official android app, but there has been
| an update or two so I'll have to double check again
| trhway wrote:
| So the system design puts the burden on what seems to be
| synchronous, not queued, writes to get easy reads. I usually
| prefer simpler cheaper writes at the cost of more complicated
| reads as the reads scale and parallelize better.
| pfraze wrote:
| you're underestimating the read load, by a lot
| skybrian wrote:
| This design makes sense if you didn't previously have any limit
| on the number of people an account could follow. But why not have
| a limit?
| whyrusleeping wrote:
| people get so up in arms when you suggest there might be a
| limit on how many people they can follow.
| timewizard wrote:
| > This process involves looking up all of your followers, then
| inserting a new row into each of their Timeline tables in reverse
| chronological order with a reference to your post.
|
| Seriously? Isn't this the nut of your problem right here?
| jsnell wrote:
| What alternative design did you have in mind, given that a
| Twitter-like data model of individual follows is likely a
| strict product requirement?
|
| There are obviously other ways of doing it (doing the timeline
| propagation in a batch job, fanning out the reads rather than
| the writes), but they've got their own problems. Probably worse
| ones.
| pphysch wrote:
| Wouldn't a hybrid approach makes sense?
|
| Periodically classify users as hot/cold based on their
| activity, build hot-follower timelines on write, and build
| cold-follower timelines on read.
| jsnell wrote:
| You'd still have exactly the same hot write path, it'd just
| have maybe 50% of the load. That could be a legit
| optimization, but not having it hardly warrants an
| incredulous "seriously?" like the OP's.
|
| (And the same for the inverse hybrid strategy of
| quarantining the writes of highly followed users and
| handling their fan-out at read time. A neat optimization,
| and maybe even absolutely once you have accounts with 100M
| followers. But the vast majority of posts would still be
| handled via the original strategy.)
| PaulHoule wrote:
| An airline reservation system has to be perfect (no slack in
| today's skies), a hotel reservation can be 98% perfect so long as
| there is some slack and you don't mind putting somebody up in a
| better room than they paid for from time to time.
|
| A social media system doesn't need to be perfect at all. It was
| clear to me from the beginning that Bluesky's feeds aren't very
| fast, not like they are crazy slow, but if it saves money or
| effort it's no problem if notifications are delayed 30s.
| singleshot_ wrote:
| Does the fact that an airline booking system must be perfect
| explain why so many flights are overbooked or cancelled?
| rconti wrote:
| No, overbooking is a business decision justified by the fact
| that, statistically, not all passengers will actually show up
| for their flight, and lower load factors cost money.
| josefresco wrote:
| What is the "no show" rate?
| nightpool wrote:
| A 2019 study of 5 European airports in 2019 had no-show
| rates of 14.4%: https://www.ozion-
| airport.com/product/comparative-analysis-n...
|
| However, my understanding is that airlines have much more
| sophisticated per-flight and per-passenger models that
| calculate the predicted no-show factor based on the
| historical rates for that particular route (e.g. you're
| more likely to get more no-shows in business class flying
| from NYC to SF compared to holiday travelers with a
| reservation on the Florida Keys)
| SteveNuts wrote:
| That blows my mind, I would expect maybe 1 or 2
| passengers per plane at most. I'm trying to think of what
| factors would cause that many no-shows, it has to be
| mostly missed connections?
|
| I can't imagine spending hundreds of dollars and just not
| showing up.
| lhoff wrote:
| A friend of mine works for a Management Consultancy firm
| and they have full flex tickets if they miss the 8pm
| flight home they can take the next one or fly back the
| next morning. All without additional fees. So I believe
| business travel is the biggest factor when it comes to
| missed flights.
|
| Side note: His employer is the biggest client of a major
| European airline.
| listenallyall wrote:
| No additional fees, but the cost of the ticket is
| typically sky-high. In many cases the company could
| purchase 3 restricted-fare tickets for less than the
| unrestricted fare. (And also, the consultants want to get
| home, they HATE staying extra nights, so they have
| inherent motivation to catch the original flight). You'd
| think that management consulting firms, you know, who are
| supposed to be good at optimization and reducing costs,
| would have figured this out.
| danielheath wrote:
| ... or perhaps they prioritize ensuring their consultants
| feel like they're being looked after - replacing them
| when they find another job is very expensive.
| listenallyall wrote:
| What part of my comment suggests consultants aren't
| "looked after?" The idea that the consultants' personal
| time is so low on the scale of priorities that the
| company prefers to pay 3x or 4x for plane tickets to
| ensure they are available to spend extra time with
| clients, rather than their families, suggests that the
| company is specifically NOT looking after the
| consultants' well-being.
| dullcrisp wrote:
| Yeah, replace consultants with clients in that comment
| and you'll have the right of it.
| pc86 wrote:
| That the consultants' personal time is high on the scale
| is why they pay for the flex-fare tickets. So you miss an
| 8pm flight for one reason or another - it happens, so
| "don't miss the flight" isn't a valid retort.
|
| Do I feel better taken care of if I can just take the
| following morning's flight at no additional cost or
| hassle, or if I now need to contact someone at Amex
| Global Business Travel, have them try to get me on a
| flight, have another expense, potentially not be able to
| get on the next flight or the one after that, etc.
| listenallyall wrote:
| The company can start with heavily-discounted restricted
| tickets and eat them when a consultant needs to extend
| the trip. Today it's even easier because airlines have
| eliminated change fees - so essentially any ticket can be
| canceled and those funds credited to a future flight.
| It's all still "no cost or hassle" to the consultant
| since they aren't paying for the flights to begin with.
| CPLX wrote:
| They aren't following the same rules as you, they're on
| negotiated rates/agreements.
| NikolaNovak wrote:
| * A massive bulk of flex rates is different price than a
| single flex ticket
|
| * when I travelled to a single city with 20+ colleagues
| for several years, our nightly hotel rate was less than
| quarter of published rate. I don't think we got anything
| like that on Airlines, but nevertheless, bulk gets
| discounts.
|
| * I forget details but between specific frequent cities
| which had hourly flights, we had full flexibility and we
| used this all the time. So we might catch a flight hour
| earlier if there was one available, or hour later if need
| be than booked.
|
| Basically to everybody's point, business travel is very
| different than vacation travel and intuition from once a
| year personal trip don't apply.
| listenallyall wrote:
| * Hotels are required by law to have a published "rack
| rate" and it's common for both leisure and business
| travelers to book at more than half, or more, off those
| rates. Airlines don't have that.
|
| * Yes of course there are negotiated discounts for major
| corporations- but full flexibility is still far more
| expensive than restricted tickets, just like business is
| still far more expensive than economy.
|
| * Most airlines offer guaranteed same-day changes, or at
| least priority standby, to high-status loyalty members
| (which most consultants are) so buying full-flex tickets
| to get an hour earlier/later flexibility is redundant -
| basically, paying for a benefit you already have.
| mjcl wrote:
| They have figured it out, the client is paying for it.
| listenallyall wrote:
| This is actually the most sensible response
| lukas099 wrote:
| > You'd think that management consulting firms, you know,
| who are supposed to be good at optimization and reducing
| costs, would have figured this out.
|
| Maybe they've figured out that enabling their employees
| to stay a few extra hours without worry to finish a deal
| is worth it.
| pests wrote:
| Similar reasons people buy overnight sleeper tickets from
| the west coast to asia for sky-high prices. That person
| has to be there, and they have to be rested. If they are
| negotiating 100m+ dollar contracts its a no-brainer.
| listenallyall wrote:
| Intercontinental business-class tickets have a tangible
| benefit, guaranteed lie-down seat and relaxing
| accommodations that enable rest, as you correctly point
| out. Flexible airline tickets have no tangible benefit,
| it is simply risk mitigation (avoiding wasting a
| nonrefundable ticket if a person can't make a flight) and
| like other risk-mitigation products such as insurance,
| extended warranties, car rental protection, etc, it is
| priced so that the cost exceeds the benefit.
|
| And as stated elsewhere, the majority of consultants are
| relatively junior people whose role has nothing to do
| with negotiating contracts.
| listenallyall wrote:
| Vast majority of consultants are working on engagements
| where "the deal" was negotiated weeks/months earlier.
|
| The goal isn't to eliminate flexibility, it's
| understanding probabilities. If fully-flexible/refundable
| flights are 3x the fare of restricted, then in aggregate,
| the company could have a chunk of consultants throw out
| their original reservations and rebook later flights, and
| still save money. Yes- sometimes consultants need to stay
| longer than planned, but in an age of prioritizing "work-
| life balance," most consultants are encouraged to stick
| to their schedule and get back home when originally
| planned.
| shagie wrote:
| Delays getting to the airport and missing the plane.
| Cancelations with full refund. "Hidden city" ticketing.
| Layover delays. Businesses booking blocks. Flexible
| flights ( https://www.travelperk.com/guides/flexible-
| travel/flexible-f... ). Changing / rebooking flights for
| an earlier or later time.
| packetlost wrote:
| I'm sure other factors such as sudden illness and
| migrateable tickets make a sizeable chunk too.
| vidarh wrote:
| Keep in mind they sell a lot of tickets where one of the
| features that allows for a premium price is that they
| allow late cancellations or changes to other flights.
| Holiday travelers are pretty "reliable", but business
| travelers might have changed needs at the drop of a hat
| (say you meet another prospective client on a business
| trip and decide to stay another day to fit in a face-to-
| face meeting).
| Spooky23 wrote:
| Remember lots of business travelers have connections or
| flex schedules. When I had to go to the West coast for
| business, I usually have full fare tickets and book a
| later flight. If I had flexibility, I'd switch to an
| earlier plane or first class.
| rsynnott wrote:
| European airports in 2019: A lot of these would be 10
| euro Ryanair/Easyjet/Whatever flights, probably.
|
| (The really ultra cheap Ryanair flights are less of a
| thing now, but in 2019 they were very much a factor)
| artee_49 wrote:
| I think you'll have to pay a team millions to figure that
| out, it is unlikely to be a static rate but rather
| decided based on multiple traits like time of year, time
| of flight, distance of flight, cost of ticket, etc.
| pc86 wrote:
| The airline has literally all of the data on this, they
| definitely do not have to pay a team millions.
| patmorgan23 wrote:
| They probably do pay millions of dollars in wages for
| business analysts to figure out what this rate is on
| their flights.
| pc86 wrote:
| They probably just have an SSRS report that prints out in
| a few dozen offices automatically on some schedule.
|
| I'm not trying to be pedantic but this is table stakes
| stuff. I know we're supposed to shy away from saying
| things like this but compared to the other engineering
| that airlines have to do, this is easy. It costs - at
| most, including wages - a few tens of thousands of
| dollars yearly to come up with these figures. It's a
| fraction of the salary of one United Airlines BA.[0] This
| cost might go up if one of the senior developers
| convinces their boss that this needs to be a machine
| learning model but unless they're resume pumping it's
| going to be at most PCA and a regression.
|
| This is not a team of people working for months on this
| one thing.
|
| [0] https://www.glassdoor.com/job-listing/analyst-
| revenue-manage...
| rconti wrote:
| Especially for a free service!
|
| Think about other ad-supported sites. If you're an engineer
| working on an ad-supported product, the perfect consistency you
| strive for in your code is not the product. The product is the
| sum of all of the content the user sees. And the costs of the
| tradeoffs you make are paid for by ads.
|
| Am I willing to see 10x more ads for perfect consistency?
| Definitely not.
| darknavi wrote:
| It's funny because from my experience airline systems are very
| imperfect (timing wise).
|
| I (unwisely) tried to purchase an Icelandair ticket via the
| Chase travel portal. I would get a reservation number, go buy
| seats on Icelandair's website, and a few days later the entire
| reservation would vanish into the ether. Rinse and repeat 3x.
|
| I can't remember the exact verbiage, but basically tickets can
| be "reserved" and "booked". One means the ticket is allocated,
| and one means the ticket is actually paid for. I eventually sat
| on the phone with an executive support person as they booked
| the ticket and got it all the way through. It turns out Chase
| reserves a ticket on an airline but as an SLA of ~3 days to
| actually pay for the ticket. Icelandair's requires a ticket to
| be paid with in 24 hours, so it was timing out.
| scarface_74 wrote:
| (Replying to both you and the parent poster)
|
| Airlines are far from perfect. They overbook flights and
| sometimes have to ask people leave and pay them for the
| inconvenience. My wife and I once got $1000 a piece and a
| hotel and food voucher to volunteer to take a flight the next
| day on a layover in Atlanta.
|
| As far as your particular situation, the number one rule of
| using a third party portal to book flights or hotels is -
| don't.
|
| I understand that Iceland Air is not a transfer partner of
| Chase. But even in that case, I would just wait to use my
| points until I could use a transfer partner.
|
| On the earning side if paying cash, the difference between
| 2x/3x points when booking directly and 5x when going through
| the portal just isn't worth the risk.
| ericpauley wrote:
| Overbooking is not a mistake, though. People miss flights
| for many reasons, and the airlines predict this with
| impressive accuracy, to the point that they can afford to
| pay tremendous sums for being wrong and yet still come out
| ahead.
| branko_d wrote:
| > afford to pay tremendous sums for being wrong
|
| Or they can just haul you forcefully from the flight you
| paid for:
|
| https://en.wikipedia.org/wiki/2017_United_Express_passeng
| er_...
| scarface_74 wrote:
| This is ridiculous. There must have been some amount of
| money that would have convinced four people to deplane
| voluntarily. $800 was ridiculously low.
| ericpauley wrote:
| Case in point! Imagine how much United(Express) settled
| this for.
| gamedever wrote:
| Miscommunication leads to bad outcomes. One missed message out
| of order could easily lead to a fight, a lawsuit, a flash mob,
| threats of violence - that then need to be taken seriously,
| swatting, DOXxing, etc...
|
| Msg 1: I hate ___insert_controversal_person_category_here___
|
| Msg 2: Is the kind of statement that really sets me off
|
| Msg 1 has a very different meaning if you don't see Msg 2.
| pjc50 wrote:
| This can already happen without help from the platform.
| gamedever wrote:
| Sure, but that doesn't mean the platform should make it
| worse.
|
| Trying to have a conversation on flaky platform is hell.
| nonrandomstring wrote:
| > airline reservation system has to be perfect (no slack in
| today's skies)
|
| The slack just gets moved. Airlines oversell by about 8
| percent. All systems need _some_ slack in them. Isn 't that
| kinda Bob's Law or something?
| rakoo wrote:
| Ok I'm curious: since this strategy sacrifices consistency, has
| anyone thoughts about something that is not full fan-out on reads
| or on writes ?
|
| Let's imagine something like this: instead of writing to every
| user's timeline, it is written once for each shard containing at
| least one follower. This caps the fan-out at write time to
| hundreds of shards. At read time, getting the content for a given
| users reads that hot slice and filters actual followers. It
| definitely has more load but
|
| - the read is still colocated inside the shard, so latency
| remains low
|
| - for mega-followers the page will not see older entries anyway
|
| There are of course other considerations, but I'm curious about
| what the load for something like that would look like (and I
| don't have the data nor infrastructure to test it)
| rconti wrote:
| > Additionally, beyond this point, it is reasonable for us to not
| necessarily have a perfect chronology of everything posted by the
| many thousands of users they follow, but provide enough content
| that the Timeline always has something new.
|
| While I'm fine with the solution, the wording of this sentence
| led me to believe that the solution was going to be imperfect
| chronology, not dropped posts in your feed.
| artee_49 wrote:
| I am a bit perplexed though as to why they have implemented fan-
| out in a way that each "page" is blocking fetching further pages,
| they would not have been affected by the high tail latencies if
| they had not done this,
|
| "In the case of timelines, each "page" of followers is 10,000
| users large and each "page" must be fanned out before we fetch
| the next page. This means that our slowest writes will hold up
| the fetching and Fanout of the next page."
|
| Basically means that they block on each page, process all the
| items on the page, and then move on to the next page. Why
| wouldn't you rather decouple page fetcher and the processing of
| the pages?
|
| A page fetching activity should be able to continuously keep
| fetching further set of followers one after another and should
| not wait for each of the items in the page to be updated to
| continue.
|
| Something that comes to mind would be to have a fetcher component
| that fetches pages, stores each page in S3 and publishes the
| metadata (content) and the S3 location to a queue (SQS) that can
| be consumed by timeline publishers which can scale independently
| based on load. You can control the concurrency in this system
| much better, and you could also partition based on the shards
| with another system like Kafka by utilizing the shards as keys in
| the queue to even "slow down" the work without having to
| effectively drop tweets from timelines (timelines are eventually
| consistent regardless).
|
| I feel like I'm missing something and there's a valid reason to
| do it this way.
| abound wrote:
| I interpreted this as a batch write, e.g. "write these 10k
| entries and then come back". The benefit of that is way less
| overhead versus 10k concurrent background routines each writing
| individual rows to the DB. The downside is, as you've noted,
| that you can't "stream" new writes in as older ones finish.
|
| There's a tradeoff here between batch size and concurrency, but
| perhaps they've already benchmarked it and "single-threaded"
| batches of 10k writes performed best.
| mifydev wrote:
| "Hot Shards in Your Area" - 10/10 heading
| NoGravitas wrote:
| The funny thing is that all of the centralization in Bluesky is
| defended as being necessary to provide things like global search
| and all replies in a thread, things that Mastodon simply punts on
| in the name of decentralization. But then ultimately, Bluesky has
| to relax those goals after all.
| ramblejam wrote:
| True. In context though Bluesky can tweak the volume knob as
| and when they see fit, whereas for Mastodon it's stuck where it
| is.
| arcastroe wrote:
| I found it odd to base the loss-factor on the number of people
| you follow, rather than a truer indication of timeline-update-
| frequency. What if I follow 4k accounts, but each of those
| accounts only posts once a decade? My timeline would be become
| unnecessarily lossy.
| crabbone wrote:
| Anecdotally, I ran into a similar solution "by chance".
|
| Long ago, I worked for a dating site. Our CTO at the time was a
| "guest of honor" who was brought in by a family friend who was
| working in the marketing at the time. The CTO was a university
| professor who took on a job as a courtesy (he didn't need the
| money nor fame, he had enough of both, and actually liked
| teaching).
|
| But he instituted a lot of experimental practices in the company.
| S.a. switching roles every now and then (anyone in the company
| could apply for a different role except administration and try
| themselves wearing a different hat), or having company-wide
| discussions of problems where employees would have to prepare a
| presentation on their current work (that was very unusual at the
| time, but the practice became more institutional in larger
| companies afterwards).
|
| Once he announced a contest for the problem he was trying to
| solve. Since we were building a dating site, the obvious problem
| was matching. The problem was that the more properties there were
| to match on, the longer it would take (beside other problems that
| is). So, the program was punishing site users who took time to
| fill out the questionnaires as well as they could and favored the
| "slackers".
|
| I didn't have any bright ideas on how to optimize the matching /
| search for matches. So, ironically, I asked "what if we just
| threw away properties beyond certain threshold randomly?" I was
| surprised that my idea received any traction at all. And the
| answer was along the lines of "that would definitely work, but I
| wouldn't know how to explain this behavior to the users". Which,
| at the time, I took to be yet another eccentricity of the old
| man... but hey, the idea stuck with me for a long time!
| detuur wrote:
| The answer to that reply is you don't need to explain it to
| your users. People are used to fuzzy/best-effort sort of
| matching, especially when it's specifically presented as a
| "matching algorithm" instead of a "filter".
| cavisne wrote:
| AWS has a cool general approach to this problem (one badly
| behaving user effecting others on their shard)
|
| https://aws.amazon.com/builders-library/workload-isolation-u...
|
| The basic idea is to assign each user to multiple shards,
| decreasing the changes of another user sharing _all_ their shards
| with the badly behaving user.
|
| Fixing this issue as described in the article makes sense, but if
| they did shuffle sharding in the first place it would cover any
| new issues without effecting many other users.
| artee_49 wrote:
| I think shuffle sharding is beneficial for read-only replica
| cases, not for writing scenarios like this. You'll have to
| write to the primary and not to a "virtual node". Right? Or am
| I understand it incorrectly? I just read that article now.
| pornel wrote:
| I wonder why timelines aren't implemented as a hybrid gather-
| scatter choosing strategy depending on account popularity (a
| combination of fan-out to followers and a lazy fetch of popular
| followed accounts when follower's timeline is served).
|
| When you have a celebrity account, instead of fanning out every
| message to millions of followers' timelines, it would be cheaper
| to do nothing when the celebrity posts, and later when serving
| each follower's timeline, fetch the celebrity's posts and merge
| them into the timeline. When millions of followers do that, it
| will be cheap read-only fetch from a hot cache.
| ericvolp12 wrote:
| This is probably what we'll end up with in the long-run. Things
| have been fast enough without it (aside from this issue) but
| there's a lot of low-hanging fruit for Timelines architecture
| updates. We're spread pretty thin from a engineering-hours
| standpoint atm so there's a lot of intense prioritization going
| on.
| curious_cat_163 wrote:
| That's insightful. Keep up the good work!
| Xunjin wrote:
| Just to be clear, you are a Bluesky engineer, right?
|
| off-topic: how has been dealing with the influx of new users
| after X political/legals problems aftermath? Did you see an
| increase in toxicity around the network? And how has you
| (Bluesky moderation) dealing with it.
| petra wrote:
| Maybe this would be
| helpful:http://daslab.seas.harvard.edu/datacalculator/
| locusofself wrote:
| Why do they "insert" even non-celebrity posts into each
| follower's timeline? That is not intuitive to me.
| giovannibonetti wrote:
| To serve a user timeline in single-digit milliseconds, it is
| not practical for a data store to load each item in a
| different place. Even with an index, the index itself can be
| contiguous in disk, but the payload is scattered all over the
| place if you keep it in a single large table.
|
| Instead, you can drastically speed up performance if you are
| able to store data for each timeline somewhat contiguously on
| disk.
| wlonkly wrote:
| Think of it as pre-rendering. Of pre-rendering and JIT
| collecting, pre-rendering means more work but it's async, and
| it means the timeline is ready whenever a user requests it,
| to give a fast user experience.
|
| (Although I don't understand the "non-celebrity" part of your
| comment -- the timeline contains (pointers to) posts from
| whoever someone follows, and doesn't care who those people
| are.)
| rubslopes wrote:
| This problem is discussed in the beginning of the Designing
| Data-Intensive Applications book. It's worth a read!
| Brystephor wrote:
| Do you know the name of the problem or strategy used for
| solving the problem? I'd be interested in looking it up!
|
| I own DDIA but after a few chapters of how database work
| behind the scenes, I begin to fall asleep. I have trouble
| understanding how to apply the knowledge to my work but this
| seems like a useful thing with a more clear application.
| bitbckt wrote:
| Yes, we used the Yahoo! "Feeding Frenzy" paper as the basis
| for the design of Haplocheirus (the timeline service).
| VWWHFSfQ wrote:
| At some point they'll end up just doing the Bieber rack [1].
| It's when a shard becomes so hot that it just has to be its own
| thing entirely.
|
| [1] - https://www.themarysue.com/twitter-justin-bieber-servers/
|
| @bluesky devs, don't feel ashamed for doing this. It's exactly
| how to scale these kinds of extreme cases.
| stavros wrote:
| Given that BlueSky is funded by Twitter, I'm assuming they
| know a lot more than us on how Twitter architects systems.
| genewitch wrote:
| I've stood up machines for this before I did not know they
| had a name, and I worked at the mouse company and my parking
| spot was two over from a J. Beibe'rs spot.
|
| So now we have Slashdot effect, HN hug, and its not Clarkson
| its... Stephen Fry effect? Maybe can be Cross-Discipline -
| there's a term for when lots of UK turns their kettles on at
| the same time.
|
| I should make a blog post to record all the ones I can
| remember.
| k1t wrote:
| TV Pickup aka the Half Time Kettle Effect.
|
| https://en.wikipedia.org/wiki/TV_pickup
| bitbckt wrote:
| We never actually had a literal "Bieber Box", but the joke
| took off.
|
| Hot shards were definitely an issue, though.
| Imustaskforhelp wrote:
| Its so crazy.
|
| Thanks a lot for sharing this link.
| rsynnott wrote:
| > and later when serving each follower's timeline, fetch the
| celebrity's posts and merge them into the timeline
|
| I think then you still have the 'weird user who follows
| hundreds of thousands of people' problem, just at read time
| instead of write time. It's unclear that this is _better_,
| though, yeah, caching might help. But if you follow every celeb
| on Bluesky (and I guarantee you this user exists) you'd be
| looking at fetching and merging _thousands_ of timelines
| (again, I suppose you could just throw up your hands and say
| "not doing that", and just skip most or all of the celebs for
| problem users).
|
| Given the nature of the service, making read predictably cheap
| and writes potentially expensive (which seems to be the way
| they've gone) seems like a defensible practice.
| christkv wrote:
| You might mix the approaches based on some cut off point
| fc417fc802 wrote:
| > I suppose you could just throw up your hands and say "not
| doing that", and just skip most or all of the celebs for
| problem users
|
| Random sampling? It's not as though the user needs thousands
| of posts returned for a single fetch. Scrolling down and
| seeing some stuff that's not in chronological order seems
| like an acceptable tradeoff.
| JadeNB wrote:
| I understand that it's a different point, but how can someone
| write a whole essay called "When imperfect systems are good"
| without once mentioning Gabriel or
| https://en.wikipedia.org/wiki/Worse_is_better?
| Artoooooor wrote:
| Are users informed that they follow too many creators and now
| they will not see every post on their timelines?
| fastest963 wrote:
| To help avoid the hot shard problem, I wonder how capping
| followers per "timeline" would perform. Especially each user
| would have a separate timeline per 1000 followers and the client
| would merge them. You could still do the lossy part, if
| necessary, by only loading a percent of the actual timelines.
| That wouldn't help the celebrity problem but it was already
| acknowledged earlier that the solution to that is to not fan out
| celebrity accounts.
| cush wrote:
| "Hot Shards in Your Area"... I died
| inportb wrote:
| An interesting solution to a challenging problem. Thank you for
| sharing it.
|
| I must admit, I had some trouble following the author's
| transition from "celebrity" with many followers to "bot" with
| many follows. While I assume the work done for a celebrity to
| scatter a bunch of posts would be symmetric to the work done for
| a commensurate bot to gather a bunch of posts, I had the
| impression that the author was introducing an entirely different
| concept in "Lossy Timelines."
| jadbox wrote:
| So, let's say I follow 4k people in the example and have a 50%
| drop rate. It seems a bit weird that if all (4k - 1) accounts I
| follow end up posting nothing in a day, that I STILL have a 50%
| chance that I won't see the 1 account that posts in a day. It
| seems to me that the algorithm should consider my feed's age (or
| the post freshness of my followers). Am I overthinking?
| imrehg wrote:
| This feels like an edge case.
|
| The "reasonable limit" is likely set based on experimentation,
| and thus on how much people post on average and the load it
| generates (so the real number is unlikely to be exactly "2000",
| IMHO).
|
| If you follow a lot of people, how likely it is that their
| posting pattern is so different from the average? The more
| people you follow, the less likely that is.
|
| So while you can end up in such situation in theory, it would
| need to be a very unusual (and rare) case.
| kevincox wrote:
| Yeah, this seems concerning to me. Maybe now as the platform is
| new this isn't much of an issue. But as accounts go inactive
| people will naturally collect "dead" accounts that they are
| still following. On Facebook it isn't uncommon of to have old
| accounts of sociable people naturally collect thousands of
| friends.
|
| It seems that what they are trying to measure is "busy
| timelines" and it seems bag they could probably measure that
| more directly. For example what is the number of posts in the
| timeline over theast 24h? It seems that it should be fairly
| easy to use this as the metric for calculating drop rate.
| brianolson wrote:
| I think the 'law of large numbers' says that it's very unlikely
| for you to follow 4k and have _none_ of them posting. You could
| artificially construct a counter-example by finding 4k open but
| silent accounts, but that's silly.
|
| The other workaround is: follow _everyone_. Write some code to
| get what you want out of the jetstream event feed.
| https://docs.bsky.app/blog/jetstream
| spoaceman7777 wrote:
| Hmm. Twitter/X appears to do this at quite a low number, as the
| "Following" tab is incredibly lossy (some users are permanently
| missing) at only 1,200 followed people.
|
| It's _insanely_ frustrating.
|
| Hopefully you're adjusting the lossy-ness weighting and cut-off
| by whether a user is active at any particular time? Because,
| otherwise, applying this rule, if the cap is set too low, is a
| very bad UX in my experience x_x
| VWWHFSfQ wrote:
| > It's _insanely_ frustrating.
|
| > at only 1,200 followed people.
|
| I follow like, 50 people on bluesky. Who is following 1,200
| people? What kind of value do you even get out of your feed?
| peoplepostphew wrote:
| 1200 people is really nothing, specially if you have a job
| tangentially related to social media (for example
| journalists). It's really simple, you are not the same type
| of user. You have 50 "acquaintances", they have 1200
| "sources".
|
| The article is talking about people who have
| following/follower counts in the millions. Those are dozens
| of writes per second in one feed and a fannout of potentially
| millions. Someone with 1200 followers, if everyone actually
| posts once a day (most people do not) gets... a rate of 0.138
| writes per second.
|
| They should be background noise, irrelevant to the
| discussion. That level of work is within reasonable
| expectation. What they're pointing out is that Twitter is
| aggressively anti-perfectionist for no good technical reason
| - so there must be a business reason for it.
| VWWHFSfQ wrote:
| Why are you following 1,200 people? What is the point of
| your home feed? What are you trying to see?
| rjeli wrote:
| there are a lot of interesting people on twitter
| coldpie wrote:
| You are on the verge of discovering why non-chronological
| timelines exist :) It's not hard to imagine that there
| are 1200 people posting at least one thing a week that
| you would find interesting. The trouble is, if they also
| post 100 things that are not interesting, how does the
| software surface the interesting stuff without drowning
| you in the non-interesting stuff? How do you do that in a
| way that feels fair to the user ("I never see Friend X's
| posts because they're drowned out by the interesting
| stuff posted by the other 1199 people I follow")? It's
| tough!
| spoaceman7777 wrote:
| Personally, I always use chronological. I like to be able
| to hop on, and mingle with whatever percent of people are
| online and posting at any particular time.
|
| Replying to people right after they post is how you
| actually get to have conversations with people, and get
| to know them well, imo
| fc417fc802 wrote:
| > What are you trying to see?
|
| A scrollable feed of accounts that post interesting (to
| me) content.
| spoaceman7777 wrote:
| They're all people that I know, and 98% of them are
| mutuals. I regularly go through my list of accounts I'm
| following every 5-6 months, and get rid of people who
| have disconnected from me, have gone inactive, or I don't
| really know.
|
| I think the part you're missing here is that there are
| certain parts of twitter where the density of
| interconnection is really high, so you'll know people
| because you see them in the comments of a lot of threads,
| or through retweets.
|
| It's really not that hard to end up knowing 1,000+ people
| if you engage with a group or "corner" of twitter. Even
| moreso if it's multiple corners. (Like, AI, but also just
| frens, but also a little political corner, people from
| specific cities, etc.)
| throw10920 wrote:
| I can come up with 100 people I'd want to follow on Twitter,
| and I don't even have an account. Don't dismiss other
| people's use-cases if you don't have or understand them.
| yibg wrote:
| I think something like this was a FB engineering interview
| (several years ago), just for instagram feeds.
| alexnewman wrote:
| I don't see much call for blusky anymore....
| rsynnott wrote:
| I mean:
| https://bskycharts.edavis.dev/static/dynazoom.html?cgiurl_gr...
|
| Posts/sec are just off record levels.
| Claudus wrote:
| Record seems to be ~115, while current is ~63, or 55%.
| rsynnott wrote:
| Yeah, there was a short spike directly after the US
| elections, but, ignoring that, post rates are stable to
| somewhat rising.
| andsoitis wrote:
| Principle: Progress over perfection.
| udioron wrote:
| > some of them will do abnormal things like... well... following
| hundreds of thousands of other users.
|
| Sounds like Bluesky Pro.
| Nemo_bis wrote:
| "Lossy timelines" have already been implemented in ActivityPub
| and Mastodon by design. Will Bluesky ever catch up? It remains to
| be seen.
| dsauerbrun wrote:
| I'm a bit confused.
|
| The lossy timeline solution basically means you skip updating the
| feed for some people who are above the number of reasonable
| followers. I get that
|
| Seeing them get 96% improvements is insane, does that mean they
| have a ton of users following an unreasonable number of people or
| do they just have a very low number for reasonable followers. I
| doubt it's the latter since that would mean a lot of people would
| be missing updates.
|
| How is it possible to get such massive improvements when you're
| only skipping a presumably small % of people per new post?
|
| EDIT: nvm, I rethought about it, the issue is that a single user
| with millions of follows will constantly be written to which will
| slow down the fanout service when a celebrity makes a post since
| you're going through many db pages.
| friendzis wrote:
| When a system gets "overloaded", typically it enters
| exponential degradation of performance state, i.e. performs
| self ddos.
|
| > Seeing them get 96% improvements is insane
|
| TFA is talking about P99 tail latencies. It does not sound too
| insane to reduce tail latencies by extraordinary margins.
| Remember, it's just reshaping of latency distribution. In this
| case pathological cases get dropped.
| aloha2436 wrote:
| > does that mean they have a ton of users following an
| unreasonable number of people
|
| They do, there are groups of users on bluesky who follow
| inordinate numbers of other accounts to try and get follows
| back.
| Beretta_Vexee wrote:
| > does that mean they have a ton of users following an
| unreasonable number of people
|
| Look at the accounts of OnlyFans models, crypto influencers,
| etc. They follow thousands or even tens of thousands of
| accounts in the hope that we will follow them in return.
| mapt wrote:
| I don't see that accommodating this behavior is prosocial or
| technically desirable.
|
| Can you think of a use case?
|
| All sorts of bots want this sort of access, but whether there
| are legitimate reasons to grant it to them on a non-sharded
| basis is another question since a lot of these queries do not
| scale resources with O(n) even on a centralized server
| architecture.
| marksomnian wrote:
| From TFA:
|
| > Generally, this can be dealt with via policy and
| moderation to prevent abusive users from causing outsized
| load on systems, but these processes take time and can be
| imperfect.
|
| So it's a case of the engineers accepting that, however
| hard they try to moderate, these sorts of cases will crop
| up and they may as well design their infrastructure to
| handle them.
| tart-lemonade wrote:
| Given enough time, you'll end up with a lot of legitimate
| users who follow a huge number of accounts but rarely
| interact with more than a handful, similar to how many
| long-time YouTubers have a very high subscriber:viewer
| ratio (that is, they have way more subscribers than you
| would expect given their average view count), and there's
| nothing inherently suspicious about it. People lose access
| to their accounts, make new accounts, die, get bored, or
| otherwise stop watching the content but never bother
| unsubscribing because the algorithm recognized this and
| stopped recommending the channel's uploads to them.
|
| Bluesky doesn't have this problem yet because it's so
| young, so the outsized follow counts are mostly going to be
| from doomscrollers and outright malicious users, but even
| if it was exclusively malicious users, there is no perfect
| algorithm to identify them, much less do so before they
| start causing performance problems. Under those
| constraints, it makes sense to limit the potential blast
| radius and keep the site more usable for everyone.
| citrus1330 wrote:
| They were specifically looking at worst-case performance. P99
| means 99th percentile, so they saw 96% improvement on the
| longest 1% of jobs.
| thmrtz wrote:
| That's quite interesting and a challenge I have not thought of. I
| understand the need for a solution and I believe this works
| reasonably well, but I am wondering what is happening to users
| that follow a lot of accounts with below-average activity. This
| may naturally happen on new social media platforms with people
| trying out the service and possibly abandoning it.
|
| The ,,reasonable limit" is likely set to account for such an
| effect, but I am wondering if a per-user limit based on the
| activity of the accounts one follows will be an improvement on
| this approach.
| flaburgan wrote:
| The solution to this problem is known and implemented already:
| the social web should be distributed between thousands of pods
| which should contain at the maximum a few thousands users.
| Diaspora is already working like this for 15 years. It is
| technically harder to build initially but it then divide all the
| problems (maintenance, moderation, load, censorship, trust of the
| owner...) Which makes the network much more resilient. Bluesky
| knows that and they are allowing other people to host their
| software but they are really not pushing for it and it highly
| doubt that the experience of a user on a small external pod is
| the same than on bluesky.com.
| grishka wrote:
| This particular problem will still exist for a fediverse
| server. You follow 10k people? Nice, now you're getting ddos'd
| by their activities. Though, most fediverse servers being
| monolithic applications definitely helps.
| dtonon wrote:
| The typical problem of a centralized infrastructure.
|
| Indeed:
|
| > This means each user gets their own Timeline partition,
| randomly distributed among shards of our horizontally scalable
| database (ScyllaDB), replicated across multiple shards for high
| availability
| buxidao wrote:
| In the fanout design, why not dynamically move on to the next
| 10,000-user page as soon as all tasks for the current page are
| either queued or processing? Would that approach improve
| throughput, or could it introduce issues like resource
| contention?
| ramblejam wrote:
| Nice problem to have, though. Over on Nostr they're finding it a
| real struggle to get to the point where you're confident you
| won't miss replies to your own notes, let alone replies from
| other people in threads you haven't interacted with.
|
| The current solution is for everyone to use the same few relays,
| which is basically a polite nod to Bluesky's architecture. The
| long-term solution is--well it involves a lot of relay hint
| dropping and a reliance on Japanese levels of acuity when it
| comes to picking up on hints (among clinets). But (a) it's
| proving extreme slow going and (b) it only aims to mitigate the
| "global as relates to me" problem.
| ultra-boss wrote:
| Love reading these sorts of "technical problem + solution"
| pieces. The world does not need more content, in general, but it
| does need more of this kind of quality information sharing.
| robbale wrote:
| the use of fan-out to followers and a lazy fetch of popular
| followed accounts when follower's timeline is served a good
| implementations in hot reload scenarios
| KolmogorovComp wrote:
| A simpler option is to put a limit on the number of accounts
| one's can follow. Who needs to follow more than 4k followers if
| not bots?
| mpweiher wrote:
| On a related note, I am pretty confident that one of the main
| reasons the WWW succeeded where previous attempts failed was that
| it very specifically allowed 404s.
___________________________________________________________________
(page generated 2025-02-20 23:02 UTC)