https://github.com/redplanetlabs/twitter-scale-mastodon Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Resources + Customer Stories + White papers, Ebooks, Webinars + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. {{ message }} redplanetlabs / twitter-scale-mastodon Public * Notifications * Fork 1 * Star 46 Twitter-scale Mastodon implementation in only 10k lines of code License Apache-2.0 license 46 stars 1 fork Activity Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights redplanetlabs/twitter-scale-mastodon This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. master Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 1 branch 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/r] Use Git or checkout with SVN using the web URL. [gh repo clone redpla] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @nathanmarz nathanmarz initial commit ... f86abdb Aug 27, 2023 initial commit f86abdb Git stats * 1 commit Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time api initial commit August 27, 2023 08:07 backend initial commit August 27, 2023 08:07 images initial commit August 27, 2023 08:07 .gitignore initial commit August 27, 2023 08:07 LICENSE initial commit August 27, 2023 08:07 README.md initial commit August 27, 2023 08:07 View code twitter-scale-mastodon Implementation overview Running locally Running on a real cluster README.md twitter-scale-mastodon Home timeline This project is a production-ready implementation of a Mastodon instance that scales to Twitter-scale (500M users, 700 average fanout, unbalanced social graph, 7k posts / second). It's only 10k lines of code and can easily be adapted into any Twitter-like social network. It shares no code with the official Mastodon implementation. The implementation is built on top of Rama, a new programming platform that enables end-to-end scalable backends to be built in their entirety in 100x less code than otherwise. Rama is a generic platform for building any application backend, not just social networks. This project is a demonstration of Rama's power. We will be starting a private beta for Rama soon, and a build of Rama is freely available to try now. We ran this code on a public instance available for anyone to use from August 15th, 2023 to August 24th, 2023. To demonstrate its scale, we ran the instance with 100M bots posting 3,500 times per second at 403 average fanout. The social graph was extremely unbalanced, with the largest user having 22M followers. During its lifetime the instance processed 2.66B posts and made 1.07T timeline deliveries. Here's a chart showing the scalability of the implementation: Scalability You can read a deep-dive exploration of how this instance is implemented in this blog post. That post also includes other performance charts. Implementation overview The implementation is structured like this: Structure The Mastodon backend is implemented as Rama modules, which handle all the data processing, data indexing, and most of the product logic. On top of that is the implementation of the Mastodon API using Spring/ Reactor. For the most part, the API implementation just handles HTTP requests with simple calls to the Rama modules and serves responses as JSON. S3 is used only for serving pictures and videos. Though we could serve those from Rama, static content like that is better served via a CDN. So we chose to use S3 to mimic that sort of architecture. All other storage is handled by the Rama modules. The backend is implemented in the backend/ folder and contains six modules: * Relationships: Implements the social graph, follow suggestions, and other relationships between entities. * Core: Implements timelines, profiles, fanout, boosts, favorites, conversations, replies, and the other core parts of the product. * Notifications: Implements notification timelines, including favorites/boosts, mentions, poll completion, new followers, and others. * TrendsAndHashtags: Implements trending statuses, links, and hashtags, as well as hashtag timelines and other hashtag-oriented features. * GlobalTimelines: Implements local and federated timelines of all statuses. * Search: Implements profile, status, and hashtag search, as well as profile directories. The API server is implemented in the api/ folder. Federation is implemented in both the API server and the Core module. Running locally Running the instance locally is very easy. We've set up the API server to also be able run an in process cluster with all the modules launched. All you have to do is run this from the api/ folder: mvn spring-boot:run Since this doesn't configuring credentials to connect to S3, you'll see an error about that. That error is ignored and the instance will continue launching. If you look at MastodonApiApplication, you can see how it also adds some data to the modules. These include users "alice", "bob", and "charlie" with passwords "alice", "bob", and "charlie" respectively. To run the frontend, you'll need to run our fork of Soapbox. Our fork contains minor tweaks to the official implementation to fix a variety of small issues. From the root of that fork run: NODE_ENV="development" BACKEND_URL="http://localhost:8080" yarn build NODE_ENV="development" BACKEND_URL="http://localhost:8080" yarn dev --port 8000 Once everything finishes launching, you can play with the instance at http://localhost:8000. Note that the instance can be run locally on macOS or Linux, but it doesn't currently work on Windows. Running on a real cluster Until Rama is publicly available, the only way to get access to the full Rama release is by applying to our private beta. About Twitter-scale Mastodon implementation in only 10k lines of code Resources Readme License Apache-2.0 license Activity Stars 46 stars Watchers 1 watching Forks 1 fork Report repository Releases No releases published Packages 0 No packages published Languages * Java 53.8% * CSS 44.7% * Other 1.5% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.