https://github.com/maxwellpeterson/kafka-worker Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Blog * Solutions + By Plan + Enterprise + Teams + Compare all + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} maxwellpeterson / kafka-worker Public * Notifications * Fork 2 * Star 77 Kafka 0.8.0 broker implementation on top of Cloudflare Workers License MIT license 77 stars 2 forks Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights maxwellpeterson/kafka-worker This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 1 branch 0 tags Code * Clone HTTPS GitHub CLI [https://github.com/m] Use Git or checkout with SVN using the web URL. [gh repo clone maxwel] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @maxwellpeterson maxwellpeterson More documentation and code cleanup ... 784715d Sep 30, 2022 More documentation and code cleanup 784715d Git stats * 65 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows src test .eslintrc.cjs .gitignore .prettierrc.json Dockerfile LICENSE README.md build.js diagrams.excalidraw jest.config.js map.png package-lock.json package.json tsconfig.json wrangler.toml View code Kafka Worker Local Demo WebSocket Shim Design What about replication? What about leadership election? Map README.md Kafka Worker A Kafka 0.8.0 broker implementation on top of Cloudflare Workers and Durable Objects. This broker supports 4 client-facing APIs: 1. Produce API (Version: 0) 2. Fetch API (Version: 0) 3. ListOffsets API (Version: 0) 4. Metadata API (Version: 0) No other APIs (such as internal APIs used for administrative purposes) or API versions are supported. This project is a relatively simple proof of concept, and not intended to be used for anything serious. Local Demo For a fully local demo, no Cloudflare account required, check out the kafka-worker-demo project. WebSocket Shim The Kafka protocol uses TCP. Cloudflare Workers does not support plain TCP connections (at least until Socket Workers is available), but it does support WebSocket connections. To make this work, we need to frame Kafka protocol messages sent over TCP into WebSocket messages. The kafka-websocket-shim project implements this functionality, and must be used to connect to this broker. Design If you are unfamiliar with Kafka's design philosophy, I would recommend reading this article by Jay Kreps first. Kafka 0.8.0 dates all the way back to 2013, which was a much simpler era. There are only 2 types of state that we need to keep track of: records stored in partitions, and information about which topics and partitions exist. In this design, a Durable Object is created for each partition, and a single Durable Object is created for topic and partition information (this is referred to as the "global cluster DO"). Just like normal Kafka, writes to separate partitions are independent operations, and records within each partition are guaranteed to be read in the same order they were written. Each client connection is handled by a "gateway worker" that runs in the Cloudflare data center (also referred to as "colo") closest to the client's location. The gateway worker handles requests from the client, and makes internal subrequests to partition DOs and the global cluster DO as needed. Each DO contacted by the gateway worker may be located in the same data center as the gateway worker instance (the ideal case), or may be located somewhere else. What about replication? What about leadership election? There is none, at least not in the application code. The point of using Durable Objects here is that we can offload these complexities onto the infrastructure layer, and keep our application code focused and minimal. As described in the initial blog post, this is the serverless philosophy applied to persistent state. Replication and strong consistency mechanisms are implemented internally, and all we need to do is use the Durable Object API to reap these benefits. Building a Kafka broker without implementing replication and leadership election feels like cheating, but also makes this a much more tractable project. In this design, each deployment has one logical broker that spans Cloudflare's entire network. The hostname and port of this broker is the hostname and port of the gateway worker. Translated to the traditional Kafka model, this broker is the leader node for all partitions, and all partitions have zero replica nodes and zero ISR nodes. Again, there is still replication happening, it's just not visible to the broker implementation. Map kafka worker map About Kafka 0.8.0 broker implementation on top of Cloudflare Workers Topics kafka cloudflare-workers Resources Readme License MIT license Stars 77 stars Watchers 1 watching Forks 2 forks Releases No releases published Packages 1 Languages * TypeScript 97.9% * JavaScript 1.9% * Dockerfile 0.2% Footer (c) 2022 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.