https://github.com/unum-cloud/ujrpc Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Blog * Solutions + For + Enterprise + Teams + Startups + Education + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} unum-cloud / ujrpc Public * Notifications * Fork 3 * Star 319 Up to 100x Faster FastAPI. JSON-RPC with io_uring, SIMDJSON, and pure CPython bindings github.com/unum-cloud/ujrpc License Apache-2.0 license 319 stars 3 forks Star Notifications * Code * Issues 2 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights unum-cloud/ujrpc This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 3 branches 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/u] Use Git or checkout with SVN using the web URL. [gh repo clone unum-c] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @ashvardanian ashvardanian Merge pull request #17 from unum-cloud/dev ... 44ecfb4 Mar 6, 2023 Merge pull request #17 from unum-cloud/dev Dev 44ecfb4 Git stats * 298 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .devcontainer Make: add autopep8 February 10, 2023 08:03 .github/workflows Make: Switch build branch and bump version March 3, 2023 09:11 .vscode Add: Long package description March 3, 2023 09:08 docs Make: Docs build March 5, 2023 13:06 examples Merge branch 'dev' of https://github.com/unum-cloud/ujrpc into dev March 6, 2023 19:01 include/ujrpc Docs: Added a C example March 6, 2023 06:38 src Fix: make server callable March 6, 2023 13:36 .clang-format Docs: Clang-format, and highlight settings January 17, 2023 22:43 .gitignore Make: Docs build March 5, 2023 13:06 CMakeLists.txt Make: Allow to make a build of ujrpc_server_uring on any linux system March 6, 2023 18:26 LICENSE Initial commit January 3, 2023 14:23 README.md Merge branch 'dev' of https://github.com/unum-cloud/ujrpc into dev March 6, 2023 06:38 VERSION Make: Bump version March 6, 2023 19:04 pyproject.toml Make: Linux x86 and Arm March 2, 2023 08:39 setup.py Add: Long package description March 3, 2023 09:08 View code Uninterrupted JSON RPC Remote Procedure Calls Up to 100x Faster than FastAPI How is that possible?! Free Tier Throughput Quick Start Roadmap Why JSON-RPC? README.md Uninterrupted JSON RPC Remote Procedure Calls Up to 100x Faster than FastAPI Discord LinkedIn Twitter Blog GitHub --------------------------------------------------------------------- Most modern networking is built either on slow and ambiguous REST APIs or unnecessarily complex gRPC. FastAPI, for example, looks very approachable. We aim to be equally or even simpler to use. FastAPI UJRPC pip install fastapi uvicorn pip install ujrpc from fastapi import FastAPI from ujrpc.posix import Server import uvicorn # from ujrpc.uring import Server on 5.19+ server = FastAPI() server = Server() @server.get('/sum') @server def sum(a: int, b: int): def sum(a: int, b: int): return a + b return a + b uvicorn.run(...) server.run() It takes over a millisecond to handle a trivial FastAPI call on a recent 8-core CPU. In that time, light could have traveled 300 km through optics to the neighboring city or country, in my case. How does UJRPC compare to FastAPI and gRPC? Setup Server Latency w 1 Throughput w 32 client clients Fast API over REST 1'203 ms 3'184 rps Fast API over 86 ms 11'356 rps 1 WebSocket gRPC 2 164 ms 9'849 rps UJRPC with POSIX C 62 ms 79'000 rps UJRPC with io_uring 23 ms 43'000 rps UJRPC with io_uring C 22 ms 231'000 rps Table legend All benchmarks were conducted on AWS on general purpose instances with Ubuntu 22.10 AMI. It is the first major AMI to come with Linux Kernel 5.19, featuring much wider io_uring support for networking operations. These specific numbers were obtained on c7g.metal beefy instances with Graviton 3 chips. * The column marks, if the TCP/IP connection is being reused during subsequent requests. * The "server" column defines the programming language, in which the server was implemented. * The "latency" column report the amount of time between sending a request and receiving a response. m stands for micro, ms subsequently means microseconds. * The "throughput" column reports the number of Requests Per Second when querying the same server application from multiple client processes running on the same machine. 1 FastAPI couldn't process concurrent requests with WebSockets. 2 We tried generating C++ backends with gRPC, but its numbers, suspiciously, weren't better. There is also an async gRPC option, that wasn't tried. How is that possible?! How can a tiny pet-project with just a couple thousand lines of code compete with two of the most established networking libraries? UJRPC stands on the shoulders of Giants: * io_uring for interrupt-less IO. + io_uring_prep_read_fixed on 5.1+. + io_uring_prep_accept_direct on 5.19+. + io_uring_register_files_sparse on 5.19+. + IORING_SETUP_COOP_TASKRUN optional on 5.19+. + IORING_SETUP_SINGLE_ISSUER optional on 6.0+. * SIMD-accelerated parsers with manual memory control. + simdjson to parse JSON faster than gRPC can unpack ProtoBuf. + Turbo-Base64 to decode binary values from a Base64 form. + picohttpparser to navigate HTTP headers. You have already seen the latency of the round trip..., the throughput in requests per second..., want to see the bandwidth? Try yourself! @server def echo(data: bytes): return data Free Tier Throughput We will leave bandwidth measurements to enthusiasts, but will share some more numbers. The general logic is that you can't squeeze high performance from Free-Tier machines. Currently AWS provides following options: t2.micro and t4g.small, on older Intel and newer Graviton 2 chips. This library is so fast, that it doesn't need more than 1 core, so you can run a fast server even on a tiny Free-Tier server! Setup Server Clients t2.micro t4g.small Fast API over REST 1 328 rps 424 rps Fast API over WebSocket 1 1'504 rps 3'051 rps gRPC 1 1'169 rps 1'974 rps UJRPC with POSIX C 1 1'082 rps 2'438 rps UJRPC with io_uring C 1 - 5'864 rps UJRPC with POSIX C 32 3'399 rps 39'877 rps UJRPC with io_uring C 32 - 88'455 rps In this case, every server was bombarded by requests from 1 or a fleet of 32 other instances in the same availability zone. If you want to reproduce those benchmarks, check out the sum examples on GitHub. Quick Start For Python: pip install ujrpc For CMake projects: include(FetchContent) FetchContent_Declare( ujrpc GIT_REPOSITORY https://github.com/unum-cloud/ujrpc GIT_SHALLOW TRUE ) FetchContent_MakeAvailable(ujrpc) include_directories(${ujrpc_SOURCE_DIR}/include) The C usage example is mouthful compared to Python. We wanted to make it as lightweight as possible and to allow optional arguments without dynamic allocations and named lookups. So unlike the Python layer, we expect the user to manually extract the arguments from the call context with ujrpc_param_named_i64(), and its siblings. #include #include static void sum(ujrpc_call_t call, ujrpc_callback_tag_t) { int64_t a{}, b{}; char printed_sum[256]{}; bool got_a = ujrpc_param_named_i64(call, "a", 0, &a); bool got_b = ujrpc_param_named_i64(call, "b", 0, &b); if (!got_a || !got_b) return ujrpc_call_reply_error_invalid_params(call); int len = snprintf(printed_sum, 256, "%ll", a + b); ujrpc_call_reply_content(call, printed_sum, len); } int main(int argc, char** argv) { ujrpc_server_t server{}; ujrpc_config_t config{}; ujrpc_init(&config, &server); ujrpc_add_procedure(server, "sum", &sum, NULL); ujrpc_take_calls(server, 0); ujrpc_free(server); return 0; } Roadmap * [*] Batch Requests * [*] JSON-RPC over raw TCP sockets * [*] JSON-RPC over TCP with HTTP * [*] Concurrent sessions * [ ] Numpy array serialization * [ ] HTTPS support * [ ] Batch-capable endpoints for ML * [ ] Zero-ETL relay calls * [ ] Integrating with UKV * [ ] WebSockets for web interfaces * [ ] AF_XDP and UDP-based analogs on Linux Want to affect the roadmap and request a feature? Join the discussions on Discord. Why JSON-RPC? * Transport independent: UDP, TCP, bring what you want. * Application layer is optional: use HTTP or not. * Unlike REST APIs, there is just one way to pass arguments. About Up to 100x Faster FastAPI. JSON-RPC with io_uring, SIMDJSON, and pure CPython bindings github.com/unum-cloud/ujrpc Topics python http flask json tcp backend dpdk linux-kernel json-rpc rpc cpython simd http-server epoll rpc-framework tcp-ip simdjson io-uring fast-api liburing Resources Readme License Apache-2.0 license Stars 319 stars Watchers 6 watching Forks 3 forks Contributors 6 * @ashvardanian * @ishkhan42 * @DarvinHarutyunyan * @Ngalstyan4 * @gurgenyegoryan * @Arman-Ghazaryan Languages * C++ 71.1% * C 19.8% * CMake 5.1% * Python 3.3% * Dockerfile 0.7% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.