https://www.infoq.com/articles/clojure-donkey-http-stack/ BT Toggle Navigation Facilitating the spread of knowledge and innovation in professional software development More * About InfoQ * InfoQ Editors * Contribute * About C4Media * Media Kit Contribute Search [ ] [Search] Sign Up / Login Email [ ] Password [ ] [Login] Forgot password ? InfoQ Account Email [ ] [Send Recovery Email] Back to login Resend Activation [ ] [Resend] Back to login Login with: Google Microsoft Twitter Facebook Don't have an InfoQ account? Sign Up Notifications 1 Login to unlock InfoQ's new features * Stay up to date and get notified * Like your favorite content * Follow your favorite editors and peers Sign Up / Login Don't have an account? Register Here Logo - Back to homepage * En * Zhong Wen * Ri Ben * Fr * Br 2,454,378 Jan unique visitors News Articles Presentations Podcasts Guides Topics Development * Java * Kotlin * .Net * C# * Swift * Go * Rust * JavaScript Featured in Development * The Kollected Kode Vicious Review and Author Q&A Addison Wesley Professional The Kollected Kode Vicious by George V. Neville-Neil aims to provide thoughtful and pragmatic insight into programming to both experienced and younger software professionals on a variety of different topics related to programming. InfoQ has taken the chance to speak with author Neville-Neil about his book. The Kollected Kode Vicious Review and Author Q&A All in development Architecture & Design * Architecture * Enterprise Architecture * Scalability/Performance * Design * Case Studies * Microservices * Service Mesh * Patterns * Security Featured in Architecture & Design * Low-Code Platforms and the Rise of the Community Developer: Lots of Solutions, or Lots of Problems? Low-code platforms are the hottest enterprise software category right now. With the current level of investment, it is hard to imagine a future that doesn't feature lots of bespoke business apps being built by non-IT staff for use by their teams. Visibility of low code solutions is the key to managing risk. Low-Code Platforms and the Rise of the Community Developer: Lots of Solutions, or Lots of Problems? All in architecture-design AI, ML and Data Engineering * Big Data * Machine Learning * NoSQL * Database * Data Analytics * Streaming Featured in AI, ML & Data Engineering * AI Applied in Enterprises: Information Architecture, Decision Optimization, and Operationalization The book Deploying AI in the Enterprise by Eberhard Hechler, Martin Oberhofer, and Thomas Schaeck gives insight into the current state of AI related to themes like change management, DevOps, risk management, blockchain, and information governance. It discusses the possibilities, limitations, and challenges of AI and provides cases that show how AI is being applied. AI Applied in Enterprises: Information Architecture, Decision Optimization, and Operationalization All in ai-ml-data-eng Culture & Methods * Agile * Diversity * Leadership * Lean/Kanban * Personal Growth * Scrum * Sociocracy * Software Craftmanship * Team Collaboration * Testing * UX Featured in Culture & Methods * Cynefin Applied: Adapting to Changing Contexts The book Cynefin: Weaving Sense-Making into the Fabric of our World by Dave Snowden describes the Cynefin framework and explores how it has developed over the years. It also provides stories where people who have applied Cynefin share their experiences. Cynefin Applied: Adapting to Changing Contexts All in culture-methods DevOps * Infrastructure * Continuous Delivery * Automation * Containers * Cloud * Observability Featured in DevOps * Leading Technical Projects - and How to Get Them Done Sarah Wells shares stories on how the Operations and Reliability team at the Financial Times built tools that are used by lots of their development teams: the challenges they faced, the things they tried and what worked for them. Leading Technical Projects - and How to Get Them Done All in devops EventsNew Helpful links * About InfoQ * InfoQ Editors * Contribute * About C4Media QCons from around the world Upcoming Events * InfoQ LiveFeb 16, 2021 * QCon PlusMay 17-28, 2021 Choose your language * En * Zhong Wen * Ri Ben * Fr * Br Featured: More * Streaming * Machine Learning * Reactive * Microservices * Containers * Observability * Service Mesh QCon Plus biggest saving before Feb 6th InfoQ Homepage Articles Donkey: a Highly-Performant HTTP Stack for Clojure Development Safe and Sane: Deployment and Launch with Reduced Risks (FEB 11th Webinar) - Save Your Seat Donkey: a Highly-Performant HTTP Stack for Clojure Like Print Bookmarks Jan 12, 2021 9 min read by * Yaron Elyashiv reviewed by * Sergio De Simone Key Takeaways * Donkey is a new Clojure open-source HTTP library, designed for ease of use and performance. * Identifying the strengths and weaknesses of a language helps with design decisions. * Clojure frees developers from the perils of writing concurrent programs, but at a price. * When concurrency is not a factor, consider using Java. * Experimenting, iterating, and frequently measuring is key to achieving optimal results. Standing on the shoulders of giants, Donkey is the product of the quest for a highly performant HTTP stack aimed to: (1) scale at the rapid pace of growth we have been experiencing at AppsFlyer, and (2) save us computing costs. Though our implementation is still in its early stages, with limited production mileage, we've seen such impressive numbers in our benchmark testing that we couldn't keep it to ourselves. Donkey highlights: * Built on Vert.x - a mature, battle-tested, highly performant toolkit for building reactive applications on the JVM. * Simple API. * Works as a drop-in replacement for any Ring-compliant server. * Outperforms the library we've been using at AppsFlyer. * Open-source. Related Sponsored Content * Best Practices for Creating Secure Applications Related Sponsor [Outsystems] What are today's hot technology trends? Join our upcoming Tech Talks to learn how you can quickly create applications that evolve as business needs and technology trends change. In this article, we'll briefly outline AppsFlyer's use-case and our growing need for a library like Donkey. Then we'll dig deeper into the benchmarks we tested to get a sense of Donkey's performance. Finally, we will discuss Clojure and immutability, Netty, and thread locality, and explain some of the design decisions we took. The Web Handler AppsFlyer is the market leader in real-time mobile attribution SaaS solutions. This means we handle a lot of traffic - more than 100 billion events per day, to be precise. Managing this in real-time means we have to process every one of those events - fast. I joined AppsFlyer to form a new team that would take ownership of the attribution flow's front line. The team is responsible for handling this massive traffic, both efficiently and in a cost-effective manner. The "Web Handler" is a Clojure service that handles all events coming from the AppsFlyer SDK, installed on more than 90% of mobile devices worldwide. [42image001] It uses a well-known Ring-compliant HTTP server and client library. Ring is the de-facto standard for HTTP servers in the Clojure universe. We have experienced exponential growth in traffic over the last few years, and the solution that once served our needs was just not cutting it anymore. We decided it was time to explore other solutions that would scale better as we continued to grow while cutting our AWS EC2 cost at the same time. Always Measure Throughout the process, we used TechEmpower to benchmark and compare different implementations. TechEmpower lists performance comparisons of an impressive number of web framework benchmarks in every language imaginable. Benchmark setup: * Hardware 2.3 GHz 8-Core Intel Core i9 MacBook Pro with 16 GB 2400 MHz DDR4 SDRAM * Software + Plaintext and JSON serialization benchmarks (from TechEmpower ) + reitit for routing + jsonista for JSON serialization Indicators: * Our key indicators were throughput and latency * The main focus was on the relative performance of each library rather than the absolute values, due to large variances. Therefore, many measurements were taken and averaged to eliminate anomalies to the greatest extent possible. Our current library's benchmark results were used as the baseline for our comparisons. Although we've seen it reach up to ~100K and ~50K requests per second (rps) in the plaintext and JSON benchmarks, the vast majority of the time it demonstrated performance in this neighborhood: Benchmark Requests/ Transfer/ Latency Total sec sec (99%) requests Plaintext 61,359 7.78MB 627.07ms 924,929 JSON 43,327 6.94MB 24.12ms 654,233 serialization TechEmpower benchmark - 15-second run First Iteration We planned to use Netty as the server implementation and wrap it in a Clojure API. The concern was how a Clojure interface would impact server performance, given the thread-safety model used by Clojure: immutability. Immutability Clojure makes writing concurrent applications easy. It frees the developer from the implications of sharing state between threads. It does so by using immutable data structures, as described by Rich Hickey in his talk The Value of Values: If you have a value, if you have an immutable thing, can you give that to somebody else and not worry? Yes, you don't have to worry. Do they have to worry about you now because they both now refer to the same value? Anybody have to worry? No. Values can be shared. Because all objects are immutable, they can be concurrently accessed from multiple threads without fear of lock contention, race conditions, proper synchronization, and all the other "fun" stuff that makes writing concurrent programs so difficult to get right. The downside is that every mutating operation produces a new object with an updated state. An inefficient implementation would cause a great deal of CPU time to be wasted on copying and creating new objects and, as a result, longer and more frequent GC cycles. Fortunately, Clojure uses a Hash Array Mapped Trie (HAMT) to model its data structures internally. By sharing structures that do not change, and copying only what does, it maintains immutability and thread-safety - and does so at a minimal cost. Another factor that can hinder performance is the wide use of the volatile fields and atomic references that are still required to maintain the Java Memory Model semantics of "happens before." The dynamic runtime of the language and features like laziness are just some of the elements that require synchronization within the JVM. Thread-locality Netty uses a different approach to thread safety: an event-driven model, with the concept of an event loop at its core. Users write handlers that are notified of incoming requests and outgoing responses. Both the incoming and outgoing handlers will always be called on the same event loop and, therefore, the same thread. As long as the handlers don't create threads that mutate their state, the state is not shared and there is no need for synchronization. [35image003] Netty Event Loop Minimal Benchmark To assess the overhead associated with Clojure, we used the JSON serialization test case. For the sake of simplicity, we used Vert.x rather than Netty directly, and we wrote the minimal amount of code to return a response, as shown below. To emphasize the overhead added when mutating objects, we used a slightly contrived way of returning a response using assoc. (fn [^RoutingContext _ctx] (assoc {} :headers {"Content-Type" "application/json"} :status 200 :body (json/write-value-as-string {"message" "Hello, World!"})))) Clojure request handler Both implementations use Jackson for serialization, so the real difference is in the relative overhead of Clojure over Java. In our benchmarks, the Clojure version underperformed by about 9-13% when comparing peak throughput. Implementation Requests/sec Transfer/sec Latency (99%) Total requests Vert.x 252,825 37.13MB 18.86ms 3,817,088 Vert.x + 226,643 32.42MB 25.63ms 3,420,219 Clojure TechEmpower - JSON serialization - 15-second run We did not see the same degradation with lower-rate throughput. Nevertheless, the comparison shows that at rates reaching the server's capacity, there is a significant decrease in performance. The next round of tests set to gauge Clojure overhead, compared different middleware implementations. The following table shows the results of benchmarking "off-the-shelf" middleware written in Clojure from ring-clojure, versus the same Ring-compliant middleware implemented internally in Java. Benchmarks were performed using Criterium. Middleware Ring (Clojure) Clojure + Java Latency Decrease Parsing query 7.943209 us 1.024412 us 87.1 % parameters Parsing JSON body 31.706660 us 21.677851 us 31.63 % Serializing JSON body 36.464090 us 12.789686 us 64.93 % Our tests showed a clear decrease in throughput and an increase in latency at full capacity. These observations led us to a two-layer design similar to Linux's user and kernel space. The Clojure layer (user space) will be limited to the API exposed to the user, whereas the Java layer (kernel space) will handle the integration with Netty. Results of the POC (aka "Green Yeti") The results from our proof of concept were reassuring. We saw a 350-400% increase in throughput and a 50% decrease in latency: Benchmark Requests/ Transfer/ Latency Total sec sec (99%) requests Plaintext 242,859 43.54MB 311.41ms 3,665,662 JSON 149,663 29.69MB 13.40ms 2,259,769 Serialization TechEmpower benchmark - 15-second run To see how the design fared in a more realistic scenario, we wrote a small Clojure service and benchmarked it using wrk2. As expected, throughput was significantly lower than the previous benchmarks at 75-80k rps and 48-171 ms latency, but promising. [30image005] Next Iteration - Donkey The scope of the project began to expand and include more and more elements requiring implementation. For the next iteration, we replaced Netty with Vert.x, giving us a higher level of abstraction with fairly low overhead. We maintained the same core design as in the POC and extended it with: * Minimalistic API: We kept the API very simple and closely resembling the current one, to facilitate easy migration. * Extensible design: The design allows for implementations other than Ring to be added in the future. * Integration with routing libraries: The library exposes a routing API. However, support for integration with existing routing libraries was added, to facilitate easy migration. * Vert.x encapsulation: There is full encapsulation of Vert.x as an implementation detail. This allows us to replace it in the future if we want to and gives us control over the functionality we want to expose. * Non-blocking behavior by default: The API promotes writing non-blocking application code. Support was added for legacy services that block, to facilitate easy migration. End Results The results were better than we had hoped. We ran the TechEmpower plaintext and JSON serialization on all the Clojure libraries featured in TechEmpower. Donkey had the highest throughput (with 811K rps) in the plaintext benchmark and the second-highest (with 190K rps) in the JSON serialization. [21image007] Plaintext [15image009] JSON Serialization Our design is effective because it leverages the relative strengths of its components in different places. As engineers, we have many tools at our disposal. Whether that be a choice of language, framework, protocol, or design pattern, it is important to select the right tool for the job. We started by selecting a mature and proven framework with the right amount of abstraction to build on. We recognized the different concurrency models used by Netty and Vert.x as opposed to Clojure and were not shy to use Java where concurrency wasn't a factor and we could save on the overhead associated with immutability and synchronization. Conclusion Benchmarking is not an exact science. Many elements beyond our control play a part and affect the results. The highest throughput we saw from Donkey was 996K and 207K rps in the plaintext and JSON serialization benchmarks, respectively. There is no doubt we won't see these kinds of numbers in a real production setting. However, we demonstrated the importance of understanding the internals of the language, recognizing its relative strengths and weaknesses, and (last but not least) taking the time to conduct experiments to achieve the best results. Our road map includes plans to integrate Donkey in all AppsFlyer services that expose a REST API, add support for HTTP2, schema validation, OpenAPI3 integration, JWT validation, health checks, and more. Donkey is now generally available on clojars.org. Pull requests are welcome. About the Author [1Yaron]Yaron Elyashiv spent the last decade working as a software engineer in the US and Israel. He started out as a Jazz musician graduating from the City College of New York and playing the saxophone at Jazz clubs in New York City. It didn't take long before he caught the coding bug and never looked back. Currently, he is working at AppsFlyer on large scale engineering challenges. His interests include JVM, networking, performance optimization, open-source projects, and writing clean code. Rate this Article Adoption ( ) ( ) ( ) ( ) ( ) Style ( ) ( ) ( ) ( ) ( ) [ ] [Submit] [ ] [Submit] [ ] Author Contacted This content is in the Java topic Related Topics: * Development * Architecture & Design * Clojure * Runtimes * Performance & Scalability * LISP * Java * JVM * W3C * Functional Programming * Networking * Architecture * Infrastructure * HTTP * JVM Languages * Virtual Machines * Related Editorial * Popular across InfoQ + Google Cloud Workflows Now Generally Available + Google Releases Eventarc into General Availability + GraalVM 21.0 Introduces a JVM Written in Java + Elastic Changes Licences for Elasticsearch and Kibana: AWS Forks Both + Performance Analysis for Arm vs x86 CPUs in the Cloud + Melissa Benua on Continuous Delivery, Quality, and DevTestSecOps Related Content * Java InfoQ Trends Report--September 2020 Java InfoQ Trends Report--September 2020 * GraalVM 21.0 Introduces a JVM Written in Java * How the HotSpot and Graal JVMs Execute Java Code How the HotSpot and Graal JVMs Execute Java Code * A Year with Java 11 in Production! A Year with Java 11 in Production! * IntelliJ IDEA 2021.1 EAP 1 Supports Java 16 * Grails Foundation Established to Advance Adoption of Grails Framework * Java 1.0 Turns 25 * Enhanced Streams Processing with Kotlin's Sequence Interface Enhanced Streams Processing with Kotlin's Sequence Interface * Red Hat Releases OptaPlanner 8 * Java Feature Spotlight: Pattern Matching Java Feature Spotlight: Pattern Matching * AdoptOpenJDK Welcomes Dragonwell * Azure + Spring Boot = Serverless - Q&A with Julien Dubois Azure + Spring Boot = Serverless - Q&A with Julien Dubois * Maximizing Applications Performance with GraalVM Maximizing Applications Performance with GraalVM * TornadoVM: Java for GPUs and FPGAs TornadoVM: Java for GPUs and FPGAs * Running Axon Server in Docker and Kubernetes Running Axon Server in Docker and Kubernetes * The Kollected Kode Vicious Review and Author Q&A The Kollected Kode Vicious Review and Author Q&A * What's New on F#: Q&A With Phillip Carter What's New on F#: Q&A With Phillip Carter * Instrumenting the Network for Successful AIOps Instrumenting the Network for Successful AIOps * Functional UI - a Stream-Based Equational Approach Functional UI - a Stream-Based Equational Approach * Operating Pivotal Application Service at Scale Operating Pivotal Application Service at Scale * Cynefin Applied: Adapting to Changing Contexts Cynefin Applied: Adapting to Changing Contexts * Npm 7 Now Generally Available, Supports Workspaces and Deterministic Builds * OpenAI Announces GPT-3 Model for Image Generation * Low-Code Platforms and the Rise of the Community Developer: Lots of Solutions, or Lots of Problems? Low-Code Platforms and the Rise of the Community Developer: Lots of Solutions, or Lots of Problems? * Deno Now Compiles to Self-Contained, Standalone Binaries * AI No Silver Bullet for Cloud Security, but Here's How It Can Help AI No Silver Bullet for Cloud Security, but Here's How It Can Help * AWS Announces Enhanced Console Experience and New v2 APIs for Amazon Lex * Newest TeamTNT IRC Bot Steals AWS and Docker Credentials * AWS Step Functions Gains Synchronous Express Workflows * Docker Engine 20.10 Released: Supports cgroups v2 and Dual Logging * Building Latency Sensitive User Facing Analytics via Apache Pinot Building Latency Sensitive User Facing Analytics via Apache Pinot * Building Tech at Presidential Scale Building Tech at Presidential Scale * AI Applied in Enterprises: Information Architecture, Decision Optimization, and Operationalization AI Applied in Enterprises: Information Architecture, Decision Optimization, and Operationalization * Google Releases Eventarc into General Availability * The Myth and Reality of a Flat Hierarchy and an Agile Transformation The Myth and Reality of a Flat Hierarchy and an Agile Transformation * Airbnb: Using Guardrails to Identify Changes with Negative Impact across Teams * Google Cloud Workflows Now Generally Available * Docker Hub and JFrog Partnership Removes Image Pull Limits for Artifactory Users * Using Language and Developer Friendly Data Structures with Couchbase * InfoQ Live: Practical Ways to Integrate Observability into Your Distributed System Architecture The InfoQ Newsletter A round-up of last week's content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example Enter your e-mail address [ ] [ ] Select your country [Select a country] [ ] I consent to InfoQ.com handling my data as explained in this Privacy Notice. [Subscribe] We protect your privacy. Hello stranger! You need to Register an InfoQ account or Login or login to post comments. But there's so much more behind being registered. Get the most out of the InfoQ experience. Tell us what you think [ ] [ ] Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p [ ] Email me replies to any of my messages in this thread [Post Message] Community comments Watch thread Close Your Reply Quote original message [ ] [ ] Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p [ ] Email me replies to any of my messages in this thread [Post Message] Cancel Close Your Reply [ ] [ ] Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p [ ] Email me replies to any of my messages in this thread [Post Message] Cancel Close OK Discuss * Development + Airbnb: Using Guardrails to Identify Changes with Negative Impact across Teams + Porting to Apple Silicon: Firefox Case + The "Wasmer" WebAssembly Runtime is Generally Available * Architecture & Design + Low-Code Platforms and the Rise of the Community Developer: Lots of Solutions, or Lots of Problems? + Service Meshes and Linkerd with William Morgan + Building Latency Sensitive User Facing Analytics via Apache Pinot * Culture & Methods + Cynefin Applied: Adapting to Changing Contexts + The Myth and Reality of a Flat Hierarchy and an Agile Transformation + Renee Troughton on Deep Introspection, Integrated Organisations and Looking beyond Agile * AI, ML & Data Engineering + OpenAI Announces GPT-3 Model for Image Generation + AI Applied in Enterprises: Information Architecture, Decision Optimization, and Operationalization + Using Language and Developer Friendly Data Structures with Couchbase * DevOps + Newest TeamTNT IRC Bot Steals AWS and Docker Credentials + Docker Engine 20.10 Released: Supports cgroups v2 and Dual Logging + Docker Hub and JFrog Partnership Removes Image Pull Limits for Artifactory Users The InfoQ Newsletter A round-up of last week's content on InfoQ sent out every Tuesday. Join a community of over 250,000 senior developers. View an example * Get a quick overview of content published on a variety of innovator and early adopter technologies * Learn what you don't know that you don't know * Stay up to date with the latest information from the topics you are interested in Enter your e-mail address [ ] [ ] Select your country [Select a country] [ ] I consent to InfoQ.com handling my data as explained in this Privacy Notice. [Subscribe] We protect your privacy. QCon Events Registration Open Learn the trends, best practices and solutions applied by the world's most innovative software practitioners to help you validate your software roadmap. Join us for an online experience for senior software engineers and architects spaced over 2 weeks. May 17-28, 2021 Home Create account Login QCon Conferences Events Contribute InfoQ Editors About InfoQ About C4Media Media Kit Events * InfoQ Live Feb 16th, 2021 * QCon Plus May 17-28, 2021 Follow us on Youtube187K Followers Linkedin16.1K Followers RSS19K Readers Twitter48k Followers Facebook20K Likes AlexaNew Stay in the know The InfoQ Podcast[infoq-podc] Engineering Culture Podcast[engineerin] The Software Architects' Newsletter[architects] General Feedback feedback@infoq.com Advertising sales@infoq.com Editorial editors@infoq.com Marketing marketing@infoq.com InfoQ.com and all content copyright (c) 2006-2021 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with. Privacy Notice, Terms And Conditions, Cookie Policy BT Is your profile up-to-date? Please take a moment to review and update. Email Address* [ ] Note: If updating/changing your email, a validation request will be sent Company name: [ ] Keep current company name Update Company name to:* [ ] Company role: [ ] Keep current company role Update company role to:* [ ] Company size: [ ] Keep current company Size Update company size to:* [ ] Country/Zone: [ ] Keep current country/zone Update country/zone to:* [--- Select a country --- ] State/Province/Region: [ ] Keep current state/province/region Update state/province/region to: [] [ ] Subscribe to our newsletter? [ ] Subscribe to our architect newsletter? [ ] Subscribe to our industry email notices? [ ] By subscribing to this email, we may send you content based on your previous topic interests. See our privacy notice for details. [ ] Subscribe to our Special Reports newsletter? [Submit] You will be sent an email to validate the new email address. This pop-up will close itself in a few moments. *