[HN Gopher] TechEmpower Framework Benchmarks: Round 20 (2021-02-08)
___________________________________________________________________
TechEmpower Framework Benchmarks: Round 20 (2021-02-08)
Author : pella
Score : 51 points
Date : 2021-02-08 18:29 UTC (4 hours ago)
(HTM) web link (www.techempower.com)
(TXT) w3m dump (www.techempower.com)
| hkarthik wrote:
| I see a lot less value in these benchmarks than I did 6 years ago
| or so when they first started appearing. Maybe I'm showing my age
| but the speed argument just isn't that important to me any more.
|
| These days when I'm evaluating a framework, I look for strong
| documentation, integration with tools for observability and
| deployment, and compatibility with a well understood runtime that
| makes operations easier.
|
| These things are way more important for building software that
| works, that scales, and is operationally efficient for a team of
| engineers to work with for several years.
|
| Everything else is just vanity metrics.
| Xevi wrote:
| Most of these benchmarks are insanely optimized. The code is
| rarely idiomatic. There's also a few frameworks in there that
| just wraps C libraries to get good scores.
|
| I wouldn't use TechEmpower as a reliable source for checking
| framework performance.
| gameswithgo wrote:
| Performance doesn't matter until it does. One thing that can be
| nice is that an efficient backend can handle an enormous amount
| of load on just 1 server, which can save you a ton a complexity
| for various reasons (deployment, integrations with other
| services, and just less cost). Then on the extreme end where
| you have a very high traffic server, efficiency might let you
| scale back from ~12vms to ~3, which can be a significant cost
| savings, especially if done with no impact in productivity,
| which I believe is often that case just by moving from
| interpreted/dynamic langs to statically typed compiled/jitted
| ones.
| fabian2k wrote:
| I think the biggest benefit of these benchmarks has been to
| trigger improvements to all the low-hanging fruit in various
| frameworks. The differences at the high end don't really matter
| for many people, but I find it quite comforting if a framework
| is fast enough that I won't run into any performance issues I
| didn't cause myself. Because those I can fix, but if I hit a
| bottleneck in the framework it gets so much harder to figure
| out a way around it.
| idoubtit wrote:
| Obviously, performance generally isn't the main criteria when
| choosing a framework. I would argue that your "software that
| scales" isn't important either in most cases.
|
| I don't care much about performance either, especially compared
| to documentation, but that does not mean these benchmarks are
| useless. For instance, comparing similar frameworks can show
| big differences which illustrate different software
| architectures. Among the Nginx/phpfpm/Mysql fullstack-with-ORM
| frameworks, Laravel has roughly 8% of the raw stack capacity,
| while the other mature frameworks can do 2x or 3x more. I
| suspect this reflects Laravel's abuse of magic (calls to non-
| existent methods are redirected at runtime to other objects
| created on-the-fly, and so on). This may not be the real cause,
| but such a poor performance makes me suspicious about the
| complexity or the quality.
| lstamour wrote:
| To me the interesting part is when you can read the benchmark
| source code and learn about a high performance feature for the
| framework or language you're already using. That's happened a
| few times for me with Go, for example. I hope more high
| performance enhancements are added even for frameworks that
| would appear near the bottom. That's what keeps this relevant
| to me. :) Well, that and knowing roughly what the performance
| ceiling is for raw socket HTTP in various languages ;-)
| blunte wrote:
| It would be very interesting to see an additional column of
| "prime examples" which link to the biggest or most heavily used
| web apps/apis built on a given stack.
|
| I suspect the examples would be few at the top of the list and
| many near the bottom.
| jsnk wrote:
| I was surpised to see javascript framework called, just-js show
| up at rank 9. In case, people are wondering, it looks like a new
| V8 runtime environment. https://github.com/just-js/just
| flockonus wrote:
| More info here: https://just.billywhizz.io/blog/on-javascript-
| performance-01...
| matt42 wrote:
| To me there is 3 main points: - Even if it's
| JS, just-js make almost no use of dynamic memory allocations.
| - The authors rewrote himself a postgresql driver that support
| batching request. - it wraps high performance c++
| libraries
|
| Even if the techempower implementation is far more verbose and
| complex than mainstream JS framework, the amount of work behind
| just-js and its preformances are just impressive!
| pella wrote:
| github repo ( if any framework is missing .. )
|
| - https://github.com/TechEmpower/FrameworkBenchmarks
| lstamour wrote:
| Note also, another reason to browse GitHub is to look at how
| the various implementations are written. For example, sometimes
| frameworks use fewer SQL statements to complete the same work,
| or they configure database libraries with different settings.
| I've noted a few instances where gaming these benchmarks is
| possible simply because a new framework implements the
| algorithm differently.
|
| Not to say they don't have value -- they do -- but keep in mind
| your code may look very different from benchmark code. I do
| think it's worth looking at the code to examine why some
| approaches scale better than others, though!
|
| Rarely is it the case that more code = faster performance, but
| sometimes features you need are sacrificed for extra
| performance even when a benchmark is classified as "realistic".
| Viewing source code is how you can determine which frameworks
| actually best fit your needs, rather than just which ones
| performed well in this test...
|
| For example, it's easy to think that raw Go performance is
| terrible compared to Rust or C++, but it's worth pointing out
| that the raw Go example uses commonly available standard
| libraries baked in to the language to accomplish almost
| everything it does. Most other examples at some level or
| another have to deal with epoll and concurrency manually
| themselves or via the web server replacements they use. Based
| on this, the overhead or raw implementation of socket
| management is often what is being benchmarked. Implementations
| like just-js use C++ and epoll under the hood to implement
| their server, like most of the other high performance examples
| at the top of the benchmarks.
|
| The question often then becomes, do you want to write your own
| HTTP server or take advantage of an API someone else has
| written that might have more features or middleware? And
| remember that configuration matters, I forget off the top of my
| head, but the Go examples with a new web server had more
| performance optimization than the raw-Go benchmark had.
| Obviously I should submit a PR to fix this, but my ultimate
| goal here is to say that reading the source code will both help
| you become a better developer but also catch the shortcuts and
| techniques used by high performance implementations...
| zinclozenge wrote:
| The original author of actix has a new framework called ntex.
| Looks like it was a fork of actix and then went from there. One
| of the differences I was able to spot was that it appears that it
| uses a (radix?) trie for the router.
| pella wrote:
| Some framework is using extreme optimization tricks, so be
| careful .. and check the source code! :)
|
| example:
|
| "-d:danger : Turns off all runtime checks and turns on the
| optimizer."
|
| https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
| capableweb wrote:
| > Some framework is using extreme optimization tricks, so be
| careful
|
| That accurately represents how I've been forced to run
| applications in production too, when under huge stress and
| we're trying to squeeze everything out of the existing
| hardware.
|
| Welcome to _Hacker_ News ;)
|
| Edit: And absolutely yes, check out the code behind it. Every
| benchmark has their flaws and biases so it's important to
| verify before making conclusions.
| open-source-ux wrote:
| Some random observations...
|
| _PHP_
|
| Impressed by PHP's performance - it makes an entrance at position
| 20. If you exclude Javascript, the next interpreted language is
| Lua at position 81, then Python all the way down at 206. Ruby
| makes its first appearance at 255.
|
| _Julia and Nim_
|
| These are both new, modern languages that tout their performance
| as a benefit. It's a shame they did not take part in the
| benchmarks.
|
| ***
|
| Regardless of what you think of the benchmarks, the rankings do
| affect people's perceptions of languages and frameworks (both
| positive and negative). For example, I don't use PHP, but these
| benchmarks tell me that PHP has leapfrogged over Python and Ruby
| in the performance stakes for web development. And not just by a
| small margin, but by a significant difference.
| mdasen wrote:
| I like the TechEmpower benchmarks, but you have to read the
| code to really understand the comparisons being made.
|
| In many languages, a bunch of web server code might end up
| being implemented in C and not the language itself. When you're
| creating a minimal endpoint for a benchmark, you might be
| exercising that C code and not the language itself. For
| example, the PHP implementation uses `htmlspecialchars` to
| encode the information which is implemented in C. That doesn't
| tell you much about the performance of PHP, but just that it
| has an optimized HTML escaper. The `asort` function used to
| sort the results is implemented in C and has more limited
| functionality compared to more general sorting functions that
| might be able to take lambdas in other languages. The PHP
| implementation even takes advantage of `PDO::FETCH_KEY_PAIR`
| which will only work if there are only two columns.
|
| Likewise, the PHP implementation doesn't use templates, but
| manually builds strings. The fastest Python implementation
| actually renders a Jinja2 template: https://github.com/TechEmpo
| wer/FrameworkBenchmarks/blob/mast.... That's much more
| realistic in terms of what you'd do in the real world, but
| you're going to be carrying the overhead of a real-world
| template system like Jinja2. Part of Python's failure here is
| that no one wanted to implement an optimized Python version
| that would just build a string instead of rendering a template.
|
| Changing the test constraints a bit would ruin a lot of the
| advantages that PHP used there. Let's say that you had to
| retrieve three columns: `id, sort_key, fortune_text` and sort
| on the `sort_key`. Now you need to read more information back
| rather than just being able to make it an associative array
| (hash map). You need to be able to sort based on that sort_key
| which means probably giving a sort call a lambda.
|
| This isn't limited to PHP. A bunch of Go implementations do
| things like allocating a pool of structs and then re-using the
| same structs to avoid the garbage collector. A lot of
| implementations create result arrays sized so that they won't
| need to be re-sized (creating additional allocations and
| additional GC work). The rules say this isn't allowed, but they
| do it anyway.
|
| So, before comparing tests, I'd look at the implementations to
| make sure that they're comparable. A Django implementation that
| actually returns objects and is rendering templates and looks
| like a canonical Django implementation is very different from
| an optimized PHP version trying to avoid running any PHP as
| much as possible. When we start looking at the popular PHP
| frameworks which will be executing a bit of PHP like
| CodeIgniter or Laravel, we start seeing performance similar to
| Python frameworks as the PHP code is doing similar things like
| rendering templates. It just happens that no one implemented a
| Python version that didn't use a fully-fledged template
| renderer.
|
| And this is the weird thing: the benchmarks changed your
| perception of PHP while comparing things that weren't similar.
| I think PHP is often faster than Python and Ruby, but probably
| not to the extent that your perception might be given these
| benchmarks.
|
| I actually find it fascinating to look at the implementations
| and see which communities care about realistic implementations
| vs. leveraging all sorts of tricks to win the benchmark.
| Xevi wrote:
| There are 3 Nim frameworks in the plaintext benchmark.
| Httpbeast, Jester and Prologue.
| tomcam wrote:
| Mad props to these guys for keeping this massive project going
| for so long. Seems like a massive amount of work only to get
| picked apart by my friends here on HN.
| bhauer wrote:
| Thanks, tomcam. It is a lot of work, but it's what we do in
| between our paying work. And we find it fascinating and fun.
|
| I think what really hits home are the stories about people
| upgrading elements at the foundation of their stack--their
| platform or framework--after those foundations have seen
| optimization efforts. It is rewarding to see application
| developers realize dramatic performance boosts within their
| applications with so little effort; we feel we have contributed
| to this delight in a small way. For example, check out this
| post by the Azure Active Directory team [1] where they saw
| massive performance improvements by upgrading from .NET
| Framework to .NET Core 3.1 (which isn't even the latest
| version).
|
| Some people focus too much on specific rank ordering and the
| optimization efforts made by those who jockey for top spots.
| It's better to consume the data in rough tiers of performance,
| however you choose to define those tiers. We tend to encourage
| people to consider performance as one part of the puzzle when
| selecting infrastructure software. Several high performance
| platforms and frameworks (in Java, Rust, C#, Go, Python, and
| more) are _also_ ergonomic and easy to work with. And using
| high performance infrastructure software means you can avoid
| premature optimization in your application code and avoid
| premature architectual complexity. You can enjoy the trifecta
| of architectural simplicity, low-latency, and good scale
| headroom.
|
| Obviously we know this project will always receive diverse and
| critical opinions. That's fine; hackers are a very opinionated
| people. For those who value the data, ourselves included, we
| are happy to keep putting the effort in.
|
| [1] https://devblogs.microsoft.com/dotnet/azure-active-
| directory...
| tomcam wrote:
| I can only imagine that while you must have been ready for an
| onslaught of criticism that you would have ended up with such
| a staggering workload so many years later. Literally
| unimaginable to me.
| billywhizz wrote:
| i think it is really interesting to see the changes in
| different languages and platforms over time and also very
| useful to the community to see the optimisations folks come
| up with. thanks for the continued work brian and team!
| jakearmitage wrote:
| The top PHP entry, "php-ngx-pgsql", uses a quite obscure way of
| running PHP and Nginx:
|
| https://github.com/rryqszq4/ngx_php7
|
| https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
|
| Very similar to OpenResty, but with PHP instead of Lua. Which is
| funny, because OpenResty ranks lower.
| lstamour wrote:
| I wrote this comment based on pre-release benchmarks, based on
| Physical benchmarks run against Git commits from Jan 23 - but it
| should still largely apply to the final benchmarks:
|
| It's kind of frustrating to spend a few hours investigating
| preview results of the latest techempower benchmarks just to come
| to the conclusion that (a) truly high performance requires tuning
| your code to linux, to the network interface cards, and a deep
| understanding of what your code is doing and (b) right now at the
| top of the benchmarks, common high performance Rust code is
| mostly "unsafe" memory-wise, and the same is true of C++ for
| obvious reasons. After that:
|
| * When looking at garbage collected languages with very minimal
| implementations (Jooby mostly), Java can come out on top but
| requires 4 Gigs of memory which is more than I would ever like to
| use.
|
| * After Java comes C# but performance slows down the second you
| want to do anything less optimized, like use third-party
| libraries.
|
| * Finally, we've Go, which in a surprising twist is basically
| neck and neck with some pure PHP server implementations, but once
| I include those, I might as well mention there's a JS
| implementation that skips the Node.js async event loop and offers
| performance on par with C++ and Rust, but heavily uses C++
| internally and is thus more "unsafe".
|
| What this tells me is kind of what I expected, implementation
| matters more than language... at this point, memory usage aside
| for Java, an efficient HTTP/1.1 server can be implemented in
| basically any language and when tuned or stripped down tends to
| run faster than the commonly used web servers, like Nginx,
| Tomcat, Express, etc. Often this means writing plain text RAW
| HTTP to a socket and managing sockets efficiently.
|
| Which brings us back to Go, though. Of all the implementations
| I've looked at so far, even the PHP one, only the Go benchmark is
| written like "standard Go" such that most people writing a
| service in Go will write something high performance without
| trying to make it high performance. Effectively, you don't need
| to ensure all your dependencies use special optimized code
| routines to get something relatively optimized working quickly in
| Go with low memory overhead unlike Java.
|
| I'm a bit shocked by this conclusion as I was really hoping that
| Rust's high performance use cases would win out, as it's true
| that Rust can get 3x faster than Go, but on the Rust side, both
| actix and ntex are too immature as neither's hit 1.0 yet, while
| tokio has hit 1.0 but its server, warp, is slower than Go.
|
| Irritating is that each of the techempower benchmarks use
| different implementations. For example, there's a benchmark
| that's supposed to measure 20 calls to Postgres, but one
| benchmark gets to the top by making only one long SQL statement
| that changes 20 rows. Another implementation uses pgx's Batch
| functionality to send multiple queries in batches (a big
| timesaver, but not technically standard libpq), but then the
| standard Go variant doesn't use Batching even when it could
| (which means we can't compare custom implementations to generic
| Go ones fairly):
| https://github.com/TechEmpower/FrameworkBenchmarks/search?l=...
| Twirrim wrote:
| > For example, there's a benchmark that's supposed to measure
| 20 calls to Postgres, but one benchmark gets to the top by
| making only one long SQL statement that changes 20 rows.
|
| That seems like a clear cheating case, over-optimised to the
| circumstances of the benchmark.
| newlisper wrote:
| Where did you see memory consumption results?
| billywhizz wrote:
| there's a nice tool here for visualising the results in more
| detail... https://ajdust.github.io/tfbvis/?testrun=Citrine_st
| arted2020...
| dmm wrote:
| > right now at the top of the benchmarks, common high
| performance Rust code is mostly "unsafe" memory-wise
|
| I only see a single "unsafe" in the Rust implementation[0]? And
| that's only in the "raw" Actix instance, not the pg. Or are you
| talking about Actix framework being mostly unsafe?
|
| > only the Go benchmark is written like "standard Go" such that
| most people writing a service in Go will write
|
| Maybe it's just me but the Actix apis seem pretty ordinary.
|
| [0]
| https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
| da_big_ghey wrote:
| Random question: why is rocket so low in the rankings? Especially
| since something like actix is very high.
| swsieber wrote:
| The latest stable rocket release doesn't use async, actix does.
|
| The master branch of rocket does have async (and stable rust,
| not nightly!) support.
| yen223 wrote:
| If there's one lesson to draw from benchmarks like these, it
| is how important non-blocking IO is to high-performance web
| services
| noncoml wrote:
| The way I like to look at this data is in the "Multiple queries"
| test and sorted by "Max latency"
| blunte wrote:
| These always make me feel bad. Everything I use or have used is
| in the < 15% performance rank.
|
| Do the ones at the top (like Actix Pg for example) provide
| everything you need to do real development, or are they stripped
| down? In other words, is this comparing track bikes to cross
| country bikes?
| bcrosby95 wrote:
| Really depends upon what you're doing. A lot of these
| benchmarks get a lot closer together depending upon that. E.g.
| compare "fortunes" to "multiple query".
| lstamour wrote:
| If you read the source code, it's fair to say that while you
| can use the frameworks and implementations at the top to build
| your own systems, they would likely perform slower than these
| benchmarks as you add more features to them. Now, you might not
| need to add features, and that's fair, but then you should
| compare implementations equally and that's impossible when not
| every benchmark is written with identical algorithms and
| implementation code, for, well, obvious reasons.
|
| It's worth looking at low level HTTP, socket and concurrency
| management if you want faster performance, but that's not
| really a language-exclusive feature at that point. And the more
| realistic you make the benchmark -- the more communication
| between microservices, for example - the more your application
| architecture, deployment hardware, kernel and network tuning,
| and so on can play a role.
|
| I am reminded of http://rachelbythebay.com/w/2020/10/14/lag/
| for example, as something really low level you probably don't
| need to worry about... until you do.
| http://rachelbythebay.com/w/2020/05/07/serv/ also. The same is
| true of most of the performance optimization at the top. Really
| fast? Yes. Useful? Depends on the rest of your code. Are you
| writing haproxy? How much does your app really need to do? Read
| the benchmark source code and get inspired, maybe.
|
| My conclusions for the moment: Switch to Go if low memory usage
| and high performance is as critical to you as post-1.0
| stability. Go is generally stable these days ;-) Otherwise if
| instability can be tolerated but you want high performance, use
| one of these newer web server techniques and write a web server
| in unmanaged C++ that has very minimal functionality with
| language bindings. Just-js can serve as an example. Heck, the
| PHP benchmarks show that if you use PHP to write your own HTTP
| server you can still achieve high performance. That tells me
| the advantage here goes to web servers that literally "do less"
| rather than picking a language as faster over another
| language... especially when most (all?) languages can interface
| with C or C++ to do the HTTP layer at high performance while
| writing your code in whatever language you choose...
| xorx wrote:
| FWIW, we use Atreugo (#15 and #21 in this round) for most of
| our applications, and it does everything we need.
| adamdusty wrote:
| The source code is available on github:
| https://github.com/TechEmpower/FrameworkBenchmarks
|
| Techempower classifies each one. Actix is what they call a
| "platform"
|
| >a platform may include a bare-bones HTTP server implementation
| with rudimentary request routing and virtually none of the
| higher-order functionality of frameworks such as form
| validation, input sanitization, templating, JSON serialization,
| and database connectivity
| jsnk wrote:
| I love looking at these benchmarks. Thank you TechEmpower for the
| work.
___________________________________________________________________
(page generated 2021-02-08 23:00 UTC)