[HN Gopher] Benchmarking CRuby, MJIT, YJIT, JRuby, and TruffleRuby
___________________________________________________________________
Benchmarking CRuby, MJIT, YJIT, JRuby, and TruffleRuby
Author : shelajev
Score : 73 points
Date : 2022-01-06 14:42 UTC (8 hours ago)
(HTM) web link (eregon.me)
(TXT) w3m dump (eregon.me)
| klelatti wrote:
| The Truffleruby results look amazing but leave me with the
| question why we don't see stories of implementation at major Ruby
| / Rails sites. Too soon, licensing or not yet 100% compatible?
| eregon wrote:
| Compatibility is one, it's hard to be 100% compatible with
| CRuby, and large codebases tend to sometimes depend
| unintentionally on weird behavior or even bugs in CRuby.
|
| Keeping up compatibility (while keeping things efficient) is a
| lot of work, TruffleRuby tries to reduce that by reusing as
| much as possible existing code, including reusing C extensions
| shipped with CRuby.
| slx26 wrote:
| Windows support?
| Mikeb85 wrote:
| Pretty sure Shopify uses it for some apps.
| klelatti wrote:
| Thanks - yes of course Chris Seaton works at Shopify.
|
| Still a bit puzzled why we haven't seen any headlines about
| major savings.
| pmahoney wrote:
| Many years ago (2011 or so?), I worked on a Rails app that used
| JRuby. The driving force was a small part of the app needing a
| Java library. I think we did see some performance benefit, or
| at least benefit from using multi-threaded web servers instead
| of process-per-request servers (like Unicorn).
|
| However, we also ran into many problems that felt like we were
| the first to encounter them (few or no similar reports on
| project bug lists or StackOverflow etc.) which is never fun if
| you're under any kind of pressure to deliver.
|
| In hindsight, if I had something small and reasonably isolated
| (a microservice?), it could make sense to use JRuby or Truffle
| (if you need some JVM lib; if it's purely for performance
| reasons I'd just grab a different language for that
| microservice). But for a larger app that pays the bills, I'd
| stick with the well-trod path (at the time that was MRI,
| Unicorn, single-threaded).
| pizza234 wrote:
| Explanation from Stripe1:
|
| > Nearly all of Stripe's codebase is implemented in Ruby
| running on the default Ruby VM (YARV). Not only did we not need
| Java VM-level interoperability, choosing either alternative
| Ruby implementation would have made for a difficult migration
| path. Stripe relies heavily on gems with native extensions, and
| as you can imagine, a multi-million line Ruby codebase over
| time starts to depend on Ruby-the-implementation, not just
| Ruby-the-language.
|
| 1: https://sorbet.org/blog/2021/07/30/open-sourcing-sorbet-
| comp...
| eregon wrote:
| That second sentence seems a fair assessment. TruffleRuby
| does support many native (C/C++) extensions so that's rarely
| an issue. But indeed in such a large codebase it's likely to
| depend unexpectedly on CRuby-specific behavior. And while
| that can be fixed it takes some effort either in TruffleRuby
| (to match CRuby) or in the app (e.g. to avoid relying on
| `RubyVM`).
| x3n0ph3n3 wrote:
| I just ran a comparison of a benchmarking script we use for our
| own product and saw wildly different results. There's no way
| I'd consider migrating to truffle today:
|
| # With MRI ruby-2.6.6
|
| Measuring [loading ruby dependencies] 3.560960 2.507366
| 6.075755 ( 6.154898)
|
| Measuring [loading configuration yaml] 0.823779 0.085605
| 0.909384 ( 0.931334)
|
| Measuring [creating objects from YAML and performing filtering]
| 0.499576 0.042023 0.541599 ( 0.552427)
|
| Measuring [object validation] 2.362139 0.226133 2.588272 (
| 2.674958)
|
| Total time: 7.246786 2.861230 10.115445 ( 10.314025)
|
| # With truffleruby-21.3.0
|
| Measuring [loading ruby dependencies] 91.818473 5.598149
| 97.427109 ( 36.676471)
|
| Measuring [loading configuration yaml] 28.032630 0.811577
| 28.844207 ( 8.616437)
|
| Measuring [creating objects from YAML and performing filtering]
| 14.952781 0.479387 15.432168 ( 4.634803)
|
| Measuring [object validation] 86.747057 2.788266 89.535323 (
| 28.324909)
|
| Total time: 221.595872 9.680172 231.286531 ( 78.264226)
| eregon wrote:
| Startup is likely to be worse, because it typically runs a
| lot of different code for not long and the JIT might not have
| enough time to optimize much of that. OTOH the JIT needs to
| learn what the program is doing, i.e., profile it in a sense,
| and that has a cost on interpreter speed. Still these numbers
| are worse than expected, especially [object validation], so
| if there is a way to reproduce it'd be great if you can open
| an issue about it.
|
| Persisting the JITed code is what we think can solve the
| slower startup entirely: https://www.graalvm.org/graalvm-as-
| a-platform/language-imple... Also other things mentioned in
| https://eregon.me/blog/2022/01/06/benchmarking-cruby-mjit-
| yj...
| ch4s3 wrote:
| It isn't 100% MRI 3.0 compatible[0] and wasn't able to run
| Rails at first. It's also not always faster for Rails which
| occupies a lot of Ruby mindshare.
|
| [0] https://github.com/oracle/truffleruby#current-status
| klelatti wrote:
| Is the Rails performance something to do with the amount of
| meta programming going on - I can imagine that being hard to
| optimise!
| joelbluminator wrote:
| Well YJIT succeeded to improve Rails by 30% in a year , so
| it IS possible. https://speed.yjit.org/
| eregon wrote:
| It's the same benchmark in the blog post, `railsbench`.
| So TruffleRuby already speeds up some Rails apps like
| that one but not every Rails app/program. (TruffleRuby
| 3.27x, YJIT 1.33x on railsbench)
| eregon wrote:
| Actually TruffleRuby optimizes metaprogramming more than
| any other Ruby implementation. Rails is not the issue, big
| codebases is the issue: more code = more chances to use
| something which is not so fast on TruffleRuby yet = more
| chance to become a bottleneck and degrade the overall
| performance.
| klelatti wrote:
| Thanks for the thoughtful and insightful responses.
| Understand that a large code base is hard - your point
| here though presumably means that some things in
| truffleruby are significantly slower than CRuby to cancel
| out the gains. Or have I misunderstood?!
| eregon wrote:
| Yes, some things are slower. Most of them I'd say are
| unexpected performance bugs. I'd think most are easy to
| fix once investigated, but some can be hard to fix
| (recent example, `File.read` is quite fast on CRuby).
|
| Some things are expected to be slower, for instance
| constantly redefining (monkey-patching) methods or
| constants is slower on TruffleRuby, but that's typically
| because the program is broken and so it'd be slow on
| CRuby as well.
| klelatti wrote:
| Thanks again and of course for Truffleruby!
| ch4s3 wrote:
| I'm not sure. If Chris Seaton pops into the thread, you
| should ask him.
| sparker72678 wrote:
| I keep forgetting just how fast TruffleRuby is.
| karmicthreat wrote:
| What is the downside to TruffleRuby? Memory consumption since
| its JVM based?
| kenhwang wrote:
| Very subtle differences in behavior. It's fine if you start
| writing your project targeting TruffleRuby, but its a
| nightmare porting a massive existing ruby codebase.
| eregon wrote:
| Compatibility is certainly an issue for massive codebases,
| OTOH I think over time TruffleRuby is getting closer and
| closer to CRuby behavior to the point it would be fairly
| rare to find a compatibility issue. Do you have personal
| experience trying to run such a codebase on TruffleRuby?
| kenhwang wrote:
| We noticed many numerical operations sometimes didn't
| return the same class as cruby
| (Fixnum/Bignum/BigDecimal), sometimes resulting in wildly
| different calculation results.
|
| Numerical to string formatting is a bit different too.
| Which makes the above problem even worse, compounded by
| how frequently ruby web code goes string to numbers to
| string to numbers.
|
| Regexes behave differently if they worked at all.
| Mikeb85 wrote:
| I use a benchmark that's pretty rough on languages, basically
| lots of maths and loops.
|
| Python 3 and Ruby 3 (with no jit) both take about 90 seconds on
| my computer.
|
| Ruby 3.1 with --YJIT takes about 35 seconds.
|
| TruffleRuby takes 9 seconds (!!)
|
| JS (Chrome) takes 8 seconds.
|
| C++ takes just over a second and C is about a second even.
| igouy wrote:
| Please use something "basically lots of maths and loops" where
| we can see the source code (like the blog post does).
|
| n-body
|
| https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
|
| https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
|
| spectral-norm
|
| https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
|
| https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
___________________________________________________________________
(page generated 2022-01-06 23:01 UTC)