[HN Gopher] Benchmarking Ruby 2.6 to 3.2
___________________________________________________________________
Benchmarking Ruby 2.6 to 3.2
Author : ksec
Score : 120 points
Date : 2022-12-26 13:31 UTC (9 hours ago)
(HTM) web link (gettalong.org)
(TXT) w3m dump (gettalong.org)
| brigandish wrote:
| If I'm reading those results right, it seems that Ruby was
| getting slower between 2.6.10 and 3.1.3 with yjit, and can still
| outperform Ruby 3+ versions without yjit. Does anyone know why
| this is?
| faraaz98 wrote:
| I'd like to know this from a more informed person as well. My
| guess is that yjit is only now production ready, they've been
| building it for a while and enabled behind an experimental flag
| maxfurman wrote:
| YJIT was disabled by default before 3.2
| gettalong wrote:
| Yes, there was a bit of a drop off in performance but only when
| YJIT is not used. With YJIT there are performance gain. Also
| note that YJIT in 3.1 was still experimental and more proof-of-
| concept. Only with the 3.2 release was it now deemed production
| ready.
|
| And YJIT's creator, Shopify, is already using it in production
| with Ruby 3.2 for all their store fronts!
| chucke wrote:
| Small nitpick: Maxime Chevalier-Boisvert is the creator of
| YJIT and the BBV optimization technique that underpins it.
| Shopify is the company which employs her and funds its
| further improvement.
| gettalong wrote:
| Yes, I know but thought it would be simpler to just say
| Shopify. Sorry for the inaccuracy.
| alberth wrote:
| I'm surprised to see how little speed-up have happened over the
| years. Especially when you compare it to something like PHP which
| has massive gains.
|
| Seems like most of the Ruby gains are related to memory usage.
| nickjj wrote:
| > Seems like most of the Ruby gains are related to memory
| usage.
|
| That would be a welcome addition, especially with Rails where
| you typically end up running your web app, background worker
| and action cable as separate processes. That's (3) running
| services each booting up your application to some degree.
|
| Combine that with most apps using very little CPU and more and
| more memory, it's nice to be able to reduce memory when you can
| to optimize the machines you host things on. It always feels
| dirty having a server using 60% memory and 5% CPU.
| influx wrote:
| Can those processes fork and save memory on duplicate pages?
| (I'm not a rails person)
| gettalong wrote:
| Yes, COW was implemented a few versions ago and the garbage
| collector also supports it to avoid unnecessary copying.
| skrtskrt wrote:
| I don't have a ton of love for Python now that I've moved on to
| static typing but it's hard for me to see myself choosing Ruby
| over Python for anything.
|
| Python has gotten significantly faster, has static typing built
| in with an ecosystem of static typecheckers, and async is an
| ergonomic first class language feature - sure async can have
| rough edges but the 99.9% case is dead simple with no
| surprises.
|
| Obviously packaging for Python is still a pain but I don't see
| Ruby being better
| bhaak wrote:
| > Python has gotten significantly faster
|
| All the synthetic benchmarks I've seen claim that Ruby is at
| least on par with Python's speed if not faster. Like this
| https://benchmarksgame-
| team.pages.debian.net/benchmarksgame/...
|
| But then speed in a dynamic language is often very context
| dependent. If I want to do Rails I won't choose Python. If I
| want to do machine learning, Python is the obvious choice.
|
| > Obviously packaging for Python is still a pain but I don't
| see Ruby being better
|
| Ruby packaging just works whereas Python's is a pain. Can't
| see how you don't see it being better.
| skrtskrt wrote:
| Those benchmarks are with Python 3.10 - the big recent
| speedups are in 3.11, with more to come
|
| > The Python 3.11 release announcement cites 10~60%
| improvements over Python 3.10 and a 1.22x speed-up for its
| standard benchmark suite.
| lraktr wrote:
| Yeah, the Debian benchmarks are independent.
|
| The recent Python benchmarks all seem to use the same
| benchmark suite (which is incredibly complex to the point
| that it's unclear what is actually being measured) or hype
| the Microsoft involvement.
|
| Even if you believe them, it is just 10-60% speedup for a
| very slow language. It won't show up in web applications.
| chucke wrote:
| Python improvements are on par with ruby, and ruby has a JIT,
| which means that upside fir real world improvement is much
| higher.
|
| Python has no static typing. It has type annotations, which a
| few 3rd party static analysers such as mypy use. Ruby has had
| the same since 3.0 with rbs. The ecosystem may be less
| mature, but it exists, and Python's isn't necessarily
| perfect.
|
| Python async is terrible. You have to colorize functions, opt
| out from a significant part of the ecosystem which either
| does not support async, or is very unstable, and completely
| relearn how to do debugging / performance measurement. Ruby
| has smth better since ruby 3.0, via the fiber scheduler,
| which supports running network related code asynchronously
| with *no changes or special function annotations*.
|
| Ruby packaging has been figured out since 2007, and has been
| influencial for other ecosystems, being the first mainstream
| language adopting lockfiles.
|
| If you prefer using Python, you are entitled to your opinion
| and choice. But those arguments are all FUD.
| foxandmouse wrote:
| I've only ever known Javascript/ Typescript. I've been
| thinking of picking up ruby to learn "practical" functional
| programming and maybe see what the infatuation with rails is,
| but I'm starting to think it might be more "valuable" to
| learn python.
| freedomben wrote:
| Purely my opinion after learning and professionally doing
| both Ruby and Python. If elegance and consistency in APIs
| matter, Ruby is by far the preference. Generally speaking
| (again from my own experience which should be taken with a
| grain of salt) the Ruby community tends to value
| readability/maintainability of code a lot higher, so
| working in the codebases is typically a lot more pleasant.
|
| IIWM I'd go with Ruby hands down.
| sodapopcan wrote:
| I wouldn't recommend python or ruby if you're trying to
| learn more about functional programming... you may as well
| stick with JS in that case (imo, at least). I'd choose a
| language that has at least had native immutability.
| Something like Elixir/Erlang, OCaml, Clojure, or F#.
| skrtskrt wrote:
| I'm seeing a pretty significant amount of Elixir job
| listings these days even in areas where you used to see
| all Ruby or Python
| sodapopcan wrote:
| I've actually been having a tough time finding another
| job in Elixir. Sooooo many companies who use it are in
| crypto and I just have no interest.
| satvikpendem wrote:
| If you already know TypeScript, just use a functional
| approach to your programs, and/or use libraries like fp-ts
| and io-ts. Ruby is definitely not more FP based than TS.
| skrtskrt wrote:
| Python and Ruby are both only really as functional as you
| choose to use them. You'll see more imperative code in the
| wild for both, though the big data stuff in Python leans
| more functional.
|
| Rails and Django are both still super popular from scanning
| job listings, so e-commerce employment wise they're both
| great.
|
| FastAPI is also growing a lot.
|
| Big Data leans heavily Python
| joshspankit wrote:
| > it's hard for me to see myself choosing Python over Ruby
| for anything.
|
| Did you mean this the other way around?
| skrtskrt wrote:
| yep fixed
| tmountain wrote:
| Pure speculation--Ruby has a deeper OOP story (everything is an
| object), and that comes with a builtin performance impact. I
| also feel like performance was never a top priority with the
| language (ergonomics instead), so certain aspects of the core
| design may not lend themselves to optimization as easily.
| noyoudumbdolt wrote:
| PHP had the benefit of Facebook pouring money into it. Same
| with JS and Google. No company has done something comparable
| for Ruby.
| yucky wrote:
| This is surprising to hear, considering how big Shopify has
| gotten and how much resources they've put into Ruby.
| Doctor_Fegg wrote:
| YJIT, Ruby's new JIT compiler, is from Shopify.
| alberth wrote:
| Let's not forget that GitHub is a giant Rails app that
| Microsoft runs.
|
| Microsoft is also one of the best language/compiler company
| in the world.
| chucke wrote:
| Microsoft didn't start pouring bucks into improving ruby
| yet. Their focus has recently been on JS, C# and python.
| theredfury wrote:
| https://shopify.engineering/shopify-ruby-at-scale-
| research-i...
| bfgoodrich wrote:
| [dead]
| alberth wrote:
| That's not exactly accurate. FB went off and did their own
| thing.
|
| PHP gains were independent of FB.
| weaksauce wrote:
| shopify is putting up a lot of money and resources into ruby.
| osrec wrote:
| I believe Stripe use Ruby. Perhaps they'll fund its
| improvement...
| iloveitaly wrote:
| Stripe has done a lot of work on a ruby type system
| (https://sorbet.org) and has been working on a ruby
| compiler as well (https://sorbet.org/blog/2021/07/30/open-
| sourcing-sorbet-comp...)
| brasic wrote:
| TruffleRuby is leading the pack here:
| https://eregon.me/blog/2022/01/06/benchmarking-cruby-mjit-yj...
| jwcooper wrote:
| I think these charts are more informative, especially for Rails
| apps: https://speed.yjit.org/
|
| Ruby 3.2 and production-ready YJIT feel like a big deal as
| someone who has been in the Rails and Ruby ecosystem for 15
| years.
|
| Rails in particular has seemed to be very difficult to optimize
| for performance in Ruby, and YJIT seems to have done it.
| samsquire wrote:
| I am interested in programming language theory and development
| but I am a beginner in both.
|
| Do JIT compilers suffer from boxing pointer chasing primitives
| since everything is an object and not a memory location value
| type?
|
| Can hot loops over arrays be efficient if every item in the array
| is a pointer?
| kaba0 wrote:
| I am by no means an expert on the topic, but I don't think that
| "array of pointers"-chasing is the primary performance bottle
| neck of most managed language. It happens all too often with C
| programs as well (or even worse, linked lists) and if the
| pointed to objects are close to each other in memory/are in
| cache (as they might be in case of a generational GC) than it
| is not a big performance hit. Also, most apps don't spend most
| of their time in a hot loop iterating over arrays, ordinary
| programs seldom have huge arrays.
|
| Dynamic properties, non-uniform object shapes may be more
| responsible for this generic slowness of very dynamic
| languages, and from what I gathered assuming a shape for these
| objects constitutes a kind of advanced optimization (done by JS
| engines, truffleruby, not sure about these engines), even if
| its done depending on language idioms it may not give that big
| of a boost. Ruby is a very dynamic language with e.g. hitting
| not implememted methods and dynamically handling, etc.
| samsquire wrote:
| Thank you for your explanation.
|
| My understanding is that determining jump target at compile
| time is a big source of performance gain at runtime. If you
| know the function to jump to at compile time and not wait to
| runtime to decide where to jump.
|
| I've looked into arrays of structs and structs or arrays for
| performance. But I am curious how to create a managed
| language that is fast. I think we look Java and C# for
| guidance.
| kaba0 wrote:
| > If you know the function to jump to at compile time and
| not wait to runtime to decide where to jump
|
| Well, I don't know about more dynamic languages, but single
| dispatch can be very well optimized away, Java for example
| can determine how many possible implementation of a given
| method is loaded, and simply JIT compile or even inline the
| function if there is only a single implementation. Later,
| it can deoptimize that call, e.g. when a new class is
| loaded at runtime that also implements that method. This is
| one area where JIT is definitely in a better position than
| AOT compilers.
| artagnon wrote:
| It's interesting that Ruby now pulls Rust as a dependency, since
| YJIT is written in Rust
| [https://github.com/ruby/ruby/tree/master/yjit].
| jwcooper wrote:
| It seems well designed, as it's an optional dependency as of
| now.
|
| > By default, YJIT does not get compiled and cargo/rustc is not
| required. If YJIT is built in dev mode, then cargo is used to
| fetch development dependencies, but when building in release,
| cargo is not required, only rustc. At the moment YJIT requires
| Rust 1.60.0 or newer.
|
| https://github.com/ruby/ruby/pull/5826
| taf2 wrote:
| Would be nice to see performance stats for running unicorn web
| app... I am pretty sure Shopify published some stats during the
| 3.1 and possibly 3.2 preview phase...
| firecall wrote:
| I thought Ruby 3 was supposed to be 3x faster than version 2?
| [deleted]
| Mikeb85 wrote:
| It is for many workloads. Just not Rails.
| joshspankit wrote:
| That would have been almost impossible.
|
| Ruby was never designed as a "fast" language, but 3x is a
| _huge_ number for anything this mature.
| galangalalgol wrote:
| Looking at the benchmark game, it seems 3x _current_
| performance would put it in the company of racket, which is
| still noticably slower than node js. While right now it is
| much closer to python.
|
| All benchmarks are misleading but it shows 10x slower than
| java, lisp, and pascal. Is there a reason that isn't actually
| representative of idiomatic use?
| WJW wrote:
| The benchmarks in the benchmark game are typically CPU
| bound, which heavily favors compiled languages for obvious
| reasons.
|
| With regards to what "idiomatic use" is for Ruby, we could
| argue that most ruby apps are web backends and most of
| those spend the majority of their waiting for DB and/or API
| calls and don't spend all that much doing work in actual
| Ruby.
| galangalalgol wrote:
| The same could be said for python with calls into numpy
| and pytorch consuming much of the time, but no one calls
| it designed for speed.
| spleen wrote:
| 3x increase when 3.0 is compared to 2.0[1]. I think it was
| largely achieved.
|
| [1] https://blog.heroku.com/ruby-3-by-3
| breckenedge wrote:
| https://www.ruby-lang.org/en/news/2020/12/25/ruby-3-0-0-rele...
| gls2ro wrote:
| Here is another benchmark from running Discourse:
|
| https://twitter.com/rafael_falco/status/1607395317661458432
| xfalcox wrote:
| Hey, that's me!
|
| Numbers were generated using
| https://github.com/discourse/discourse/blob/main/script/benc...
| if anyone wants to try it out too.
|
| We already have 3.2 working internally and plan to roll it out
| to everyone soon.
| stillSlow wrote:
| Oh look Ruby is still slow.
| gettalong wrote:
| Yes, it is slower but not that slow, e.g
| https://hexapdf.gettalong.org/documentation/benchmarks/optim...
| lakomen wrote:
| How many concurrent requests is 3.2. able to handle on an average
| consumer PC 5 years old, 4 cores? Just plain hello world.
|
| Go(lang) about 5-6k.
| claudiug wrote:
| grab ruby, run a test and show us.
| joshmn wrote:
| How many of us need 5-6k concurrent requests?
| WJW wrote:
| I did 65k concurrent requests back in 2018 on a single core in
| Ruby 2.4/2.5 or so (see
| https://wjwh.eu/posts/2018-10-29-double-hijack.html) as long as
| each request doesn't do much. Having 6k open connections is not
| that much. Did you actually mean request/seq?
|
| EDIT: I was curious on how many req/s modern Ruby would
| actually do for simple hello world, so I coded up the smallest
| Hello world example from https://github.com/rack/rack and ran
| it with the (default) Puma webserver, limited to 4 threads (and
| thus cores) with `-t 4`. According to `ab`, it averages around
| 8500 requests per second over a sample size of 100k requests. I
| didn't find out how to run rack apps with YJIT yet, but it
| seems reasonable to expect that to speed it up a bit further.
| xfalcox wrote:
| Use the ENV flag `export RUBY_YJIT_ENABLE=1` and run it
| again.
| WJW wrote:
| 10445.25 req/s average over 100k requests, so about 2k
| req/s extra.
___________________________________________________________________
(page generated 2022-12-26 23:01 UTC)