[HN Gopher] More evidence for problems in VM warmup
___________________________________________________________________
More evidence for problems in VM warmup
Author : ltratt
Score : 60 points
Date : 2022-11-16 08:06 UTC (1 days ago)
(HTM) web link (tratt.net)
(TXT) w3m dump (tratt.net)
| hinkley wrote:
| If I were the King of the Forest, our routing fabric would
| utilize something like nginx's ramping weight feature to throttle
| new servers for a few minutes before they reached full membership
| in the cluster.
|
| As things are we end up running something more like blue-green
| deployments and hit the dormant side with a stress testing tool.
| We haven't really come up with a better solution, though we have
| steadily reduced both the time necessary to warm up the servers
| and the worst case behavior if you accidentally skip the warming
| step. Today you will just have a very bad time. Originally
| circuits would blow like crazy.
| adwn wrote:
| > Here's an example of a "good" benchmark from our dataset which
| starts slow and hits a steady state of peak performance
|
| Am I misinterpreting their graph? The difference between "slow"
| and "peak performance" seems to be a factor of about 1.005, so a
| whopping _0.5% improvement_ after warmup?
| SethTro wrote:
| A good blog post has lots of hard parts (layout, scope, visuals,
| audience). Laurence Tratt, you nailed it for me! I loved the
| details you put in "Benchmarking methodology" and the clean
| layout with instructive visuals.
| hinkley wrote:
| It's interesting how often the same stories play out for
| different interpreter implementations. Everyone tries:
|
| - a multi-pass JIT
|
| - interpretting the input directly to reduce time to first op
|
| - making the fastest transpiler they can and skipping the
| interpreter
|
| All of these are addressing different constraints and affect each
| other. For instance, a cheap transpiler is only slightly slower
| than an interpreter loop, and allows you to move the threshold
| for the JIT farther to the right. If you can avoid trying to
| optimize things that you will only be slightly successful at, you
| can invest more of your CPU budget in deeper optimization on the
| hottest paths. You are also running on-stack replacement less
| often, and in fewer scenarios, which may mean you make different
| tradeoffs there as well.
| winrid wrote:
| Note that certain things can cause your code to be un-jitted. If
| you run into a NPE, for example, what actually happens is the
| compiled code _segfaults_. Adding null checks everywhere would be
| slow, so instead the JVM lets it blow up. When the jitted code
| segfaults that bytecode goes back into interpreted mode and has
| to be compiled again w / updated heuristics.
|
| Still, covering cases like this is easier than writing C. :)
| winrid wrote:
| Also what's missing in this article is a relative comparison.
| What if the testing machines have some kind of cron or process
| that's messing up the microbenchmark? I didn't see that
| comparison here.
| pizlonator wrote:
| For this research to be useful, the authors should switch to
| large benchmarks.
|
| VMs win in the average. Therefore, VMs have great cases, average
| cases, and crappy cases. The crappy cases will always exist, and
| that's known to VM architects and it's not a bug.
|
| Essentially this work is like criticizing a professional gambler
| for his biggest losses when the gambler is ahead in the average
| (or conversely praising him for his biggest wins when they're
| behind in the average).
|
| Source: I build VMs for a living.
| stcredzero wrote:
| _The crappy cases will always exist, and that's known to VM
| architects and it's not a bug._
|
| I think this is very much the lens of the VM developer. From a
| different POV, why isn't it valid to ask, "Why can't this also
| be fast?" Why _is_ that a feature and not a bug?
|
| _Essentially this work is like criticizing a professional
| gambler for his biggest losses when the gambler is ahead in the
| average_
|
| The losses are still losses. Maybe we should try to eliminate
| the "gambling?"
| pizlonator wrote:
| VMs are fundamentally about gambling. If you knew statically
| what to do in the compiler then you'd run an ahead of time
| (AOT) compiler. VMs are for those cases where an AOT would
| have been slower, and empirically, languages like JS, Java,
| and Lua perform much better with a dynamic VM than with an
| AOT and that's why we use VMs.
|
| So, eliminating gambling is like saying we should just do
| AOTs. We could but then we'd be slower.
|
| > From a different POV, why isn't it valid to ask, "Why can't
| this also be fast?" Why is that a feature and not a bug?
|
| A professional gambler ought not ask this question. A
| professional gambler ought instead ask: how am I doing in the
| average? Hence the flaw in this research. It's bad gambling.
| nightpool wrote:
| I appreciate that these issues definitely complicate research and
| benchmarking of VM performance, but I'm confused about this
| question: "If you're a normal user, the results suggest that
| you're often not getting the performance you expect". Is that
| true? Most VMs are not simple, predictable systems that would
| ever reach a "steady state"--they're dynamic systems that are
| constantly executing different parts of the codebase and taking
| wildly different codepaths at different times. V8 is a great
| example of this--the type of code executed on a page changes
| wildly depending on what you're doing and what actions you take.
| Why would we even _want_ to optimize it for "reaching a steady
| state", when different parts of the codebase may be more or less
| useful at different times? It seems more important to me to work
| on optimizations that can allow us to deoptimize parts of the
| codebase that we don't think we will be useful again, to save on
| memory, even if it involves sacrificing this theoretical notion
| of a "steady state"
| chrisseaton wrote:
| > Most VMs are not simple, predictable systems that would ever
| reach a "steady state"--they're dynamic systems that are
| constantly executing different parts of the codebase and taking
| wildly different codepaths at different times. V8 is a great
| example of this--the type of code executed on a page changes
| wildly depending on what you're doing and what actions you
| take.
|
| That is already the exact point being made by Laurie and Edd's
| research.
|
| > Why would we even want to optimize it for "reaching a steady
| state", when different parts of the codebase may be more or
| less useful at different times?
|
| A steady state _for whatever the application is doing right
| now_. Not a single, statically known steady state, without
| considering what it 's doing.
|
| > It seems more important to me to work on optimizations that
| can allow us to deoptimize parts of the codebase that we don't
| think we will be useful again, to save on memory, even if it
| involves sacrificing this theoretical notion of a "steady
| state"
|
| I think that's a completely solved problem isn't it? You can GC
| a compiled method. What more do you want?
| hinkley wrote:
| > the type of code executed on a page changes wildly depending
| on what you're doing and what actions you take.
|
| I can't speak for anyone else, but I feel like one of my goals
| for mature software is to start factoring out common bits of
| code so that a lot of the workloads either share code in
| common, or share output in common (avoiding code re-execution).
|
| There's a high correlation between Choose Your Own Adventure
| code that's doing random stuff for every request and my
| unhappiness on a project. I usually find at least a few highly
| effective people on any team who sympathize or agree.
|
| If at least 20% of the code isn't "in common" I'm
| uncomfortable. In common doesn't necessarily mean every task,
| but multiple tasks. EG, every task runs 5 of these 20 common
| concerns.
|
| I'm a sucker for complementary code and processes though. I am
| sure that some of my preferences in software design have an
| origin story in hardware sympathy, some in
| ergonomics/cognition, and a number in both. I couldn't begin to
| guess which opinions started with which observation at this
| point. Did I rationalize a human optimal solution or
| rationalize a hardware optimal one?
| hedora wrote:
| After thinking about these issues for over a decade, and having
| the rug repeatedly pulled out from under me by Java, I've come
| to the conclusion that the easiest path forward is to stick
| code snippets into godbolt, or to disassemble optimized
| binaries from ahead of time compiled languages.
|
| (I didn't say doing this is easy; just that it is easier than
| reasoning about a JIT)
| PathOfEclipse wrote:
| What's the difference in practice between a JIT and an AOT
| compiler that supports optimizations based on runtime
| analysis (I forgot the technical term for that. Profile-
| guided optimization, maybe?)? I think the real issue is, the
| higher level the language, the harder it is to reason about
| what your code will actually get compiled down to. If you
| want predictable performance in Java, or any high-level
| language, in my experience you have to essentially write C
| code in that language, or at least get as close as possible.
| In Java, that usually means:
|
| * sticking to primitives or arrays of primitives as much as
| possible. The language has no support for user-defined value
| types!
|
| * Sticking to regular for loops, while loops, and other
| similar control structures.
|
| * Using offheap memory via Unsafe or similar API when
| feasible.
| jeffbee wrote:
| The things that most Java applications need to do to reduce
| the variance of runtime outcomes are sadly a little more
| basic: use a modern VM, set the compilation flags
| appropriately, and make sure the code caches are large
| enough to hold all your hot functions.
| PathOfEclipse wrote:
| My experience leads me to the exact opposite conclusions.
| For example, your code cache is either big enough or it's
| not. It's a pretty binary situation and a simple checkbox
| to ensure your app didn't run out of space. To quote from
| https://stackoverflow.com/questions/7513185/what-are-
| reserve...:
|
| "Normally you'd not change this value. I think the
| default values are quite good balanced because this
| problems occur on very rare occasions only (in my
| experince)."
|
| Similarly, your organization tends to run the version of
| Java that the whole company supports. I have never worked
| anywhere where someone was allowed to use a different
| version than the rest of the company for performance
| reasons.
|
| In contrast, the advice to write low-level code for
| predictable, high performance has been virtually timeless
| and generally applicable across languages.
| jeffbee wrote:
| I'm not suggesting that devs should just wildcat a newer
| JDK, but surveys indicate that significant population of
| organizations are still on 8 or even 7. If that's your
| org, this is probably performance left on the table.
|
| As for the code caches, you're right that it's a binary
| outcome, but the signals are subtle and if you aren't
| looking at the right stats a non-expert without a good
| calibrated gut feeling for how good the performance
| should be might just conclude that the app was written
| poorly, or Java just sucks, or whatever.
| fulafel wrote:
| I think it depends on whether you are thinking about backend
| code or things like browsers.
|
| The results section of the article leads with: The research
| question in TCPT I am most interested in is RQ1 "Do Java
| microbenchmarks reach a steady state of performance?"
___________________________________________________________________
(page generated 2022-11-17 23:01 UTC)