[HN Gopher] Lessons from debugging a tricky direct memory leak
       ___________________________________________________________________
        
       Lessons from debugging a tricky direct memory leak
        
       Author : SerCe
       Score  : 16 points
       Date   : 2023-12-27 01:28 UTC (21 hours ago)
        
 (HTM) web link (medium.com)
 (TXT) w3m dump (medium.com)
        
       | brancz wrote:
       | I'm incredibly confused why profiling wasn't even mentioned in
       | this post.
       | 
       | I would highly recommend anyone troubleshooting something like
       | this to do continuous profiling and the root cause of this type
       | of problem becomes painfully obvious within seconds.
       | 
       | (Disclaimer I founded a company in this space, and we recently
       | open sources a library for rust to do this:
       | https://www.polarsignals.com/blog/posts/2023/12/20/rust-memo...)
        
         | marginalia_nu wrote:
         | Profiling will catch many of these types of errors, but not all
         | of them. Although I do think a profiler is indeed probably the
         | first thing you should be reaching for, especially in Java,
         | which has some of the best profiling in any language.
        
           | brancz wrote:
           | Right, but judging by the detail of the post, and not
           | mentioning profiling makes it seem like it wasn't even
           | attempted.
        
       | Thaxll wrote:
       | Java flight recorder wouldn't help here?
        
         | marginalia_nu wrote:
         | Yes[1], but you need a relatively new version of JFR and you
         | also need to enable a few flags to get much value out of it.
         | 
         | [1] https://www.morling.dev/blog/tracking-java-native-memory-
         | wit...
        
       | TheNewAndy wrote:
       | If you have a memory leak, then take a process that has been
       | leaking for a while and look at its memory. Just pole a few
       | places at random and statistically you should quickly find what
       | is leaking. I am pressure this would have cut down a lot of the
       | investigation needed in this story (though it is always easy to
       | say this kind of thing after the biug has been found)
        
       | SerCe wrote:
       | I stumbled upon this article in the latest edition of the 'Java
       | Performance Tuning Newsletter'. Relatively recently, I also
       | published a blog article [1] diving into all aspects of JVM
       | memory, including native memory profiling.
       | 
       | [1]: https://serce.me/posts/01-02-2023-jvm-field-guide-memory.
        
       | benrow wrote:
       | We ran into a problem where we blew the direct memory limit
       | recently.
       | 
       | To trace the problem, I added a breakpoint on
       | ByteBuffer.allocateDirect() - this is how DM is normally
       | allocated. But rather than suspending the process, we set it to
       | print the stacktrace and continue.
       | 
       | This gave us handy output of possible causes. Focusing on the
       | cases which were called from our own code (likely to be the root
       | cause), we found a call to Files.readAllBytes() - which
       | immediately sounds suspicious to cause a memory leak. Indeed this
       | was the culprit, and explained why the problem was sporadic and
       | hard to reproduce.
       | 
       | Fortunately, this "by inspection" approach got us to the answer.
       | 
       | Others have stated profiling would work well. We'd have needed a
       | local reproduction of the issue, which wasn't available at the
       | time. Otherwise we could have used our SaaS APM software which
       | runs in deployed envs, but that takes a sampling approach and
       | doesn't really dig deep enough for this kind of stuff.
        
       ___________________________________________________________________
       (page generated 2023-12-27 23:01 UTC)