[HN Gopher] Building a database buffer pool in Zig using io_urin...
       ___________________________________________________________________
        
       Building a database buffer pool in Zig using io_uring's new fixed-
       buffer mode
        
       Author : gavinray
       Score  : 47 points
       Date   : 2022-10-15 20:11 UTC (2 hours ago)
        
 (HTM) web link (gavinray97.github.io)
 (TXT) w3m dump (gavinray97.github.io)
        
       | cmrdporcupine wrote:
       | Aha! I was thinking about trying this exact same approach
       | recently, so I'm glad to see I wasn't insane! Very clever.
       | 
       | I have been "in my spare time" (that is, never finishing)
       | building out a page buffer mgmt system based on the one in the
       | Umbra paper (see
       | https://db.in.tum.de/~freitag/papers/p29-neumann-cidr20.pdf)
       | which has a kind of novel approach where they essentially
       | allocate the same physical address at multiple virtual addresses
       | but with different block/slab sizes so that they can reduce
       | fragmentation and still support dynamic page sizes. Anyways, it
       | seems this author is also on the same "page" (sorry) as me but
       | focusing more on exploiting `io_uring`. Nifty.
       | 
       | I'm working in Rust, not Zig, but a lot of nice stuff to read
       | through here.
       | 
       | I will have to read through this in more detail later when I have
       | more brain-space available.
        
         | gavinray wrote:
         | Jinx, I'm also working on this, as a tutorial series!
         | 
         | I had the pleasure of hearing Neumann present the Umbra
         | architecture on the CMU DB series webinar recently.
         | 
         | I'm slowly working through re-implementing the Coroutines async
         | buffer pool paper. Blocked on trying to properly implement and
         | understand "Morsel-Driven Parallelism":
         | 
         | https://db.in.tum.de/~fent/papers/coroutines.pdf
         | 
         | I have the async, io_uring buffer pool written in Kotlin using
         | it's native Coroutines but the scheduling and morsel stuff I'm
         | still trying to understand
        
           | cmrdporcupine wrote:
           | I watched the same CMU lecture, very cool.
           | 
           | Are you getting paid to work on this kind of thing, anywhere?
           | Do you want to be?
        
             | gavinray wrote:
             | > Are you getting paid to work on this kind of thing,
             | anywhere?
             | 
             | My dayjob is a fair bit higher level. I work on a sort of
             | query engine on the JVM that translates an internal query
             | IR/AST into dialect-specific SQL + other languages so you
             | can query anything with the same frontend language:
             | 
             | https://techcrunch.com/2022/06/28/hasura-now-lets-
             | developers...
             | 
             | I think it'd be neato to work on a database, but my dayjob
             | is also pretty interesting to me.
        
         | gavinray wrote:
         | > I'm working in Rust, not Zig, but a lot of nice stuff to read
         | through here.
         | 
         | FYI, Datadog has a Rust library for scheduling things to run
         | thread-per-core with io_uring
         | 
         | It'd be really useful for DB use cases:
         | 
         | https://github.com/DataDog/glommio
        
           | cmrdporcupine wrote:
           | Nifty. I started from the part of the Umbra presentation that
           | initially struck me as most interesting -- the allocator
           | portion (where he talks about repeatedly MMAP_ANONYMOUS on
           | the same address range to reduce fragmentation, etc.) I
           | stopped working when I started pondering the bits about
           | pointer swizzling and how I'd make that work nicely in Rust,
           | and never got as far as the pieces that would actually do the
           | syncs to disk.
           | 
           | I also came across another fellow who was inspired by the
           | optimistic locking btree presented in LeanStore Umbra:
           | https://github.com/NeowayLabs/bplustree
        
       | gavinray wrote:
       | One note: I mistakenly thought that fixed buffers were part of
       | the 5.19 changes for provided buffers
       | 
       | Jens Axboe corrected me on this, fixed buffers have existed for a
       | while apparently!
       | 
       | https://twitter.com/axboe/status/1581392910687567873?t=gx_tQ...
        
       ___________________________________________________________________
       (page generated 2022-10-15 23:00 UTC)