[HN Gopher] Considerations when building embedded databases
       ___________________________________________________________________
        
       Considerations when building embedded databases
        
       Author : tzmlab
       Score  : 37 points
       Date   : 2024-08-02 18:45 UTC (4 hours ago)
        
 (HTM) web link (interrupt.memfault.com)
 (TXT) w3m dump (interrupt.memfault.com)
        
       | de_aztec wrote:
       | Since it's not clear from the title: embedded here does not mean
       | sqlite-inside-your-app but small devices with microcontrollers
        
         | hgyjnbdet wrote:
         | That's what I initially thought too, though having read and
         | understood little of the article, is sqlite not a consideration
         | too? Or does it take up too much memory?
        
           | vitaminCPP wrote:
           | SQLite would require a file system. Embedded systems
           | typically do not have this.
        
             | ComputerGuru wrote:
             | SQLite doesn't require a file system per se, as you can
             | create an in-memory db and then use vfs to load/write the
             | actual data from/to your IO layer.
        
             | interroboink wrote:
             | Though note you can define your own "Virtual File System"
             | (VFS) for SQLite to use[1]. It might be a bit of an
             | undertaking, but I think you could use that to run on just
             | about anything with storage you control. (I haven't done it
             | myself, just some research back in the day)
             | 
             | [1] https://www.sqlite.org/c3ref/vfs_find.html
        
           | tyingq wrote:
           | There's a pretty good write up on running sqlite on an STM32
           | here: https://github.com/luismeruje/SQLite-STM32
           | 
           | Keeping in mind that STM32 is on the high end of what you
           | might call embedded, and the writeup is pretty clear that it
           | barely fits/works there, lots of caveats, etc.
        
       | ComputerGuru wrote:
       | The chosen solution isn't ideal for all cases, especially those
       | with many or large records. It doesn't require _parsing_ all the
       | bytes of a payload but it does require reading them (because it's
       | a sequential scan of all content).
       | 
       | A better solution would separate the keys from the values (a la
       | MFT) because reading a page of flash is going to be the slowest
       | step. If you only have to sequentially scan the header table, you
       | have to read an order of magnitude or more less data to find the
       | record you are searching for.
        
         | foota wrote:
         | A columnar database format might perform well out of the box in
         | that case.
         | 
         | Reading more, it seems like a columnular database built on LSM
         | trees would probably work great, since you could easily move
         | the log around in flash to distribute write wear out, and only
         | occasionally compact LSM tree files.
        
           | ComputerGuru wrote:
           | It would, but there are any small and simple enough to use in
           | a standalone (no-OS, non-POSIX) environment?
        
       | lights0123 wrote:
       | > However, if you compile on an AWS EC2 instance (as you may want
       | to do if there is a cloud component to your fishtank), then you
       | get a 16-byte structure because int there is 8 bytes
       | 
       | I'm not familiar with any OS that EC2 would offer that uses an
       | 8-byte int. A better example might be that _long_ is the size of
       | a pointer on Unix and microcontroller systems, but always 4 bytes
       | on Windows.
        
       ___________________________________________________________________
       (page generated 2024-08-02 23:00 UTC)