[HN Gopher] Show HN: WAL Implementation in Golang
       ___________________________________________________________________
        
       Show HN: WAL Implementation in Golang
        
       I wrote this simple WAL library in Golang that I use to write data
       that my kafka producer fails due to errors like Broker going down
       or some other issue. Took inspiration from etcd/wal
        
       Author : stym06
       Score  : 27 points
       Date   : 2024-07-08 18:50 UTC (4 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | eatonphil wrote:
       | Did I miss it or is there no call to os.File.Sync(), i.e. fsync,
       | anywhere?
       | 
       | Since you mention etcd/wal:
       | 
       | https://github.com/etcd-io/etcd/blob/v3.3.27/wal/wal.go#L671
       | 
       | https://github.com/etcd-io/etcd/blob/v3.3.27/pkg/fileutil/sy...
        
         | stym06 wrote:
         | I used the bufio flush mechanism
         | https://pkg.go.dev/bufio#Writer.Flush
         | 
         | Thanks for your comment, I'll definitely check it out. It was
         | my first attempt at this. How can I make it better?
        
           | sakras wrote:
           | Essentially, unless you `fsync`, there's no guarantee that
           | your data will be durably written to disk. This is because
           | the operating system keeps data buffered in in-memory caches,
           | so if the machine crashes you may lose some data. The `fsync`
           | system call forces the data to be flushed from the in-memory
           | OS cache to the disk. As far as I could tell, the Flush you
           | use does not `fsync`.
        
             | stym06 wrote:
             | Thanks for your input @sakras. I'll fix this
        
       | tjungblut wrote:
       | Besides what Phil mentioned below, I can't write more than one
       | record to the WAL. You're closing the file after every write, the
       | second time you write the error `seek data/rebuf.tmp: file
       | already closed` is returned.
       | 
       | I also think your rotation will delete the wrong segment when you
       | have more than ten segments - imagine you're writing rebuf-1 to
       | rebuf-10 - what's the "oldest file" to delete now? Besides,
       | should you really delete those files?
        
         | stym06 wrote:
         | Yes there are a lot of bugs since I just wrote this in one
         | sitting today. Will be fixing all of this. For log rotation,
         | I'll sort by the last_modified_at ts and then purge those
        
           | tjungblut wrote:
           | Your generational approach to segment numbering is fine, if
           | you prepend enough zeros to format the files properly then
           | you're also able to sort them correctly. etcd uses the same
           | trick.
        
       | stym06 wrote:
       | OP here! Pls feel free to raise any bugs you encounter! I'll be
       | doing the following immmediate fixes:
       | 
       | 1. Use fsync for durable writes in case of system crashes
       | 
       | 2. Fix log-rotation-purging logic
       | 
       | 3. Fix `file already closed` bug on consecutive writes
       | 
       | 4. Add CRC checksum
        
       | neonsunset wrote:
       | This reminds me of ZoneTree which is persistent LSM tree project
       | based on top of WAL, written in C#:
       | https://github.com/koculu/ZoneTree
       | 
       | Similar to RocksDB.
        
       | Smaug123 wrote:
       | This is one of the absolutely classic cases where I'd expect a
       | very small amount of property-based testing to flush out a very
       | large number of bugs, by the way.
        
       | Smaug123 wrote:
       | golangci-lint finds three errors in rebuf.go at commit 615209d.
       | It's never safe to write golang without the linters!
        
       ___________________________________________________________________
       (page generated 2024-07-08 23:01 UTC)