[HN Gopher] Erasure Coding for Distributed Systems
       ___________________________________________________________________
        
       Erasure Coding for Distributed Systems
        
       Author : eatonphil
       Score  : 47 points
       Date   : 2024-08-26 20:03 UTC (2 hours ago)
        
 (HTM) web link (transactional.blog)
 (TXT) w3m dump (transactional.blog)
        
       | hinkley wrote:
       | Years back someone proposed a cute algorithm for erasure codes
       | that depended not on spinning rust but on multipath networking.
       | 
       | I believe they called it network coding and the idea was in a
       | network with multiple routes I might get a file faster by pulling
       | an erasure code that used two parts of the file or even two files
       | from one upstream instead of waiting for the entire file from the
       | primary server.
        
       | wahern wrote:
       | Has anybody used Wirehair in a project?
       | https://github.com/catid/wirehair
       | 
       | I'm curious if it's well-defined enough to base a standard around
       | --informally if not formally--for building a large file
       | archiving/data recovery project I've been mulling over for nearly
       | 10 years. It's the only large block erasure code I've found that
       | has both the ideal (or nearly ideal) algorithmic performance
       | _and_ API. That makes it a nice blackbox for my use case, unlike
       | something like RaptorQ, which leaks little details all over the
       | place, driving up the complexity and rigidness of the rest of the
       | stack. But Wirehair isn 't a spec, it's an (experimental?)
       | implementation of an idea. It seems stable, but unless/until I
       | try writing a second implementation, or it's seen substantial use
       | (exposing any sharp edges in the algorithm) I worry how easily it
       | would translate to a reliable specification or second
       | implementation.
        
       ___________________________________________________________________
       (page generated 2024-08-26 23:00 UTC)