[HN Gopher] Deserializing Binary Data Files in Rust
       ___________________________________________________________________
        
       Deserializing Binary Data Files in Rust
        
       Author : lukastyrychtr
       Score  : 25 points
       Date   : 2021-06-25 07:34 UTC (3 days ago)
        
 (HTM) web link (adventures.michaelfbryan.com)
 (TXT) w3m dump (adventures.michaelfbryan.com)
        
       | crateless wrote:
       | > We get lucky here because the flags field is at offset 202
       | 
       | How does one calculate this offset based on the fields in the
       | struct?
        
       | Labo333 wrote:
       | For such a problem, I really recommend https://kaitai.io/
       | 
       | From their website:
       | 
       | > Kaitai Struct is a declarative language used to describe
       | various binary data structures, laid out in files or in memory:
       | i.e. binary file formats, network stream packet formats, etc.
       | 
       | They even have a Rust interface: https://github.com/kaitai-
       | io/kaitai_struct_rust_runtime
        
         | rudibe wrote:
         | A native Rust solution, with meta-programming:
         | https://crates.io/crates/packed_struct (written by me)
        
       | coldnose wrote:
       | I really wish there was a safe way to initialize structs with
       | arbitrary binary data in rust. Perhaps I'm missing something, but
       | it seems intuitively safe, especially if the struct only contains
       | integer primitives.
        
         | sophacles wrote:
         | How would it be possible to make it safe? The compiler can't
         | verify that N random bytes input from an arbitrary source are
         | going to be valid for the types in the struct - you have to
         | tell tell the compiler "trust this even if you can't verify it
         | from the source code" - hence the unsafe keyword.
        
           | coldnose wrote:
           | For integer primitives, all possible values are valid. Just
           | allowing binary initialization for structs that consist of
           | integer types would allow compatibility with C structure. As
           | it is, I have to either resort to unsafe code or manually
           | deserialize every struct
        
           | Alexendoo wrote:
           | Enter safe transmute - https://github.com/rust-lang/project-
           | safe-transmute/blob/mas...
        
       | mirekrusin wrote:
       | I think c struct is missing alignment attribute? This idea has
       | been explored in flatbuffers, capnproto and probably others.
        
         | masklinn wrote:
         | C structs are generally assumed to be laid out in the order
         | defined with proper padding automatically inserted (as most
         | platforms at best dislike unaligned accesses). Alignment
         | information should only be necessary if you want to pack the
         | structure or need to apply wider than standard alignment (e.g.
         | need to align a 32 or 64b type to 128 bytes).
         | 
         | The one possible issue is that reading padding is UB (i think).
         | But here there should be no padding: the struct is a pair
         | number of bytes (alignment 1) and a word (alignment 2), which
         | us aligned due to the previous, and the struct is if alignment
         | 2.
        
           | gameswithgo wrote:
           | yeah C will be order defined in code, rust will reorder to
           | save space unless you attribute it not to. interesting trivia
           | a blind guy wrote the reordering code. if anyone knows his
           | name please post.
        
             | masklinn wrote:
             | The rust structure is defined as repr(C) so there is no
             | reordering.
        
               | sumtechguy wrote:
               | Does the C spec actually specify it needs to be in that
               | order? Been awhile since I messed that that sort of
               | thing. I know in practice they typically just put them in
               | the same order and align on some sort of default packing
               | depending on CPU. Usually to minimize on extra
               | instructions. I have eeked out some extra perf on some
               | CPUs and structs just by moving stuff around and making
               | sure it fits into a cacheline. Sometimes the extra
               | instructions are worth it to unpack some vars if you can
               | get the data into a cacheline. But that depends on your
               | data and usage.
        
               | kzrdude wrote:
               | Yes, it does
        
             | kzrdude wrote:
             | Implemented in https://github.com/rust-lang/rust/pull/37429
             | ; his website seems to be https://ahicks.io/
        
           | diath wrote:
           | That's his point, you want to 1-align the struct to disable
           | the padding to ensure the size of the structure is portable.
        
           | alfalfasprout wrote:
           | Wider than standard alignment can be pretty important though.
           | Especially for usage with SIMD instructions.
        
       ___________________________________________________________________
       (page generated 2021-06-28 23:02 UTC)