[HN Gopher] Kaitai Struct: declarative binary format parsing lan...
___________________________________________________________________
Kaitai Struct: declarative binary format parsing language
Author : djoldman
Score : 59 points
Date : 2025-10-14 14:51 UTC (9 days ago)
(HTM) web link (kaitai.io)
(TXT) w3m dump (kaitai.io)
| zzlk wrote:
| I wanted to use this a long time ago but the rust support wasn't
| there. I can see now that it's on the front page with apparently
| first class support so looks like I can give it a go again.
| jdp wrote:
| I also like Protodata [1]. It's complementary as an exploration
| and transformation tool when working with binary data formats.
|
| [1]: https://github.com/evincarofautumn/protodata
| woodruffw wrote:
| Kaitai Struct is really great. I've used it several times over
| the years to quickly pull in a parser that I'd otherwise have to
| hand-roll (and almost certainly get subtly wrong).
|
| Their reference parsers for Mach-O and DER work quite nicely in
| abi3audit[1].
|
| [1]:
| https://github.com/pypa/abi3audit/tree/main/abi3audit/_vendo...
| theLiminator wrote:
| Is the main difference from https://github.com/google/wuffs being
| that Kaitai is declarative?
| setheron wrote:
| Looking at that repo.. i have no clue how to get started.
| Sesse__ wrote:
| They overlap, but none does strictly more than the other.
|
| Kaitai is for describing, encoding and decoding file formats.
| Wuffs is for decoding images (which includes decoding certain
| file formats). Kaitai is multi-language, Wuffs compiles to C
| only. If you wrote a parser for PNGs, your Kaitai
| implementation could tell you what the resolution was, where
| the palette information was (if any), what the comments look
| like and on what byte the compressed pixel chunk started. Your
| Wuffs implementation would give you back the decoded pixels
| (OK, and the resolution).
|
| Think of Kaitai as an IDL generator for file formats, perhaps.
| It lets you parse the file into some sort of language-native
| struct (say, a series of nested objects) but doesn't try to
| process it beyond the parse.
| nigeltao wrote:
| See https://github.com/google/wuffs/blob/main/doc/related-
| work.m...
|
| > Kaitai Struct is in a similar space, generating safe parsers
| for multiple target programming languages from one declarative
| specification. Again, Wuffs differs in that it is a complete
| (and performant) end to end implementation, not just for the
| structured parts of a file format. Repeating a point in the
| previous paragraph, the difficulty in decoding the GIF format
| isn't in the regularly-expressible part of the format, it's in
| the LZW compression. Kaitai's GIF parser returns the compressed
| LZW data as an opaque blob.
|
| Taking PNG as an example, Kaitai will tell you the image's
| metadata (including width and height) and that the compressed
| pixels are in the such-and-such part of the file. But unlike
| Wuffs, Kaitai doesn't actually decode the compressed pixels.
|
| (I am the Wuffs author.)
| mturk wrote:
| Kaitai is absolutely one of my favorite projects. I use it for
| work (parsing scientific formats, prototyping and exploring those
| formats, etc) as well as for fun (reverse engineering games,
| formats for DOSbox core dumps, etc).
|
| I gave a guest lecture in a friend's class last week where we
| used Kaitai to back out the file format used in "Where in Time is
| Carmen Sandiego" and it was a total blast. (For me. Not sure that
| the class agreed? Maybe.) The Web IDE made this super easy --
| https://ide.kaitai.io/ .
|
| (On my youtube page I've got recordings of streams where I work
| with Kaitai to do projects like these, but somehow I am not able
| to work up the courage to link them here.)
| setheron wrote:
| Great timing! I just published https://github.com/fzakaria/nix-
| nar-kaitai-spec and contributed kaitai C++ STL runtime to nixpkgs
| https://github.com/NixOS/nixpkgs/pull/454243
| layoric wrote:
| I discovered this project recently and used it for Himawari
| Standard Data format and it made it so much easier. Definitely
| recommend using this if you need to create binary readers for
| uncommon formats.
| okanat wrote:
| Even if you don't want to use it since it is not as efficient as
| a hand-written specialized parser, Kaitai Struct gives a perfect
| way of documenting file formats. I love the idea and every bit of
| the project!
| sitkack wrote:
| What was the Python based binary parsing library from around
| 2010? Hachoir?
|
| https://hachoir.readthedocs.io/en/latest/index.html
| ctoth wrote:
| Construct?
| ginko wrote:
| No pure C backend?
| imtringued wrote:
| https://en.wikipedia.org/wiki/Data_Format_Description_Langua...
|
| DFDL is heavily encroaching on Kaitai structs territory.
| dgan wrote:
| Wow this is good. My only complaint is annoyingly verbose yaml.
| What if I would like to use Kaitai instead of protobuffs, my
| .proto file is already a thousand lines, splitting each od these
| lines into 3-4 yaml indented lines is hurting readability
___________________________________________________________________
(page generated 2025-10-23 23:00 UTC)