[HN Gopher] Fq: Jq for Binary Formats
___________________________________________________________________
Fq: Jq for Binary Formats
Author : philosopher1234
Score : 400 points
Date : 2021-12-23 00:04 UTC (22 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| AzzieElbab wrote:
| I am sure it is awesome but the name of this utility is somewhat
| unfortunate
| pokstad wrote:
| Don't see the issue. I naturally pronounce this "eff-queue".
| You've really got to work hard to make it vulgar.
| CyberShadow wrote:
| This related project, on the other hand, embraced it (for
| better or for worse):
|
| https://github.com/jzelinskie/faq
| coldtea wrote:
| Only in puritan cultures
| wwader wrote:
| :) i didn't choose it to be provocative or so, apologies if
| that is the case. I've always pronounced jq yay-queue so fq is
| eff-queue for me. Also f and q can be written with one hand on
| qwerty which is nice and quick
| zeroimpl wrote:
| I'd have gone with bq myself
| dinvlad wrote:
| D'oh! Big ol BigQuery CLI would like to have a word
| snidane wrote:
| then bbq.
| fullstop wrote:
| I know that jokes don't go over well here, but...
|
| I'd make an improved one, one which is better. And I'd name
| it bbq.
| hinkley wrote:
| And I will write a fuzzer for bbq called omgwtf.
| aaron_m04 wrote:
| Yeah, two syllables is too long. Let's just sound it out...
| philsnow wrote:
| ... 'feek' ?
|
| If it were 'fk' sure, but the Q on the end makes me think of
| all the English words that come from French and end in
| 'ique', like technique. 'fq' looks like 'feek' to me.
| zx2c4 wrote:
| Relatedly, check out GNU Poke: http://www.jemarch.net/poke
| pabs3 wrote:
| Also Kaitai Struct:
|
| https://kaitai.io/
|
| And the other things mentioned in the fq README:
|
| https://github.com/HexFiend/HexFiend
| https://github.com/binspector/binspector
| dloss wrote:
| Nice! Some other tools and parsers:
| https://github.com/dloss/binary-parsing
| wwader wrote:
| Lots of tools i didn't know about, thanks
| pabs3 wrote:
| You may want to rename that awesome-binary-parsing, having
| awesome in the URL helps in some circumstances.
| pdpi wrote:
| It's interesting how people work. Seeing that section of the
| readme with a laundry list of alternatives made me want to
| try fq even more. It tells you that the author actually cares
| about the problem space.
| wwader wrote:
| Hi! yes i'm very interested in binary analysis and decoders
| in general and fq was not built with the intention to
| compete or replace anything. I usually use fq together with
| lots of other tools, they all fill different purposes. The
| more the merrier!
| pabs3 wrote:
| The listing of alternatives in the README really should be
| standard practice for open source projects. OTOH, some
| maintainers don't like to do that when they haven't
| evaluated the projects. Perhaps they could still add them
| with a disclaimer though.
| nitrogen wrote:
| This looks _incredible_. I 'm on my phone so I haven't tried
| this, but it looks like this supports slicing into MP3
| bitstreams? That would have saved me a month of research and tons
| of development back in 2013.
| wwader wrote:
| Hi, it depends a bit, if the mp3 stream uses bit reservoir it
| might be tricky to to "pure" remuxing with any tool. fq:s
| mp3_frame decoder do try to know what bits are parts of the
| current frame or part of a future frame but not sure how much
| that helps. If the stream does not use reservoir you should be
| able to slice using fq '.frames[100:200][]' file.mp3 >
| sliced.mp3 or something similar.
| chris_wot wrote:
| So is this meant for _any_ binary formats?
| kitd wrote:
| Supported formats:
|
| https://github.com/wader/fq/blob/master/doc/formats.md
|
| They should probably make this a bit more prominent. It's an
| impressive list for a new project.
| heinrich5991 wrote:
| Permanent link (press 'y' on any Github link): https://github
| .com/wader/fq/blob/eb4a6fdbd6ef3a09fc59802e96e...
| wwader wrote:
| Thanks! any suggestion how to make it more prominent?
| linuxdude314 wrote:
| Of course not. It can only work on formats that the team head
| written parsers for.
| chris_wot wrote:
| I ask because I'd be interested in helping write an EMF+
| filter
| rav wrote:
| Does this support out-of-tree format decoders? From an initial
| glance it looks like all decoders are in-tree and written in
| golang. We have a lot of internal binary formats at $WORK that I
| would like to use this on...
| masklinn wrote:
| Seems unlikely since the decoders are defined not just in-tree
| but in "host" code, the definitions are neither data-driven nor
| a DSL.
|
| So it would require some sort of native (Go) plugins system,
| which I understand is about as bad as in Rust owing to there
| being no standard ABI (or plugins system for that matter).
|
| therefore the way to have bespoke / internal formats would be
| to maintain an internal fork of the tool.
| akavel wrote:
| _" (...) some sort of native (Go) plugins system (...)"_
|
| See: https://pkg.go.dev/plugin
| wwader wrote:
| Hi! yes it's kind of support but in a very go:ish at the
| moment. You can use fq as submodule, import/register your own
| format decoders and then run cli.Main. More or less what
| https://github.com/wader/fq/blob/master/fq.go does. I have
| private version of fq for work with some proprietary formats
| that does this and it works great. One issue is that the
| decoder and format API might change, not sure i can give any
| stability guarantees atm and i want to evolve a bit more. Also
| it would be great to be able to hook into existing formats more
| in some way.
|
| In the future i hope to support writing decoders in jq and or
| support some declarative format like kaitai.
| userbinator wrote:
| For something that is supposed to be an analog of jq, there is a
| notable omission from the list of formats: ASN.1.
| wwader wrote:
| Hi, here is issue related to this where i explain a bit what
| would be required https://github.com/wader/fq/issues/20 and how
| protobuf support currently works.
| dikei wrote:
| Can't blame him. ASN.1 is one of the most complicated binary
| formats, with so many encoding rules that there's no free
| decoder that can process them all.
| XAMPPRocky wrote:
| Shameless plug, but you may be interested in my library
| (which is MIT/Apache-2.0) that offers decoding from
| BER/DER/CER all from a single model in code, there's no
| UPER/APER support at the moment, but it's coming in the next
| few months. :)
|
| https://github.com/XAMPPRocky/rasn
| chii wrote:
| Except for interoperability with existing systems, is there
| any reason why anyone would use this ASN.1 protocol/format?
|
| What's it good for?
| IshKebab wrote:
| No. It mostly exists so that people who haven't tried to
| use it can tell other people that they should have used
| ASN.1.
| dikei wrote:
| In my opinion, avoid ASN.1 if you can.
|
| There's a reason why all the cool companies invented their
| own serialization formats: Google's Protobuf, Facebook's
| Thrift, etc.. even when ASN.1 had been an international
| standard for years: It's too complicated.
| p_l wrote:
| The big part of the reason is combination of NIH with bad
| reputation mostly related to X503 and such rather than
| anything else - hard to advocate for it when the main
| library you can point to is OpenSSL, and most commonly
| known encoding is DER (which has certain _implementation_
| complexity, effectively being sorted BER, which has
| certain important value in cryptography).
|
| Both Protobuf and Thrift evolved from RPC systems that
| possibly started out too simple for ASN.1, combined with
| above issue where good tools were probably commercial and
| expensive (FWIW, my experience also suggests that Thrift
| is _shitty_ rpc system, compared even to Sun /ONC RPC,
| but maybe things changed)
| NavinF wrote:
| ASN.1 is incredibly good for one use case: As a cautionary
| tale against design by committee.
| mtve wrote:
| BER/DER/PEM encodings are mostly quite simple, have very few
| subtle details.
| dikei wrote:
| Yeah, those are the easy one. I worked in Telecom, and
| dealing with unaligned PER is a PITA.
| naikrovek wrote:
| note that the tool linked is for binary files. ASN.1 is text,
| isn't it?
| stormbrew wrote:
| I assume they meant the binary encodings of asn.1 like
| BER/DER.
| crehn wrote:
| And tangentially PEM, which is Base64-encoded DER.
| fanf2 wrote:
| That's what I thought until recently, but it turns out
| that PEM refers to just base64 wrapped with ----BEGIN----
| and ----END---- lines, and the encapsulated data does not
| have to be DER.
|
| https://datatracker.ietf.org/doc/html/rfc7468
| rahimiali wrote:
| It's interesting to see how they introduce a new binary format in
| their catalogue. I was expected to find a domain specific
| language to define the grammar of binary bitstreams, maybe as a
| context free grammar. Instead, they built a nice library of
| routines that helps them design custom parsers by hand for each
| new format.
| [deleted]
| mongol wrote:
| I wonder if it would support also non-binary formats. The tool
| could evolve to handle json, yaml, xml, ini etc...
| wwader wrote:
| Hi, fq actually do support json so other similar text formats
| could work the same way. But it's currently implemented in a
| big hacky way, it's just a big blob that happens to be work
| as normal JSON. I've done some attempts at implementing it as
| a normal fq decoder but it's hard to figure out a way to
| represent the whitspace between values etc, ends up very
| clunky or not very user friendly. Any suggestions are very
| welcomed.
| Cayde-6 wrote:
| Not to be confused with https://github.com/circonus-labs/fq, the
| message queue.
| arendtio wrote:
| I am alternating between WOW and wtf. Pretty cool stuff.
|
| I just wonder how on earth you want to be able to support all the
| binary formats out there. I mean, jq supports json, not all
| structured text data, like json, xml, csv, ini, ...
| code-faster wrote:
| https://github.com/tyleradams/json-toolkit
|
| Convert json <-> xml, csv, yaml, logfmt
|
| So to support all formats, you write a binary <-> json
| converter.
| luto wrote:
| apparently they went ahead and implemented a bunch of them in
| Go: https://github.com/wader/fq/tree/master/format
| wwader wrote:
| Hi, i can give some background how i ended up with go instead
| of using something more declarative. Maybe 1.5 years ago i
| start to prototype different approaches for what query
| language to use (sql, jsonpath, my own basic jq version and
| few more) and what language to implement decoders in (lisp,
| kaitai, tcl, "scripted" go, normal go and some more). What i
| found was that for my use cases, detailed parsing of big
| media files, anything scripted was just too slow. I did look
| into translating kaitai etc into something compiled which
| would probably be fast, but next on my list was i wanted was
| to be able to select and decode subformats in quite
| complicated ways (like mp4 samples), flexible ways of demux
| and join blob to decode, calculate checksums, samples counts
| in various way. All felt clunky or hard to fit into a purely
| declarative description. But i was also biased towards go as
| i had good experience using it and know that it would
| probably be fast enough (turn out smart memory usage is
| probably the main speed factor for fq when you keep track of
| lots of things). Also it would provide good tooling like IDE
| support, refactoring (gopls gofmt -r, rf) and it's a
| reasonably strongly typed language i think. Last but not
| least the quick build times really fits my way of working,
| usually use lots of watchexec etc. For query language i
| didn't prototype much, i know i really wanted jq as i had
| already used it extensively and know it was very powerful and
| had a terse syntax when working with structured data. I had
| some ideas of maybe using the C-version of jq via bindings or
| somehow let fq be tool that you used like this 'fq file | jq
| ... | fq' but it just felt strange and not very user
| friendly. Then i found gojq and i just felt that i have to
| make it work somehow, even if it would require lots of hard
| work and change to it (see
| https://github.com/wader/gojq/commits/fq, the JQValue change
| it probably to most interesting and support or custom
| iterators/functions that has been merged). And it turned out
| much better than i would expected, large parts becuse gojq's
| code is very nice and author has been very helpful. There is
| more things i would like that talk about but i think this is
| long enough for now :)
|
| But all that said i think you could use kaitai or something
| similar together with fq:s decode API if you want. I also
| have some ideas and plans on supporting writing deocders in
| jq, hopefully will get some time for that next year.
| arendtio wrote:
| Thanks for the extensive reply. I also had some good
| experience with Go so far, so I can understand how you came
| to that point ;-)
| endymi0n wrote:
| Honestly, it all makes sense: The plugin system and open source
| nature makes it really easy to write a definition for the file
| format you want to work on, which will not just leverage the
| whole ecosystem, but benefit everyone.
|
| This is one of the seriously great ideas where I'm thinking:
| How didn't anyone come up with that before?
| ape4 wrote:
| Hopefully json will be superseded someday. Cool tool.
| wwader wrote:
| Interestingly fq works by being kind of a superset of JSON/jq.
| It has types that can behave as jq values when needs but then
| with special functions or key accessors can be something else.
| blondin wrote:
| this is quite an interesting project! combining kaitai structs or
| similar with the command line.
|
| however, i am a little disappointed that the jq syntax was
| chosen. jq has a very non-intuitive syntax. there are more
| intuitive query syntaxes out there. (linq or even basic sql come
| to mind.)
| wwader wrote:
| Yes i can empathize with finding jq hard to understand, it's
| quite different and took a while to grasp. The reason i choosed
| it anyway was that after prototyping some common type of
| queries i would like to do (basic value access in deep
| structures, multiple recursive traverse with filtering,
| transform objects and arrays) in various languages, jq was more
| or less then only one that felt terse enough. Also i think it's
| quite nice that you can output to JSON and then load into
| whatever language or environment you want Maybe there are some
| alternatives i should look at?
| nine_k wrote:
| Linq is a bit more wordy, and SQL is sadly not very composable.
| leshenka wrote:
| This looks like wireshark's panel for inspecting packets
| lido wrote:
| > This project would not have been possible without itchyny's jq
| implementation gojq.
|
| Another approach is to take the convert binary to object part of
| your code, output that as JSON on stdout and feed that into jq.
|
| Basically, a binary front end + jq = fq
| jrockway wrote:
| I wrote a small script to convert CSVs to JSON strictly to use
| jq on the output. Querying things like your GCP bill with jq is
| quite enjoyable.
|
| gojq is also nice. I work with a lot of structured logs and
| wrapped jq with a little bit of format-understanding and output
| sugar to make looking at and analyzing such logs an enjoyable
| experience: https://github.com/jrockway/json-logs
| awild wrote:
| Miller Csv can process json in record format and has a much
| saner DSL in my experience.
|
| https://github.com/johnkerl/miller
| darrenf wrote:
| > _I wrote a small script to convert CSVs to JSON strictly to
| use jq on the output_
|
| Note that you can use jq to consume simple CSVs (and produce
| them) without anything else. There's an entry in the cookbook
| wiki https://github.com/stedolan/jq/wiki/Cookbook#convert-a-
| csv-f... - I posted some usage examples a few months back
| https://news.ycombinator.com/item?id=27379423
| cglong wrote:
| I wish this was only the binary front end so I could pick my
| parser (e.g. PowerShell). I see fq seems to support sending the
| whole JSON to stdout; I wonder if there's a way to make this
| the default behavior: # JSON for whole file
| fq tovalue file
| donio wrote:
| It would be hard to get the full fq functionality that way. How
| would you encode the data in a way so that you can do both:
| .frames[100].header.sample_rate
|
| for the individual field and
| .frames[100].header|tobytes[:0x10]
|
| for the first few bytes of the entire header structure?
|
| Or decode a binary slice as a particular format:
| tobytes[0x234:0x325]|avc_sps.max_num_ref_frames
| wwader wrote:
| Hi, i wrote a bit about this in my reply above
| https://news.ycombinator.com/item?id=29661575
| pxeger1 wrote:
| Well fq too!
| lindig wrote:
| I'd like to see support for FIT files as they are emitted by
| Garmin fitness devices. It's a clever binary format that in-
| stream defines the format of records which then contain the
| actual measurements which may be scaled for more compact
| representation. These multiple layers make the format not obvious
| to parse but the tool supports already an impressive list of
| formats that probably use similar techniques.
| buryat wrote:
| why not bq?
| gnubison wrote:
| Interesting project. Unfortunate that its name conflicts with one
| of nq's executables (https://github.com/leahneukirchen/nq), but
| I'm not sure anything can be done about it.
| benatkin wrote:
| IMO ones that only have one non-prefixed executable take
| precedence over ones that don't, even when the one with
| multiple non-prefixed executables is older.
| [deleted]
| hawk_ wrote:
| It says it supports protobuf. Is there a protobuf file format
| i.e. for multiple records or do they mean a single protobuf
| record file?
| sandermvanvliet wrote:
| Typically people separate protobuf messages by a length value
| of the next message. Perhaps that's what they did.
|
| Also protobuf can contain embedded messages or even just the
| binary representation of a list of embedded messages.
| wwader wrote:
| Hi, currently the protobuf support can either decode the
| wire-format or in some cases a format decoder uses protobuf
| as subformat and passes it a "schema" so it can do some more
| fancy decoding. But yes it would be interesting adding
| support for reading protobuf schemas somehow.
| Cloudef wrote:
| I wrote protobuf parser in ragel for work.. its still used
| to replace reflection as c++ protobuf implementation
| explodes our binaries to huge sizes.
___________________________________________________________________
(page generated 2021-12-23 23:02 UTC)