[HN Gopher] Why isn't there a Swagger/OpenAPI for binary formats?
___________________________________________________________________
Why isn't there a Swagger/OpenAPI for binary formats?
Author : woodrowbarlow
Score : 35 points
Date : 2022-03-25 19:13 UTC (3 hours ago)
(HTM) web link (anachronauts.club)
(TXT) w3m dump (anachronauts.club)
| jayd16 wrote:
| Does COM count?
| onetimeuse74 wrote:
| you are looking for katai struct: https://kaitai.io/
|
| katai allows you to declaratively define a binary format bit for
| bit - they even have an IDE
| Khelavaster wrote:
| It's SignalR or WCF with binary serialization!
| ysleepy wrote:
| Automatic generation of wireshark dissectors would be amazing.
|
| I wrote a few in Lua, but it still felt second class citizen to
| built-in C dissectors, loading them was annoying and the UI to
| use them was too.
|
| A good dissector will annotate fields with semantic meaning and
| group and nest them, giving wireshark the precise bit
| offset+length where a value came from.
|
| Those are some different requirements than other parsers and
| reuse is therefore a bit difficult.
| woodrowbarlow wrote:
| preach! by the way: wireshark can load ASN.1 source files
| directly.
| https://www.wireshark.org/docs/wsdg_html_chunked/CreatingAsn...
| Flocular wrote:
| DFDL ("Daffodil")
| https://en.wikipedia.org/wiki/Data_Format_Description_Langua...
| was made pretty much exactly for describing/parsing binary
| formats in a schema-like format. It is however based on XML and
| not JSON.
| pjmlp wrote:
| Hello DCE RPC, SOM, COM, CORBA,.....
| senko wrote:
| Well, there's ASN.1: https://en.m.wikipedia.org/wiki/ASN.1
|
| Welcome to hell, abandon all hope, etc.
| woodrowbarlow wrote:
| (author here, the blog is under an alias) ASN.1 is in a similar
| category to protobuf: it doesn't allow definition of the
| physical representation of the data. (it does support multiple
| serialization profiles, but ultimately ASN.1 data can only be
| serialized in a few different ways.) you can't write an ASN.1
| file to parse arbitrary binary formats found in the wild.
| synack wrote:
| RecordFlux https://github.com/Componolit/RecordFlux
| woodrowbarlow wrote:
| (author here, the blog is under an alias) oh, interesting! i
| hadn't seen this one yet. i think this one would fall under the
| category "transpile-first languages" (like wuffs or spicy);
| i.e. it is useful for portable parsers but faces obstacles for
| generating ancillary tooling.
| octagons wrote:
| Tangential to the rest of the conversation, but I often use/write
| binary templates [0] for 010 Editor [1]. The GUI-driven approach
| to navigating structures (such as PE headers) can be very useful.
|
| I wish there was a similar tool that allowed me to simply
| highlight and bookmark blocks of binary data when I'm dealing
| with an unknown file format, such as a firmware dump for
| something exotic and/or proprietary.
|
| [0] https://www.sweetscape.com/010editor/templates.html [1]
| https://www.sweetscape.com/010editor/
| emteycz wrote:
| Protobuf, MessagePack?
| woodrowbarlow wrote:
| (author here, the blog is under an alias) those are
| serialization formats, they don't allow for defining the
| physical representation of the data. for instance, you can't
| write a protobuf schema that de-serializes a .gif header.
| alophawen wrote:
| This interests me a lot. If blog author reads this, is there some
| way to reach you in pm?
| woodrowbarlow wrote:
| that's me, please see my profile. :) (i've added an edit on the
| bottom of the article)
| alophawen wrote:
| You got mail
| defanor wrote:
| I think OpenAPI is a rather arbitrary and approximate description
| of models built upon HTTP with JSON, but that's constrained
| enough, and the problems solved are often similar enough, for
| such a solution to be useful. Actually maybe JSON schema would be
| more useful to consider here, since that is about formats, rather
| than interfaces or protocols.
|
| With binary formats, on the other hand, there's neither data
| model nor usage constraints like that, and a generic tool to
| handle all the things would be just the Turing machine -- and
| then we have a bunch of languages for that. As for kaitai struct
| and similar attempts to handle arbitrary binary formats, I hear
| of those from time to time, and in some cases they seem to be
| useful, but they too should be either Turing-complete (and then
| it probably makes more sense to use general-purpose languages) or
| would fail to be useful occasionally.
| NovemberWhiskey wrote:
| This seems obviously to be the right answer to me.
|
| Think about handling a ZIP file: you can't start at the
| beginning of the file and work forward from there. No; you have
| to start at/near the end of file and scan for an end-of-
| central-directory record based on a signature (it's not at a
| fixed offset due to a variable-length comment field). That then
| enables you to find the actual offset for the central directory
| itself; which in turns allow you to find the files themselves.
| woodrowbarlow wrote:
| (author here) aye, this is a crucial point. any description
| language for binary formats quickly approaches turing complete,
| and i think this is what leads to so many of the "transpile-
| first parser languages". i'm impressed how well kaitai has
| managed to wrangle that tendency towards turing-completeness;
| they do have an embedded expression language for attribute
| binding, and they do have 'repeat' and 'switch' constructs, but
| the language is astoundingly declarative.
|
| i think something very similar to kaitai could work if the
| tools to work with the files programmatically were ergonomic
| and available in many different languages. e.g., if the kaitai
| compiler were a native compiled library with lots of packaged
| FFI wrappers for many languages.
| defanor wrote:
| Well, nothing stops Turing-complete languages from being
| declarative: lambda calculus is both, for instance, as well
| as some of the languages based on it (in particular parser
| combinators in Haskell and similar languages are neat and
| flexible).
|
| And there's indeed a practice of embedding some languages
| into others, though I think it ends up quite awkward to use,
| and in this case potential limitations (actually I don't know
| if kaitai is Turing-complete) may be undesirable. I imagine
| in many cases it may be simpler to ship, say, a
| parser/decoder (and/or printer/encoder) library providing a C
| API that is easy to call via FFI, rather than a description
| in a less common language, to be used with a particular
| library. But then again, probably that'd have its uses too.
| [deleted]
| nemetroid wrote:
| > both of these tools are only interested in allowing the author
| to model the logical representation of the data, not the
| physical. they are useful if you have data and want to
| automatically convert that into a file format or protocol, but
| not useful if you want to describe an existing file format or
| protocol.
|
| OpenAPI isn't necessarily great at this either. If your JSON API
| wasn't designed with OpenAPI in mind (or in a way that's similar
| to typical OpenAPI designs), it might not be possible to write an
| OpenAPI spec that describes the format _well_. But "type:
| object" is valid, so it's always possible to write a "spec".
|
| I think the question ("why is there no schema language for binary
| formats?") is valid, but that OpenAPI isn't as encompassing as
| it's laid out to be.
| emj wrote:
| Yes and that's a fundamentally different problem; I think this
| is because there is always some kind of state in a protocol.
| SOAP has this problem, and also YANG[1] and tools that try to
| replace SNMP. You can never abstract away the state that the
| consumer and producer of the protocol has. There are lots of
| tools to disect, debug, resend and parse network protocols like
| IP and TCP but in those protocols we know there is a state
| built in that can not be broken, so we do not build OpenApi for
| them.
|
| OpenApi specifications that need three or four requests to
| reason with are a lot harder to work with than ones that are
| only one request, and these are different beast they become
| protocols. In my personal experience the binary specification
| often are protocols with a lot more implicit state in them,
| e.g. binary diff of in memory struct etc. You quickly lose the
| usefullness of Swagger the more state you have.
|
| That said there are lots of good example of binary protocols
| that try to be stateless.
|
| Conclussion: This blog post need to be scoped a bit better,
| what kind of binary communications do you want an OpenApi for
| and how is that different from what exists today?
|
| As soon as you have to encode the state of the application in
| the specification of the protocol (like TCP above) things get
| really complicated.
|
| [1] https://en.wikipedia.org/wiki/YANG
| Jtsummers wrote:
| I don't know OpenAPI and only did a quick read. Does it support
| any formats other than JSON and XML? If not, then it hardly seems
| like an appropriate analog to what you're asking for.
|
| Can it support s-expr based formats? Plain text keyword:value
| pairs? Any arbitrary input? Because that's what you're asking for
| for "binary formats". Which is a wide-open field. To have a
| universal system for that you will need something closer to the
| parser description formats that you have already rejected. Just
| like you'd have to do to properly support any arbitrary plain
| text format instead of _just_ JSON and XML.
| [deleted]
| panick21_ wrote:
| gRPC
| woodrowbarlow wrote:
| grpc is a layer on top of protobuf
| randomdata wrote:
| Extending it to define transport concerns, akin to
| Swagger/OpenAPI.
| pianoben wrote:
| In my view, gRPC is the closest "binary" analog to OpenAPI,
| which is a transport-layer concern first and foremost with
| discoverability features.
|
| If your problem space is "a way to assign meaning to
| arbitrary bitfields", I don't feel that OpenAPI is a very
| good comparison. Its purview is "representing web APIs in
| terms of structured JSON/XML payloads". That is, OpenAPI
| _also_ deals in logical and not physical representation.
| (unless it specifies UTF-8 or something - it might, I don 't
| know!)
| woodrowbarlow wrote:
| you know, i think you're right. on reflection, a comparison
| to json schema[1] (rather than openapi) would be more
| appropriate, as mentioned be defanor in another comment.
|
| 1: https://json-schema.org/
| mkl95 wrote:
| GRPC has a reflection protocol [1] that can be used to build
| Swagger style tools [2]
|
| [1] https://github.com/grpc/grpc/blob/master/doc/server-
| reflecti...
|
| [2] https://github.com/grpc-swagger/grpc-swagger
| woodrowbarlow wrote:
| (author here) i did not know about the reflection protocol.
| maybe this could be a model for how kaitai struct could avoid
| the "centralized bottleneck" they have now in terms of
| ecosystem growth.
| divbzero wrote:
| OP might be conflating OpenAPI [1] which specifies HTTP endpoints
| of APIs with JSON Schema [2] which specifies data schemas of JSON
| documents. OpenAPI specs often use the _application /json_ media
| type with schemas specified using JSON Schema, but they can also
| use other media types including binary formats. What OP is really
| asking is: Why isn't there {JSON Schema,XML Schema,.proto files}
| for arbitrary binary formats?
|
| Kaitai [3] mentioned by OP and sibling comments looks
| interesting, don't think I've seen something like it before.
|
| [1]: https://swagger.io/specification/
|
| [2]: https://json-schema.org/
|
| [3]: https://kaitai.io/
| CMCDragonkai wrote:
| I did research into session types to try and solve this exact
| problem. However the theory still new and implementations are
| lacking. Again session types encode the logic, you still need
| some sort of structure/content language to describe the messages.
| Session types should be able to encode complex protocols and
| combined with kaitai, maybe something can emerge.
___________________________________________________________________
(page generated 2022-03-25 23:00 UTC)