[HN Gopher] BuffDB is a Rust library to simplify multi-plexing o...
___________________________________________________________________
BuffDB is a Rust library to simplify multi-plexing on edge devices
Author : opensourcemaxi
Score : 81 points
Date : 2024-08-25 16:34 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| sgt wrote:
| Another local first library. The movement is taking off.
| jakjak123 wrote:
| What does "Simplify multiplexing on edge devices" even mean?
| 01HNNWZ0MV43FF wrote:
| I guess they mean number of frontends times number of backends?
| https://github.com/buffdb/buffdb/issues/19
| big_hacker wrote:
| I think it's a proxy between your software and SQLite with a
| new database API. I guess "multiplexing" is a big word for
| saying you can someday swap SQLite for something else.
|
| The CLI shows key-value store features.
|
| I don't know if this software has real world savings in
| performance. I don't think I would ever use this software.
| jakjak123 wrote:
| Like if you want your frontend to write to both Sqlite and
| RocksDB?
| jakjak123 wrote:
| Is it a protobuf based database store, or is it a database that
| uses grpc as its connection communication? Could be a bit clearer
| from the frontpage
| 38 wrote:
| > protobuf
|
| such an awful format, I wish people would stop using it
| ithkuil wrote:
| Why?
|
| The format is not that bad.
|
| The binding/libraries OTOH are often awful and they often
| require unnecessary full-message deserialization
| 38 wrote:
| not self describing. if it was just the field names I could
| deal with that, but even the values are ambiguous, since
| the same type is used for Bytes and embedded Messages. the
| worst part is the wire type integer has two unused values,
| so they easily could have added a wire type for embedded
| messages
| cryptonector wrote:
| Self-describing is point-less for serializations. There
| is a great deal of history here. ASN.1 has self-
| describing encoding rules such as BER/DER/CER, XER (XML),
| JER (JSON), and GSER (never mind), and it has non-self-
| describing serializations like PER (packed encoring
| rules) and OER (octet encoding rules). XML and JSON are
| self-describing, naturally. FastInfoSet is a PER-based
| fast encoding for XML, because it turns out that XML is
| slow (imagine that). XDR is a non-self-describing
| serialization format that resembles OER but with 4-octet
| alignment. Flat buffers is essentially an OER-ish
| encoding for the same IDL as protobufs, and is much
| better than protobufs.
|
| It would be nice if the next serialization format either
| is truly original or just solves problems that somehow
| none of the _many_ existing schemes do.
|
| How many serialization formats are there? See:
| https://en.wikipedia.org/wiki/Comparison_of_data-
| serializati... (which is NOT a complete list).
| 38 wrote:
| > Self-describing is point-less for serializations
|
| you couldn't be more wrong. what happens when you lose
| the schema, or never had access to it in the first place?
| think from the point of view of reverse engineering
| cryptonector wrote:
| Why would you lose it? NFS never lost its XDR schema, for
| example. Do you have any examples where the schema got
| lost?
| eddd-ddde wrote:
| I imagine when someone is choosing a serialization
| solution they probably don't really care about people
| trying to reverse engineer it... And if they did, they
| would just make schemas available instead.
|
| And if you lose your own schema, then you probably have
| more serious underlying problems.
| 38 wrote:
| just imagine a world where protobuf replaces JSON, would
| you really want that? you're not thinking big picture
| eddd-ddde wrote:
| I actually don't think it'd be so bad.
|
| Ideally it is capnproto instead of protobuf, but in
| general the same applies.
|
| It is a _transport_ protocol. You can still write your
| config or data in any format you like.
| cryptonector wrote:
| Protobuf is a tag-length-value (TLV) encoding. It's bad.
| TLV is the thing that everyone loves to hate about ASN.1's
| DER.
| ithkuil wrote:
| > It's bad
|
| I'm not sure that's a helpful way to look at things.
|
| There are tradeoffs. Can you elaborate more about what
| aspects of a TLV encoding you find problematic? Is it
| decoding speed? The need to copy the encoded value into a
| native value in order to make use of it? Something else?
| cryptonector wrote:
| TLV encodings are always redundant and take more space
| than non-TLV encodings. Therefore they are a
| pessimization. As well, definite-length TLV encodings
| require two passes to encode the data (one to compute the
| length of the data to be encoded, and one to encode it),
| thus they are a) slower than non-TLV binary encodings, b)
| not on-line for encoding.
| ithkuil wrote:
| Yes all those are indeed pessimizations during encoding
| but are features when decoding: the decoder can skip
| decoding fields and tolerate unknown fields.
|
| Now, you may disagree that tolerating unknown fields is a
| features (as many people do), but one must understand the
| context where protobuf has been designed, namely the
| situation where it takes time to roll out new versions of
| binaries that process the data (either in API calls or on
| stored files) and thus the ability to design a schema
| evolution with backward and forward compatibility is
| worth a few more cycles during encoding.
|
| Not all users have that need and hence there exist other
| formats, but I wouldn't dismiss the protobuf encoding as
| flatly "wrong" just because you don't have the
| requirements it has been designed for.
| cryptonector wrote:
| > the decoder can [...] tolerate unknown fields.
|
| See ASN.1's extensibility rules. If you mark a record
| type (SEQUENCE, in ASN.1 parlance) as extensible, then
| when you later add fields (members, in ASN.1 parlance)
| the encoding rules have to make it possible to
| skip/ignore those fields when decoded by software
| implementing the pre-extension type. PER/OER will include
| a length for all the extension fields for this purpose,
| but it can be one length for all the extension fields in
| each round of extensions rather than one per (which would
| only save on the type "tag" in TLV).
|
| > the decoder can skip decoding fields
|
| This is mainly true for on-line decoders that produce
| `{path, leaf value}` tuples _and_ which take paths or
| path filters as arguments.
|
| > Now, you may disagree that tolerating unknown fields is
| a features (as many people do), but one must understand
| the context where protobuf has been designed, namely the
| situation where it takes time to roll out new versions of
| binaries that process the data (either in API calls or on
| stored files) and thus the ability to design a schema
| evolution with backward and forward compatibility is
| worth a few more cycles during encoding.
|
| This is the sort of thing I mean when I complain about
| the ignorant reinvention of the wheel that we all seem to
| engage in. It's natural and easy to do that, but it's not
| really a good thing.
|
| Extensibility, versioning, and many many other issues in
| serialization are largely not new, and have been well-
| known and addressed for decades. ASN.1, for example, had
| no extensibility functionality in the early 1980s, but
| the only encoding rules at the time (BER/DER/CER), being
| TLV encodings, naturally supported extensibility
| ("skipping over unknown fields"). Later formal support
| for extensibility was added to ASN.1 so as to support
| non-TLV encodings.
|
| ASN.1 also has elaborate support for "typed holes", which
| is what is referred to as "references" in [0].
|
| ASN.1 gets a _lot_ of hate, mainly for a)
| its syntax being ugly (true) and hard to parse (true-ish)
| b) the standard being non-free (it used to be non-free,
| but it's free now, in PDF form anyways) c)
| lack of tooling (which is not true anymore).
|
| (c) in particular is silly because if one invents a new
| syntax and encoding rules then one has to write the non-
| existent tooling.
|
| And every time someone re-invents ASN.1 they miss
| important features that they were unaware of.
|
| Meanwhile ASN.1 is pluggable as to encoding rules, and
| it's easy enough to extend the syntax too. So ASN.1
| covers XML and JSON even. There's no other
| syntax/standard that one can say that for!
|
| Next time anyone invents a new syntax and/or encoding
| rules, do please carefully look at what's come before.
| [0] https://en.wikipedia.org/wiki/Comparison_of_data-
| serialization_formats
| ithkuil wrote:
| You make it sound like protobuf was invented yesterday.
| Sure it's not so old like ASN.1 but protobuf now about a
| quarter century old and battle tested as the main
| interchange format for at least one giant company with
| gazillions of projects that needed to interoperate.
|
| One of the design requirements was simplicity and ease of
| implementation and for all the love in the world I can
| muster for ASN.1 I must admit it's far from simple.
|
| IIRC complete and open implementations of ASN.1 were/are
| rare and the matrix of covered features didn't quite
| overlap between languages.
| jakjak123 wrote:
| I think it has many good parts. There are waaaaay worse
| formats I have worked with over the years. Like COM, java
| RMI, many variants of SOAP, handcrafted json in countless
| variants...
| jamil7 wrote:
| What's an alternative you would recommend?
| anacrolix wrote:
| Anything prefix length encoded, with no schema.
| itishappy wrote:
| How do you handle schema instead?
| ilayn wrote:
| Yet another title to drive embedded device designers and control
| engineers up the wall.
|
| Why don't you make up your own bullshit words instead of randomly
| picking stuff from other places? That's not even multiplexing.
___________________________________________________________________
(page generated 2024-08-26 23:01 UTC)