[HN Gopher] Protobuffers Are Wrong (2018)
___________________________________________________________________
Protobuffers Are Wrong (2018)
Author : alihm
Score : 23 points
Date : 2023-03-23 21:29 UTC (1 hours ago)
(HTM) web link (reasonablypolymorphic.com)
(TXT) w3m dump (reasonablypolymorphic.com)
| jsnell wrote:
| Significant previous discussions:
|
| https://news.ycombinator.com/item?id=21871514 (211 comments)
|
| https://news.ycombinator.com/item?id=18188519 (298 comments)
|
| In particular, Kenton's rebuttal at
| https://news.ycombinator.com/item?id=18190005 is worth a read.
| shaftway wrote:
| Personally I avoid Map fields in protos. I don't find a huge
| amount of value for a Map<int, Foo> over a repeated Foo with an
| int key. That tends to be more flexible over time (since you can
| use different key fields) and it sidesteps all of the issues
| about composability of them.
|
| I think required fields are fine, provided that you understand
| that "required" means "required forever". If you're already using
| protos this isn't exactly a brand new concept. When you use a
| field number that field number is assigned forever, you can't
| reuse it once that field is deprecated. It requires a bit more
| thoughtful design, and obviously not all fields should be
| required, but it has value in some places.
| summerlight wrote:
| There are some legit criticisms here (remember, protobuf is a
| more than 20 years old format with extremely high expectation of
| backward compatibility so there are lots of unfixable issues),
| but in general this article reveals a fundamental
| misunderstanding of protobuf's goal. It is not a tool for elegant
| description of schema/data, but evolving distributed systems
| composed of thousands of services and storage. We're not talking
| about just code, but about petabytes of data potentially
| communicated through incomprehensible level of complex topology.
| morelisp wrote:
| protobufs are a data exchange format. The schema needs to map
| clearly to the wire format _because it is a notation for the wire
| format_ , not your object model du jour.
|
| If the protobuf schema code generator had to translate these
| suggestions into efficient wire representations and efficient
| language representations, it would be more complex than half the
| compilers of the languages it targets.
|
| I do mourn the days when a project could bang out a great
| purpose-built binary serialization format and actually use it.
| But half the people I hire today, and everyone in the team down
| the hallway that needs to use our API, can no longer do that. I'm
| lucky if they know how two's complement works
| mytailorisrich wrote:
| We use C and all our nodes are x86... so we just dump packed
| structs on the wire and the schema is a header file. I suspect
| this sort of simple approach works in most cases...
|
| And really that's how the network stack on Linux and Co. is
| implemented (modulo taking care of endianness)
| thadt wrote:
| Yep, that does work quite well in a fixed scope.
|
| Till one day someone wakes up and wants a UI in C#. Which is
| also not a big deal, but you have to either hand roll the
| serialization code or build a header file parser and generate
| it. And then someone needs to add a field - so we just tack
| it onto the end of the struct. This works fine as long as you
| have the length represented out of band somehow. If not then
| you get to make struct FooBar2 with your extra field. Then a
| year later someone has the great idea that it would be great
| to send text, and now you're making a variable length packet
| or just always sending around structs with a bunch of empty
| space and a length field (which is also not too bad). But
| wait, now the UI team totally needs that data in JavaScript
| for their new Web UI, so you're back to either generating or
| hand jamming dozens of structs. All the while tamping out
| bugs in various places where someone is running old code
| against new data or old data against new code - all of which
| have various expectations. Or that microcontroller that is
| blowing up because it just doesn't like that unaligned
| integer that is packed into the struct.
|
| Not that I'm bitter about that life - it just involved a lot
| more troubleshooting protocols over the years than I'd have
| liked. Anyway, those are problems that Protocol Buffers helps
| solve. But as long as you're using just C and not changing
| much - packed structs are quite lovely.
| mytailorisrich wrote:
| Sure, a small header that contains type and length is
| trivial and pretty much implied in my previous comment.
|
| I don't think people should think too much about "what if
| we change language", etc. because (1) that's unlikely to
| happen, (2) you have years ahead of you, (3) it may be
| simpler to convert structs (not the most complicated thing
| in the world amd supported in most languages in one form or
| another) into whatever else when/if you actually need it
| than to overengineer now 'just in case'.
| KerrAvon wrote:
| > protobufs are a data exchange format. The schema needs to map
| clearly to the wire format because it is a notation for the
| wire format, not your object model du jour.
|
| Yes, exactly this. I don't understand the vitriol here.
| Protobufs work fine for a wide variety of purposes and where
| the criticisms matter people use a different tool.
|
| > I do mourn the days when a project could bang out a great
| purpose-built binary serialization format and actually use it.
| But half the people I hire today, and everyone in the team down
| the hallway that needs to use our API, can no longer do that.
| I'm lucky if they know how two's complement works
|
| Would be great to work at a place where everyone knew how the
| machine worked, but the vast majority of developers entering
| the workplace since about 2000 have learned Java and web sh*t
| exclusively.
| ninkendo wrote:
| The problem is that protobufs _aren't_ just an interchange
| format, they're also a system for generating the code used to
| interact with said interchange. Said code has a habit of
| leaking into the types used by your code base. It's too easy
| to just pass protoc-generated objects around and use them all
| over your code base, hence the majority of the criticisms.
|
| Protobuf seems to encourage this... instead of a hard
| boundary where your serialization logic ends and your
| business logic begins, every project I've worked on that uses
| protobuf tends to blur the lines all over the place, even
| going as far as to make protoc-generated types _the_ core
| data model used by the whole code base.
| bsenftner wrote:
| I notice this too. What happened to CS degrees? All I meet are
| React-infused-piles-of-tools people that don't seem to be able
| to code anything without a mile long pile of tools.
| paxys wrote:
| There are ~30K CS graduates from US universities per year. A
| big chunk of them are international students who have to (or
| want to) go back home. The remaining ones are being fought
| over to fill an estimated 500K-1.5M open software engineering
| jobs. So chances are your company is likely not hiring the
| cream of the crop.
|
| Numbers are all rough, but seemed to be in the same ballpark
| in all the sources I checked (like
| https://thescalers.com/development-deep-dive-how-many-
| softwa...).
| theamk wrote:
| This was hard for read because of the glaring omissions of
| existing engineering practices. It feels author only cares about
| programming purity and writing programs which transform metadata.
|
| Making all fields in a message required makes messages into
| product types but loses compatibility with older or newer
| versions of protocol. Auto-creating objects on read dramatically
| increases chances that a field removal in future version of
| protocol will be handled properly. Auto-creating on write
| simplifies writing complex setters, etc...
|
| You could argue that those problems could be solved better (and I
| might agree, protobufs are one of my least favorite serialization
| protocols) but not even acknowledging reasons and pretending the
| designers didn't know better makes writer seem either ignorant or
| arguing in bad faith.
| paxys wrote:
| > They're clearly written by amateurs
|
| Protocol buffers were written by Jeff Dean and Sanjay Ghemawat.
| Whether you see issues with the implementation or not and whether
| they are fit for your use case or not are valid discussions, but
| if your argument starts with "omg these idiots couldn't design a
| simple product" then I'm already reading the rest of it with a
| huge grain of salt.
| grumpyprole wrote:
| I think the emotional response is probably due to the fact that
| Protocol buffers is often forced upon teams, due to the desire
| for a common standard and the marketing power that Google have.
| But the author raises many valid issues and you would be wrong
| to dismiss them completely. The first version of Protocol
| buffers, IIRC, didn't even support static offsets and
| extracting a field without deserialising the entire message. So
| it was both inelegant and inefficient. I was never impressed
| myself personally and have seen quite a few better closed-
| source wire formats inside various corporates.
| paxys wrote:
| Yes you can have many valid issues with Protocol buffers.
| "Written by amateurs" is not one of them, and that is the
| specific one I was dismissing.
|
| Reading through the rest of the post though, it is pretty
| clear that it is a standard case of trying to use what is a
| standard for data transfer over the network to describe your
| entire application's data schema. It isn't the right tool for
| the job, and doesn't have to be.
|
| If it is being forced on you then, well, that's a complaint
| for your management, not the technology itself.
| bragr wrote:
| >I think the emotional response is probably due to the fact
| that Protocol buffers is often forced upon teams
|
| True, but a lack of self awareness is never a good look.
| rektide wrote:
| The type system woes are quite legit. I wonder how many of these
| were improved in FlatBuffers. Which has a ton of other good
| things going for it, chiefly ideally less serialization demands.
|
| Once you start doing rpc, the demand goes so up. Cap'n'Proto's
| ability to return & process future values is such an exciting
| twist, that melds with where we are with local procedure calls:
| they return promise objects that we can pass all around while
| work goes on.
|
| Kenton popped up a couple months ago to mention maybe finding
| time for mutli-party support, where I can for ex request say the
| address of a building & send the future result to another 3rd
| party system. Now this is less just a way to serialize some
| stuff, & more a way to imagine connecting interesting systems.
___________________________________________________________________
(page generated 2023-03-23 23:03 UTC)