[HN Gopher] Show HN: Hexi - Modern header-only network binary se...
___________________________________________________________________
Show HN: Hexi - Modern header-only network binary serialisation for
C++
Over the last few years, I've needed an easy way to quickly
serialise and deserialise various network protocols safely and
efficiently. Most of the libraries that existed at the time were
either quite heavy, had less than stellar performance, or were an
abstraction level above what I was looking for. I decided to put
together my own class to do the job, starting with an easy, low-
overhead way to move bytes in and out of arbitrary buffers. Along
the way, it picked up useful bits and pieces, such as buffer
structures and allocators that made the byte shuffling faster,
often being able to do it with zero allocations and zero copies.
Safety features came along to make sure that malicious packet data
or mistakes in the code wouldn't result in segfaults or
vulnerabilities. It's become useful enough to me that I've
packaged it up in its own standalone library on the chance that it
might be useful to others. It has zero dependencies other than the
standard library and has been designed for quick integration into
any project within minutes, or seconds with a copy paste of the
amalgamated header. It can be used in production code but it's also
ideal for for those that want to quickly hack away at binary data
with minimal fuss.
Author : Chaosvex
Score : 64 points
Date : 2025-03-28 17:37 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| gregschlom wrote:
| Semi off-topic, but I just love the header image and the advice
| frog in the readme. Makes reading the documentation more fun and
| enjoyable.
| Chaosvex wrote:
| Thanks, that was the hope! :)
|
| I would have liked a different froggy reaction for each section
| but the project budget was zero. :^)
| LorenDB wrote:
| The problem is that without image captions or subtitles, you
| lose accessibility for anyone with a screen reader.
| Chaosvex wrote:
| I made sure to set the alt text for the images for that
| reason.
| nightowl_games wrote:
| Wow that api looks fantastic! Bravo!
|
| I'd like to read an even more thorough overview of how it works
| and all the gotchas before I'd consider using this 'in
| production' but the API looks very easy to use and very elegant.
|
| EDIT: just hit the section on portability, seems like you would
| always have to use that API, yeah? I feel like when you are
| writing network code you simply have to make it portable from the
| get-go. I guess I'm always thinking about having it run on client
| machines.
| Chaosvex wrote:
| Thanks. The documentation could definitely be fleshed out with
| some more examples.
|
| You'd likely want to always use that API (or layer something on
| top of it) unless you're in control of both ends and know they
| were built with the same toolchain & settings. One area where
| I've skipped over it is by writing a basic code gen tool
| (albeit unfinished as most personal projects) that generates
| the serialisation functions at compile-time from a very basic
| DSL that describes the network structures (of a game protocol I
| don't control). If it detects that the current toolchain is
| going to generate a binary-compatible struct layout and there
| aren't any variable length fields in there (no strings,
| basically), it'll generate a memcpy (via using get/put on the
| stream) rather than per-field (de)serialisation. If it can
| guarantee alignment of the buffer, which is a tougher
| requirement to meet, it'll give you a view directly into the
| network buffer so you effectively have zero-overhead
| deserialisation. Very much a work in progress but there's scope
| for making things quite efficient with just a few basic
| building blocks.
| karparov wrote:
| That code-gen would be fantastic. I have commercial
| applications for this, so I'll keep an eye on your space.
| klaussilveira wrote:
| It's been a while since I saw a new library with such a clean
| interface. Congrats!
| Chaosvex wrote:
| Thanks very much. :)
| codedokode wrote:
| It doesn't look like zero-copy though in this example:
| UserPacket packet; stream >> packet;
|
| That is at least one copy.
| Chaosvex wrote:
| Correct but it doesn't claim to be zero-copy overall. Apologies
| if it was misleading. In the README, zero-copy is mentioned in
| context of using view() and span() to obtain views into the
| buffer, which does allow for it. Very much a "there be dragons
| but if you're sure..." feature but it's there as an option.
|
| I've also built some tooling on top that makes use of those
| functions to do zero-copy deserialisation where viable, so it
| is possible in the right scenarios with a bit of work but it
| definitely isn't going to always fit.
| Jyaif wrote:
| Your lib requires manually creating both a serializing and
| deserializing function. If the functions are out of sync, bad
| things happen.
|
| Consider copying Cereal, which solves this problem by requiring
| you to create a single _templated_ function (
| https://uscilab.github.io/cereal/ )
| Chaosvex wrote:
| Thanks, that is definitely a downside to the shift operator
| overloading approach. I'll take that onboard and investigate
| whether a single operator to handle both would mesh with the
| current design.
| lionkor wrote:
| like boost serialize, which overloads the & operator
| jcelerier wrote:
| You can just use the boost.pfr technique to iterate fields
| though. Or if you want, starting from C++26 and e.g.
| clang-21: https://gcc.godbolt.org/z/G1TqP3a8P
| #include <print> #include <type_traits>
| namespace hexi { struct streamer { };
| template<typename T> void operator<<(streamer& s,
| const T& e) { if constexpr(std::is_integral_v<T>)
| { std::println("number: {}", e); }
| else if constexpr(std::is_aggregate_v<T>) {
| auto& [...mems] = e; (std::println("member:
| {}", mems), ...); } } }
| struct user_type { int x; std::string
| y; }; int main() {
| hexi::streamer s; s << 123; s <<
| user_type{.x = 456, .y = "foo"}; }
|
| or with boost.pfr as a polyfill until then, which allows to
| do this back to C++14
| codedokode wrote:
| By the way I looked through the code, and had to read about
| metaprogramming in C++. I wonder why is it so complicated? For
| example, why constraints like std::is_integral are represented by
| structs. Doesn't make much sense to me. A function wouldn't be
| better here?
| mandarax8 wrote:
| Because the only way to do metaprogramming in C++ is via the
| type system. Thismakes it so you need to implement 'functions'
| as types.
| etyp wrote:
| Practically, it's all through this `type_traits` header that
| (often) end up in unreadable messes. It's all possible because
| of the catchy acronym SFINAE. It doesn't make much sense to me
| either, so I avoid it :)
|
| https://en.cppreference.com/w/cpp/language/sfinae
| Wumpnot wrote:
| You don't really need to use sfinae anymore, concepts are
| cleaner and easier to follow, also this library appears to
| use concepts
| connicpu wrote:
| I know it's a convention since the inception of the language, but
| the operator overload abuse of the bitshift operator still makes
| me sad every time I see it :(
| Chaosvex wrote:
| On the plus side, it's optional. The same thing can be achieved
| with put()/get() equivalents.
| bluGill wrote:
| You are not alone. many on the standard committee are trying to
| get rid of it. std::print is the new way to do io instead of
| cout in part so you don't have to abuse shift for io. This is
| new in c++23 though so few people know about it.
|
| Bjarne appears to prefer cout though, so it isn't universal.
| soursoup wrote:
| Danish Bjarne may have his < right next to lshift. He needs
| to use shift+. to enter colon.
|
| On US layout colon is a single keypress but < is shift+.
|
| This may explain the discrepancy.
|
| --- from someone who read Bjarne at 16yo. All hail the Bjarne
| abcd_f wrote:
| In the same vein, but without needing to create separate de- and
| serialize functions:
|
| https://github.com/eliasdaler/MetaStuff
|
| Another take on the same idea with even simpler interface:
|
| https://github.com/apankrat/cpp-serializer
| Chaosvex wrote:
| I'll likely add additional functionality for specifying both
| operations with a single function since it's been mentioned a
| few times. Thanks for the repos.
| huhtenberg wrote:
| What are the exact constraints on the struct contents, i.e. what
| is it that your library _can 't_ serialize?
|
| I tried adding std::string to the UserPacket (from the README)
| struct UserPacket { // uint64_t user_id; //
| uint64_t timestamp; // std::array<uint8_t, 16> ipv6;
| std::string test; };
|
| and the compilation fails - https://onlinegdb.com/B_RJd5Uws
___________________________________________________________________
(page generated 2025-03-28 23:00 UTC)