[HN Gopher] Resizable structs in Zig
___________________________________________________________________
Resizable structs in Zig
Author : rvrb
Score : 147 points
Date : 2025-07-26 21:43 UTC (1 days ago)
(HTM) web link (tristanpemble.com)
(TXT) w3m dump (tristanpemble.com)
| rvrb wrote:
| I am the author of this post, let me know if you have any
| questions or feedback :)
| azemetre wrote:
| I sincerely mean this when I write this: please make more Girls
| on the Beach albums. I really love Splif Tape, reminds me of
| Julie Ruin during that timeframe as well (2015ish).
| rvrb wrote:
| oh, man! you made my day with this comment. those days are
| long behind me. we were young, dumb, and inebriated. if
| you're looking for the bands we were trying to sound like,
| there was a resurgence of this surf/girl group sound in the
| early 2010s. look for bands like Shannon and the Clams, Hunx
| and His Punx, Nobunny, Harlem, Ty Segall, Thee Oh Sees..
| basically anyone on the now defunct Burger Records
| atmikemikeb wrote:
| I thought about dynamically sized types (DSTs) in zig recently.
| Was considering writing about it. I came to a different
| conclusion. Why not use zig's opaque?
|
| It's pretty clean at this imo: Less metaprogramming but I think
| nicer to use in some cases. const Connection =
| opaque { pub const Header = struct { host_len:
| usize, // add more static fields or dynamic field
| lengths here //buff_len: usize, };
| pub fn init(a: std.mem.Allocator, args: struct { host: []const u8
| }) *@This() { var this = a.allocWithOptions(u8,
| @sizeOf(Header) + host.len, @alignOf(Header), null);
| @memcpy(this[@sizeOf(Header)..], host); return
| this.ptr; } pub fn host(self: *const
| @This()) []const u8 { const bytes: *u8 =
| @ptrCast(self); const header: *Header = @ptrCast(self);
| const data = bytes[@sizeOf(Header)..]; const host =
| data[0..header.host_len]; return host; }
| };
|
| going off memory so I expect it to not actually compile, but I've
| definitely done something like this before.
| rvrb wrote:
| I would describe this approach as 'intrusive' - you're storing
| the lengths of the arrays behind the pointer, enforcing a
| certain layout of the memory being allocated.
|
| Because the solution outlined in the article stores the lengths
| alongside the pointer, instead of behind it, there is room for
| it to work across an ABI (though it currently does not). It's
| more like a slice in this way.
|
| You could in theory implement your opaque approach using this
| as a utility to avoid the headache of alignment calculations.
| For this reason, I think that makes the approach outlined in
| the article more suitable as a candidate for inclusion in the
| standard library.
| atmikemikeb wrote:
| Yeah I think mine is more about being able to provide a
| `host()` helper function instead of a `.get(.host)` meta
| function. It is somewhat boilerplate-y. I think it's really a
| matter of taste haha. Likely yours would be useful regardless
| if this is done a lot, since it abstracts some of it, if one
| wants that.
| rvrb wrote:
| I've entertained further expanding this API to expose a
| comptime generated struct of pointers. From the Connection
| use-case detailed in the article, it would look something
| like this: pub fn getPtrs(self: Self)
| struct { client: *Client, host: []u8,
| read_buffer: []u8, write_buffer: []u8, } {
| return .{ client: self.get(.client),
| host: self.get(.host), read_buffer:
| self.get(.read_buffer), write_buffer:
| self.get(.write_buffer), } }
|
| I haven't done this because I'm not yet convinced it's
| worth the added complexity
| ethan_smith wrote:
| The opaque keyword in Zig doesn't support method definitions -
| it creates an incomplete type that must be implemented
| elsewhere, not a type that can have inline methods and fields
| like your example attempts.
| pbaam wrote:
| https://ziglang.org/documentation/master/#opaque
|
| > opaque {} declares a new type with an unknown (but non-
| zero) size and alignment. It can contain declarations the
| same as structs, unions, and enums
|
| So it can contain methods, just like structs. The only thing
| it cannot contain is fields (which the example above doesn't
| contain). An opaque type can only be used as a pointer, it's
| like a `typedef void* ...` in C but with the possibility to
| add declarations within a namespace
|
| Edit: the documentation doesn't contain any example of an
| opaque type which happens to have declarations inside. But
| here is one: https://codeberg.org/andrewrk/daw/src/commit/38d
| 3f0513bf9bfc...
| mananaysiempre wrote:
| > Zig does not, and will not, have VLAs in the language spec.
| Instead, you can allocate a slice on the heap. If you want to
| have the data on the stack, use an array as a bounded backing
| store, and work with a slice into it[.]
|
| Too bad, aligned byte-typed VLAs (and a license to retype them as
| a struct) are what you need to get stack allocation across ABI
| boundaries the way Swift does it. (A long long time ago, SOM,
| IBM's answer to Microsoft's COM, did this in C with alloca
| instead of VLAs, but that's the same thing.) I guess I'll have to
| use something else instead.
| rvrb wrote:
| I think there may be room to expand this implementation to
| support such a use case. Right now it enforces an `.auto`
| layout of the struct provided in order to ensure alignment, but
| its easy to imagine supporting an `extern struct` with a
| defined layout.
|
| Conceivably, an implementation of this `ResizableStruct` that
| uses an array buffer as backing rather than a heap allocation,
| and supports the defined layout of an extern struct, could be
| used to work across the ABI
| throwawaymaths wrote:
| you can certainly allocate on the stack (like alloca). you just
| have to overallocate a compile-time known size and have some
| sort of fallback mechanism or fail if the size requested
| exceeds the amount created.
|
| moreover since the stack allocator is just an allocator, you
| can use it with any std (or user) datastructure that takes an
| allocator.
| AndyKelley wrote:
| Note that not having runtime-known stack allocations is a key
| piece of the puzzle in Zig's upcoming async I/O strategy
| because it allows the compiler to calculate upper bound stack
| usage for a given function call.
|
| At a fundamental level, runtime-known stack allocation harms
| code reusability.
|
| Edit: commenters identified 2 more puzzle pieces below, but
| there's still one that didn't get asked about yet :P
| Conscat wrote:
| Does anything stop a user from doing this with inline
| assembly?
| AndyKelley wrote:
| Wisdom
| AshamedCaptain wrote:
| How does this work given... recursion?
|
| Even on languages without VLAs one can implement a simulacra
| of them with recursion.
| AndyKelley wrote:
| All Zig code is in one compilation unit, so the compiler
| has access to the entire function call graph. Cycles in the
| graph (recursion) cause an error. To break cycles in the
| graph, one must use a language builtin to call a function
| using a different stack (probably obtained via heap
| allocation).
| atmikemikeb wrote:
| what about extern functions?
| AndyKelley wrote:
| Zig's linker will calculate this information
| automatically in most cases when statically linking (via
| analysis of machine code disassembly). Otherwise, there
| is a default upper bound stack value, overridable via
| user annotation.
| dev-ns8 wrote:
| Does this mean it's impossible in Zig to do strictly
| Stack related recursion and just by the mere inclusion of
| a recursive function your implicitly getting heap
| allocations alongside?
| AndyKelley wrote:
| You can put a big buffer on the stack, and use this
| buffer to break your cycles. At some point you'll run out
| of this buffer and be forced to handle failure, rather
| than triggering a stack overflow segfault.
|
| So it will be the same thing but with more (error
| handling) steps.
|
| This annoyance can be avoided by avoiding recursion.
| Where recursion is useful, it can be done, you just have
| to handle failure properly, and then you'll have safety
| against stack overflow.
| eru wrote:
| Does Zig offer (guaranteed) tail call optimisation?
|
| > Where recursion is useful, [...]
|
| Recursion is so useful, most imperative languages even
| have special syntax constructs very specific special
| cases of recursion they call 'loops'.
| messe wrote:
| > Does Zig offer (guaranteed) tail call optimisation?
|
| Yes[1]. You can use the @call builtin with the
| .always_tail modifier.
| @call(.always_tail, foo, { arg1, arg2, ... });
|
| [1]: https://ziglang.org/documentation/master/#call
| CJefferson wrote:
| Wait, so how do I write mutually recursive functions, say
| for a parser? Do I have to manually do the recursion
| myself, and stick everything in one big uber-function?
| ants_everywhere wrote:
| > All Zig code is in one compilation unit
|
| How do incremental compilation and distributed
| compilation work?
| pyrolistical wrote:
| Subtrees should be cacheable and parallelizable?
| wavemode wrote:
| Single compilation unit does not imply that the results
| of the compilation of the different parts of that unit
| cannot be cached.
| ants_everywhere wrote:
| I see. So in some sense the actual unit of compilation is
| smaller and units can be combined or "linked" even if
| compiled at different times on different machines?
| omnicognate wrote:
| https://github.com/ziglang/zig/issues/1006
| omnicognate wrote:
| > there's still one that didn't get asked about yet :P
|
| C libraries?
| NobodyNada wrote:
| Or function pointers (especially given that Zig's been
| moving towards encouraging vtables over static dispatch)?
| AndyKelley wrote:
| Bingo. That completes the picture:
| https://github.com/ziglang/zig/issues/23367
| mananaysiempre wrote:
| > Note that not having runtime-known stack allocations is a
| key piece of the puzzle in Zig's upcoming async I/O strategy
| because it allows the compiler to calculate upper bound stack
| usage for a given function call.
|
| That's a genuinely interesting point. I don't think known
| sizes for locals are a hard requirement here, though
| threading this needle in a lower-level fashion than Swift
| would need some subtle language design.
|
| Fundamentally, what you want to do is construct an
| (inevitably) runtime-sized type (the coroutine) out of (by
| problem statement) runtime-sized pieces (the activation
| frames, itself composed out of individual, possibly runtime-
| sized locals). It's true that you can't then allow the
| activations to perform arbitrary allocas. You _can_ ,
| however, allow them to do allocas whose sizes (and
| alignments) are known at the time the coroutine is
| constructed, with some bookkeeping burden morally equivalent
| to maintaining a frame pointer, which seems fair. (In Swift
| terms, you can construct a generic type if you know what type
| arguments are passed to it.) And that's enough to have a
| local of type of unknown size pulled in from a dynamic
| library, for example.
|
| Again, I'm not sure how a language could express this
| constraint on allocas without being Swift (and hiding the
| whole thing from the user completely) or C (and forcing the
| user to maintain the frames by hand), so thank you for
| drawing my attention to this question. But I'm not ready to
| give up on it just yet.
|
| > At a fundamental level, runtime-known stack allocation
| harms code reusability.
|
| This is an assertion, not an argument, so it doesn't really
| have any points I could respond to. I guess my view is this:
| there are programs that can be written with alloca and can't
| be written without (unless you introduce a fully general
| allocator, which brings fragmentation problems, or a parallel
| stack, which is silly but was in fact used to implement
| alloca historically). One other example I can give in
| addition to locals of dynamically-linked types is a bytecode
| interpreter that allocates virtual frames on the host stack.
| So I guess that's the other side of being opinionated--those
| whose opinions don't match are turned away.
|
| Frankly, I don't even know why I'm defending alloca this
| hard. I'm not actually happy with the status quo of just
| yoloing a hopefully maybe sufficiently large stack. I guess
| the sticking point is that you seem to think alloca is
| obviously the wrong thing, when it's not even close to
| obvious to me what the right thing is.
| bobthebuilders wrote:
| Alloca is a fundmentally insecure way of doing allocations.
| Languages that promote alloca will find themselves stuck in
| a morass of security messes and buffer overflows. If Zig
| were to adopt alloca, it would make the catastrophic
| mistake that plagued C for over several decades and
| introduce permanently unfixable security issues for another
| generation of programming languages.
| rurban wrote:
| Didn't stop rust from using it internally.
| surajrmal wrote:
| How does it do that?
| kibwen wrote:
| I don't think Rust uses alloca internally for anything.
| You may be thinking of Swift, which I think uses alloca
| for ABI shenanigans.
| steveklabnik wrote:
| I don't know why you're downvoted, alloca is a mistake.
| johnisgood wrote:
| Any thoughts on the use of strdupa()? I do not use it,
| but I wonder if that is dangerous too, considering it
| uses alloca().
| mananaysiempre wrote:
| I've been defending alloca() here, but no, strdupa() (not
| to be confused with shlwapi!StrDupA on Windows) is a bad
| idea. In cases that I think are acceptable, the size of
| the allocation is reasonably small and does not come from
| outside the program. Here you're duplicating a string
| that you probably got somewhere else and don't really
| control. That means you don't really know if and when
| you're going to overflow the stack, which is not a good
| position to be in.
|
| (Once upon a time, MSLU, a Microsoft-provided Unicode
| compatibility layer for Windows 9x, used stack-allocated
| buffers to convert strings from WTF-16 to the current
| 8-bit encoding. That was also a bad idea.)
| Yoric wrote:
| Out of curiosity, why is knowing the size of locals
| required, exactly? Because it avoids dynamic allocations
| for each fiber?
| do_not_redeem wrote:
| A comptime_int-bounded alloca would achieve those goals, plus
| would be more space-efficient on average than the current
| strategy of always pessimistically allocating for the worst
| case scenario. @alloca(T: type, count: usize,
| upper_bound_count: comptime_int)
|
| with the added bonus that if `count` is small, you can avoid
| splitting the stack around a big chunk of unused bytes. Don't
| underestimate the important of memory locality on modern
| CPUs.
| AndyKelley wrote:
| 2017: https://github.com/ziglang/zig/issues/225
|
| when I had been only thinking about zig for 2 years, I
| thought the same.
| do_not_redeem wrote:
| I'd be curious if you expanded your reasoning, your
| comments in that thread never explained anything for me.
|
| > It's too tempting to use incorrectly.
|
| A compile-time-determined upper bound would solve this.
|
| > The stack is allocated based on a compile-time-
| determined upper bound.
|
| A compile-time-determined upper bound would solve this
| too.
|
| Shouldn't a performance-oriented language give the
| programmer tools to improve memory locality? And what's
| wrong with spexguy's idea?
| throwawaymaths wrote:
| instead of being namby pamby with the stack, it's simply
| better to take all of the desired maximimum. in zig,
| there's even provided a way to wrap it in an allocator so
| you can pretend like it's on the heap!
| do_not_redeem wrote:
| This conversation would benefit from using more rigorous
| technical terminology than "namby pampy". There is
| nothing namby pamby about allocating the precise amount
| of space that you need, and keeping your app's memory
| footprint optimized. That's called engineering.
| throwawaymaths wrote:
| real question: what are you going to do with the rest of
| the stack? are you in a situation where the stack and the
| heap might collide because you're that tight on
| resources? and let's say you take a function call that is
| about to overflow the stack. what should happen? error?
| panic? return null? silent fail?
|
| there are no good choices in the case where you really
| need that thing you claim to need. recognizing that fact
| and picking different strategy is _good engineering_.
| do_not_redeem wrote:
| This whole post is a strawman. I never said my reason was
| being tight on resources. Please reread the thread. Also
| don't forget that on modern architectures, the stack and
| heap can't "collide", because of guard pages.
|
| > what are you going to do with the rest of the stack?
|
| I'll leave it for the rest of the system. My app will use
| less memory, and since memory locality is improved, there
| will be fewer cache misses, meaning it runs faster too.
|
| > let's say you take a function call that is about to
| overflow the stack
|
| Stack overflows are impossible thanks to the comptime
| upper_bound parameter. That's the entire premise of this
| thread.
| uecker wrote:
| The first stack / heap collisions were not using VLA but
| fixed size arrays on the stack. Nowadays compilers do
| stack probing, which solves this problem also for VLAs.
| Yes, you get a segfault for stack overflow, but this has
| not much to do with VLAs or not, but with putting too
| much stuff on the stack. The thing is, VLAs allow you to
| reduce stack usage, by putting the right amount of stack
| on the stack and not a worst case. The only downside is
| that they make it harder to control stack usage, but not
| a lot harder. So no, I do not think avoiding VLAs is good
| engineering.
| CJefferson wrote:
| Yes, let's be "namby Pamby" with the cache lines storing
| the hot part of the stack, that sounds like an awesome
| idea!
|
| I thought Zig was all about maximum performance.
| Sometimes I just want a little bit of stack memory, which
| will often already be in L1 cache.
| achierius wrote:
| Zig does allow this, that's what GP is saying. You don't
| actually need to relocate your stack, you can just
| declare a portion of your stack (i.e. what would
| otherwise be the next N frames) to be the stack you'll
| use for recursion, and thereafter use _that_ to recurse
| in to.
| travisgriggs wrote:
| > Note that not having runtime-known stack allocations is a
| key piece of the puzzle in Zig's upcoming async I/O strategy
| because it allows the compiler to calculate upper bound stack
| usage for a given function call.
|
| Sigh. So I have to choose between something I think _might_
| be useful, for something that too many languages have already
| soiled themselves with. Hopes that Zig has a better solution,
| but not optimistic.
|
| Our stack compels me to work in Swift, Kotlin, Elixir, and
| Python. I use the async feature of Swift and Kotlin when some
| library forces me to. I actually preferred working with GCD
| before Swift had to join the async crowd. Elixir of course
| just has this problem solved already.
|
| I frequently ask others who work in these languages how often
| they themselves reach for the async abilities of their
| languages, and the best I ever get from the more adventurous
| type is "I did a play thing to experiment with what I could
| do with it".
| dnautics wrote:
| RE: Elixir I have a feeling that the zig's i/o strategy
| will enable me to bring back the zig-async-dependent
| yielding nifs in zigler. I'm _really_ hopeful io interface
| will have a yield() function, that would be even better!
|
| https://www.youtube.com/watch?v=lDfjdGva3NE&t=1819s
| arthurcolle wrote:
| Love the excitement
| ManDeJan wrote:
| How does this work with interrupts, say in an embedded
| context, that execute on the current stack, and that _may_ in
| some cases, be interrupted themselves. Do you add the maximum
| stack depth of all interrupt routines that could go off at
| the same time?
| h4ck_th3_pl4n3t wrote:
| I am wondering if this is more of an unclearly defined memory
| ownership problem rather than a problem of what types you have
| to use to interact with C ABIs or FFI calls.
|
| I mean you could also just abstract the allocation away and
| handle it after the function pointer to your bridge, right?
| konstantinua00 wrote:
| one thing I never understood about VLAs - discussion about them
| always hits a "can't put it on stack safely" and gets halted,
| forever
|
| why not to make it heap-only type? it seems such a useful
| addition to type system, why ignore it due to one usecase?
| Out_of_Characte wrote:
| Because arrays simply do not deal with fragmentation. Yes, you
| could probaly get decent performance on a modern system that
| has memory overcommit strategy where you could allocate sparse
| adress ranges where you would probaly never run out of pointers
| unless you actually write to your variable array.
|
| But its just kind of mediocre and you're better off actually
| dealing with the stack if you can actually deal with certain
| fixed sizes.
| konstantinua00 wrote:
| ...what are you talking about?
|
| array-like storage with dynamic size has existed since
| forever - it's vector. over or undercommitting is a solved
| problem
|
| VLA is the way to bring that into type system, so that it can
| be it's own variable or struct member, with compiler auto-
| magic-ing size reading to access members after it
| Out_of_Characte wrote:
| > auto-magic-ing size reading to access members after it
|
| From the article
|
| >we now have everything we need to calculate the size,
| offset and alignment of every field, regardless of their
| positioning in the struct. >init to allocate the memory
| >get to get a pointer to a field >resize to resize the
| arrays >deinit to free the memory
|
| You're now suggesting to do exactly what the article is
| about without being aware of it.
| ori_b wrote:
| Those effectively exist. They're called slices.
| uecker wrote:
| You can also put them safely on the stack. The VLA is
| discussion is just irrational.
| quotemstr wrote:
| Zig articles tend to get a little too excited about rediscovering
| longstanding techniques.
|
| The author has described a metaprogramming utility for allocating
| a contiguous hunk of memory, carving this hunk into fields (in
| the article's example, a fixed-sized Client header, then some
| number of bytes for host, then some number of bytes for
| read_buffer, and then some for write_buffer). I'll acknowledge
| the syntax is convenient, but
|
| 1. we've done this since time immemorial in C. See
| https://learn.microsoft.com/en-us/windows/win32/api/evntcons...
|
| 2. you can implement this pattern ergonomically in C++, and even
| moreso once the C++26 reflection stuff comes online
|
| 3. the zig implementation is inefficient. It desugars to
| const Connection = struct { ptr: [*]u8, lens:
| struct { host: usize, read_buffer: usize,
| write_buffer: usize, } }
|
| That first pointer is needless indirection and probably a cache
| miss. You should (unless you have specific performance data
| showing otherwise) store the sizes in the object header, not in
| an obese pointer to it. (It's bigger than even a fat pointer.)
| rvrb wrote:
| You have raised a good discussion point or two, but I am not
| inclined to engage with them due to the tone you've created
| with the rest of your comment.
|
| Would it be productive to jump into a thread on a Ruby article
| and puff your feathers about how you've always been able to do
| this in Perl, and also in Python 4 you can do XYZ? I don't
| think so.
|
| For whatever reason, inevitably in threads on systems
| languages, someone comes in and postures themselves defensively
| like this. You might want to reflect on that.
| quotemstr wrote:
| I don't need Zig and neither does anyone else. Go ahead, keep
| reinventing the wheel and writing about discovering
| roundness. Why would I want to engage with a community
| combining ignorance and hair trigger hypersensitivity and
| tone policing?
|
| I'm not sure you people understand how grating this kind of
| talk is to normal people nowadays. It's like you're emerging
| from cryogenic sleep and think it's still 2018. Nobody wants
| to engage with a community that shuts down technical critique
| with to mild technical critique with finger wagging instead
| of engaging with inconvenient facts.
| rvrb wrote:
| I assure you, I am not the one who appears to be reacting
| in a hair trigger hypersensitive manner in this thread :)
| AndyKelley wrote:
| I think you're misinterpreting OP's excitement here. The
| technique isn't novel but the ergonomics are.
|
| > you can implement this pattern ergonomically in C++,
|
| lmao
| rvrb wrote:
| > I think you're misinterpreting OP's excitement here. The
| technique isn't novel but the ergonomics are.
|
| This, of course, but I also consciously made the decision to
| write for an audience less familiar with the language and
| concepts being discussed. That excitement can translate to
| engaging a less disgruntled and more curious reader than the
| OP.
| qalmakka wrote:
| The author is literally proposing to implement arrays of variant
| types
| edem wrote:
| Should I try Zig? I wanted to learn something that's more low
| level than the JVM/Node and i have been contemplating rust and
| go. zig never occurred to me until now.
| hiccuphippo wrote:
| I think it's simpler than rust, the basics are easier to learn
| and will get you a long way before you need the more advanced
| stuff. Go might be easier because it's garbage collected. If
| you want something closer to C, but not C itself, Zig is a good
| choice.
___________________________________________________________________
(page generated 2025-07-27 23:02 UTC)