[HN Gopher] Rust's worst feature
___________________________________________________________________
Rust's worst feature
Author : aw1621107
Score : 116 points
Date : 2025-01-27 01:34 UTC (3 days ago)
(HTM) web link (mina86.com)
(TXT) w3m dump (mina86.com)
| 0x1ceb00da wrote:
| > Even an obvious optimisation of moving the buffer declaration
| outside of the loop isn't available to the compiler.
|
| Why? Can't the programmer just do this himself?
| Arch-TK wrote:
| The compiler cannot assume that the read call won't read from
| the mutable reference (well, it might be able to given a
| sufficiently sophisticated optimizer and/or aggressive
| inlining).
|
| The programmer, on the other hand, can do this, but the point
| is to make this implicitly possible by making it more explicit
| that read does not read from the buffer (and therefore allowing
| it to accept uninitialized memory).
| dietr1ch wrote:
| And I don't think it can ensure that all the bits were
| written. I've been bit by people trying to reuse
| buffers/objects like this that were not fully rewritten in
| one of the possible re-uses. It's a bit puzzling how a change
| that just adds a new continue leads to memory corruption.
| hansvm wrote:
| Zig has a couple features to help with that. I assume Rust
| should too (probably not directly applicable to
| BorrowedBuf, but for the case of a reusable pool of
| objects)?
|
| It comes down to a piece of syntactic sugar, plus "result
| location semantics" guaranteeing that you won't have a
| copy. E.g.: my_ptr.* = .{ .x = 42,
| .y = 53 };
|
| No matter how you choose to construct the intermediate
| fields (x and y in this example) with continues or other
| control flow, the very last step should be something that
| sets every field at once. If you miss one, the compiler
| will yell at you. If it compiles, the assembly is as if you
| filled in each field by hand.
| dietr1ch wrote:
| It's cool to have syntax for that, but I feel that the
| optimisation there works at a too high level.
|
| I'd like the backend to notice that a variable is being
| dropped and "re-created" on every iteration, and then
| figuring out how to initialise over a "dropped" zombie
| value. It'd be nice to have something like this, because
| I'm pretty sure people don't do this kind of optimisation
| all the time (it's annoying to leak the variable that
| should survive loops outside of the loop becase it's not
| mentioned elsewhere).
| Waterluvian wrote:
| Arguably the buffer belongs in the loop scope because it is
| only relevant there. It's probably also safer from wrong use.
|
| This feels like exactly what you want the compiler to think
| about: a case where optimization comes at the cost of
| organization.
| lmm wrote:
| > Why?
|
| Because as far as the compiler is concerned it appears to
| change the behaviour, unless the compiler gets very fancy.
|
| > Can't the programmer just do this himself?
|
| Yes, but it's not really desirable for them to have to (and
| would arguably make the code less maintainable if they did).
| Doing the right thing should be easy.
| auggierose wrote:
| Not if they are using Rust ... which is why I am not.
| vlovich123 wrote:
| What language would automatically be able to hoist the array
| outside the loop in that kind of code?
| jhasse wrote:
| C, because reading uninitialized memory is undefined
| behavior so the compile can assume it never happens.
| jeroenhd wrote:
| But wouldn't that change behavior? An empty, zero
| initialised array will contain data and a bunch of zeroes
| after a read, but if the next read only partially fills
| the buffer you end up with a buffer containing data from
| two reads.
|
| In this specific example there's no issue because the
| result of read() is being used to only write as much data
| as was read, but to me this seems like a pretty
| complicated and unlikely assumption to write
| optimizations for.
| vlovich123 wrote:
| Care to present some proof? Here's an counter proof that
| the compiler isn't able to reason about the memory in
| that way https://godbolt.org/z/x7j8xoMxY
|
| There are cases where C can do loop hoisting, but the
| cases are a subset of what Rust does and this isn't one
| of those.
| uecker wrote:
| You example doesn't show this. Also the allocation _is_
| hoisted out of the loop. The initialization is not and
| this would be invalid in general. It could eliminated in
| this case, but this would be dead store elimination.
| uecker wrote:
| Reading uninitialized memory is not undefined behavior.
| kbolino wrote:
| An explanation of why:
| https://stackoverflow.com/a/11965368/814422
|
| There are some important caveats, though, around trap or
| non-value representations. Basically, the value held by
| the storage for a variable may not correspond to a valid
| value of the variable's type.
|
| For example, a bool variable usually takes a full byte
| but only has 2 valid representations in many ABIs (0 for
| false, 1 for true). That leaves 254 trap representations
| with 8-bit bytes, and trying to read any of these is
| undefined behavior.
|
| Furthermore, a variable may be stored in a register
| (unless you take its address with &), and registers can
| store values wider than the variable type--e.g., even
| though int has no trap representations in memory of the
| same size, nowadays it's usually smaller than a register
| --or be in a state that makes them unreadable. Trying to
| read such a value is also undefined behavior.
|
| So, reading _memory_ in general is defined behavior (just
| with an indeterminate value) but it has to actually _be_
| memory and you have to be reading it into a type that can
| accept arbitrary bit patterns.
| dgacmu wrote:
| This is incorrect. It's trivial and compiles just fine. The
| argument here is that maybe for reasons the programmer
| doesn't want to - such as not wanting the buffer to outlive
| its use inside the loop, and they don't want to have to
| double-nest: { let mut buf =
| [0; 4096]; loop { ...
| } }
|
| That accomplishes exactly the same goal but there's an
| argument -- not well made in the blog post -- that the
| compiler should be able to do some form of this hoisting
| automatically. In C, it would be automatic, because C doesn't
| make a zero-initialized promise for stack-allocated
| variables. In Rust it's not because the array is specified as
| zero-initialized. Of course, C's behavior comes with certain
| drawbacks of its own. ;)
|
| Rust's behavior isn't unreasonable. It's just a potential
| missed optimization, but automating it is challenging.
| jeroenhd wrote:
| Adding an extra scope here is slightly annoying, but it's
| not always possible. I think the example in the blog post
| was poorly chosen, because the complexity of BorrowedBuf
| together with MaybeUninit doesn't make much sense when your
| fix makes for much more readable code.
|
| Out of all problems I have encountered with Rust, this is a
| particularly minor one.
| auggierose wrote:
| I see. It is a very bad example indeed. Terrible, terrible
| example.
|
| Switching off Trump mode for a moment, I don't see why you
| would want to declare the buffer inside the loop, given
| that keeping it alive for the entire time of the loop is
| actually the semantics you want.
| kmeisthax wrote:
| If people wrote the most optimal code the first time, we
| wouldn't need optimizing compilers, and all the undefined
| behavior that optimization passes necessarily bring
| along. The whole point of the example is to be poorly
| written in a way that the compiler "obviously" should be
| able to fix, but can't.
|
| A less obvious example would be...
|
| - A struct, A, which has an init_from_file method that
| deserializes data from a Read: R
|
| - Another struct, B, which has its own init_from_file,
| and a variable number of A as one of its fields.
| B::init_from_file needs to deserialize by calling
| A::init_from_file in a tight loop.
|
| This example is the same as the first, except now we've
| disguised the inefficiency with separation of concerns. A
| compiler can inline A::init_from_file into
| B::init_from_file to yield the same code as in the
| example.
| auggierose wrote:
| So you are saying that the buffer would be allocated in
| the program in A::init_from_file? And the compiler would
| be able to optimise that away by allocating the buffer
| outside the loop?
|
| If the compiler actually does that, that would be a good
| example. As long as I don't have to be careful to write
| my code in a way that some obscure compiler optimisation
| understands.
| dgacmu wrote:
| Because you want the buffer to go out of scope after the
| last iteration of the loop. Motivating that requires
| bringing in more rust - It could be as simple as wanting
| to reuse the variable name later, but more likely it
| would be because you were using something that had a
| reference that you wanted to go away so you could borrow
| it again without the borrow checker yelling at you.
| auggierose wrote:
| Ok, then the nested scope is indeed exactly what you
| want. I don't see how obfuscating this purpose and trying
| to rely on obscure compiler optimisations and intricate
| semantics would be a good idea.
| dgacmu wrote:
| Because, as someone else noted, it might be hidden for
| you because you're using something that's inside another
| function, macro, struct, etc.
| auggierose wrote:
| Yes, that is a valid reason. If I get that optimisation
| for free, why not?
|
| I still would not rely on that optimisation, though. If I
| think this could be an actual bottle neck, I would make
| the shared buffer explicit in my code.
| vlovich123 wrote:
| Because the compiler doesn't know what read/write are doing to
| the buffer. And since it's declared as [0; 4096], the compiler
| wouldn't be able to do anything other than 0'ing the entire
| 4kib region on every read instead of what's dirtied if it
| attempted to automatically hoist. BorrowedBuf is an attempt to
| let you declare [MaybeUninit::uninit(); 4096] which the
| compiler _could_ hoist although there it doesn't matter either
| since the allocation of the uninit array is just an adjustment
| of the stack pointer.
| mmastrac wrote:
| This API has basically been adopted from Tokio. Like most of Rust
| buffer types, it's "not bad" to use as a caller and "awkward" to
| use as a consumer.
|
| The pain of paying for buffer init is real, however. The last two
| projects have both seen perf hits from it.
| tmtvl wrote:
| I'm not a Rust person (give me Lisp any day), but this stuck out
| to me:
|
| > _A motivated programmer can try adding necessary support to
| actively maintained packages [...] but what if one is stuck at an
| older version of the crate or deals with apparently abandoned
| crates_
|
| One could maybe do some programming? I mean, hell, if most of the
| work has already been done for you, then what's holding you back?
| Besides, why would you want to use outdated, bug-ridden libraries
| filled with vulnerabilities instead of something well-maintained?
| layer8 wrote:
| The notion that a library must be "bug-ridden" and "filled with
| vulnerabilities" just because it hasn't changed in a while is a
| strange one. If it were true, it would also be true for
| libraries that do change. It's not like libraries accumulate
| new bugs and vulnerabilities by not changing. Libraries that
| constantly gain new features, on the other hand, are prone to
| also gain new bugs and vulnerabilities.
| bbarnett wrote:
| Agree, I think we need to speak of abandoned, versus
| maintained.
|
| No activity doesn't mean abandoned, it can just mean super
| stable and few bugs to fix.
|
| Abandoned is a big worry though.
|
| I wonder if we need a canary file, but in the repo. An "I'm
| still here" file.
| layer8 wrote:
| If a library is stable with no major bugs, then I don't see
| much of a difference between abandoned and maintained.
|
| The one thing I would look at instead is open issues.
| bbarnett wrote:
| Abandoned means no one around to deal with security
| issues, or impacting bugs.
|
| Maintained means they are. There may be no new features
| though.
|
| (these are my definitions, but I think there should be
| similar concepts in all dev's heads)
| layer8 wrote:
| It changes nothing about the current version of the
| library though. It only impacts how quickly or easily one
| would be able to get a fix in case some issue does come
| up. But I'm questioning the expectation that such issues
| will come up, and the associated expectation that one
| will have to regularly update the library because new
| issues won't ever stop coming up.
| 3pt14159 wrote:
| It's about perception. When I look at a old page on
| GitHub and there haven't been any code changes in two
| years but it does look like the Issues page is addressed
| quickly then I can trust the old code. On the other hand
| if the Issues page isn't addressed I can assume that the
| code will cause me problems in the future and I'll look
| for a different library.
| bbarnett wrote:
| There are endless examples of libraries being updated,
| due to bad code in the library. It's not a rare event,
| it's very, very common.
|
| If a library is abandoned, and no one is around updating
| the code for vulnerabilities, it's trouble. That's
| because one day you can use it, and the next you cannot.
|
| (Yes you can personally patch it, but that's an issue
| that's come up. And how will you know? Look at all the
| bugtrackers daily to see people yelling "HEY!"?)
| travisjungroth wrote:
| I still question that. It's generally as good as the day it
| came out. Discovered vulnerabilities are going to be in the
| language, probably not in the library itself. As long as
| it's not dependent on a language that's so old it's not
| getting security fixes, it should be fine.
|
| Actually, a big exception is libraries that have external
| dependencies that change. A client library for an API, for
| example. Those can break quickly.
| actionfromafar wrote:
| Hm, vulns are more common in libraries in general than in
| the compiler, I'd say. Or am I misunderstanding?
| travisjungroth wrote:
| By "discovered vulnerabilities", I mean a security issue
| that wasn't known when the library was first written but
| then came to be known. This is what's fixed when a
| library is maintained.
|
| This entirely depends on the library, but I just
| generally don't see a lot of security fixes in library
| updates. But for compiler updates, I do.
|
| I'm speaking _super_ broadly, and this will be very
| different for a Python graphing library versus a C
| networking library.
| bluGill wrote:
| The vulnerability is almost never in the compiler (not
| never - I have seen a case, but very rare). Most attacks
| are in the library itself. If your library has a buffer
| overflow you are vulnerable. If your library has a C
| style buffer length + size parameters and you mess them
| up is it the libraries fault for such a bad API?
| ramses0 wrote:
| Reading through, they're describing kindof "eras" of best
| practices. Like "hex" and "base64" might never change (so
| might never update), but those are kindof prime name
| tokens.
|
| It made me think: `std::hex,base64` vs `boost::hex,base64`,
| but then assuredly those would be incompatible because
| $REASONS.
|
| ...but what if it was like: `v2025::std::hex,base64` vs
| `boost::v2025::std::hex,base64` (ie: explicitly stating
| you're adhering to the v2025 guidelines w.r.t. memory
| management or parameter naming or whatnot).
|
| It's a roundabout way of saying: `tokio::async_foo`,
| `boost::async_bar`, `v2026::std::async::foo,bar`, where as
| the marketplace of ideas settles on "better" ways of
| dealing with async (or whatever) there can eventually be
| compatibility between different object "modalities" (ways
| of working).
|
| `std::hex,base64` should be quite stable, but having a path
| for `...::webapp:: _` and `...::database::_ ` to eventually
| interop seems really useful for a language to encourage?
| queuebert wrote:
| I like Donald Knuth's approach where his version numbers of
| TeX and Metafont asymptotically approach pi and e,
| respectively, emphasizing the slow approach toward
| perfection.
| hobs wrote:
| And communicating nothing to the end user except a sense of
| whimsy? Nah, that's useless.
| wtetzner wrote:
| What do the version numbers of programs communicate to
| end users anyway? Basically just that it's newer than the
| last version.
|
| Libraries are a bit different though, because of semver
| compatibility, but unfortunately they are wrong often
| enough that you can't really rely on them anyway.
| DannyBee wrote:
| Agreed -
|
| It's beyond wrong. For example, at the core of plenty of
| numerical libraries is 30+ year old fortran code from netlib
| that works great. It's just done.
|
| It does what it's supposed to. It does not have meaningful
| bugs. There is no reason to change it.
|
| Obviously, one can (and people do) rewrite it in another
| language to avoid having to have a fortran compiler around,
| or because they want to make some other tradeoff (usually
| performance).
|
| But it's otherwise done. There is no need to make new
| releases of the code. It does what it is supposed to do.
|
| That's what you want out of libraries past a certain point -
| not new features, you just want them to do what you want and
| not break you.
| estebank wrote:
| > It's not like libraries accumulate new bugs and
| vulnerabilities by not changing.
|
| That's not actually true, for libraries that interact with
| the platform. For example, Mac OS changed some type
| definition in header files in their arch transition, which
| makes building a pre-M1 rust project that depends on a
| contemporary library version that interfaces with the OS
| (like SDL) on an ARM host without _any_ changes impossible.
| You need to update your dep (and potentially the way you
| consume them) just to be able to build the project, or
| procure a host supported by your dependency (one that existed
| when it was written, so an X86 machine).
|
| We could argue all day that this is "not a bug" or the user's
| fault for using a new host and it was never supported or any
| other deflection. But it is a concrete example of "this code
| hasn't changed and the passage of time shifted the ground
| from under it".
| tmtvl wrote:
| If the author intends for:
|
| > _apparently abandoned_
|
| To mean:
|
| > _hasn 't changed in a while_
|
| Then I will agree that that doesn't need to mean it has to be
| laden with problems. I was going more on the basis of it
| meaning that there are a number of unresolved bug reports as
| well as a lack of activity. In the Common Lisp community
| there is generally agreement that software can be simply
| finished (although there is also an amount of broken
| abandonware).
|
| My main point is if people are using libraries with problems
| that aren't getting fixed, they shouldn't be afraid to take
| over maintenance or to migrate to libraries which don't have
| those problems.
| lmm wrote:
| Why is it that the frozen semantics are actually needed? Is there
| no way to represent what people actually want here - memory that
| is entirely uninitialised, for which tautology might be false,
| until written? I.e. something that's a bit like MaybeUninit but
| more so?
| api wrote:
| There's probably no way for the compiler to prove safety. Rust
| is designed to allow 100% safe bare metal development, like a
| perfectly safe C that still allows you to get close to the
| hardware, and that's tough.
| ajross wrote:
| I'm failing to understand the correlation to "safety" here.
| Reading a byte for which you don't know the value isn't
| "unsafe". It's literally (!) the desired behavior of foreign
| data being read from an external source, which is in fact the
| use case in the article.
|
| There's no safety problem as long as the arbitrary value is
| deterministic, which it is, being process RAM. The
| "uninitialized data read" bugs reported from instrumentation
| tools in C code are because the code is assuming the value
| has some semantics. The read itself has no value and is
| presumably an artifact of the bug, _but it is safe_.
| gpm wrote:
| > There's no safety problem as long as the arbitrary value
| is deterministic, which it is, being process RAM.
|
| The article discusses how it is in fact, on Linux with
| memory returned from at least one very common allocator,
| not deterministic. Ctrl-f tautology.
| ajross wrote:
| That's just a terminology collision. All RAM access is
| deterministic in the sense that the value will not change
| until written. It's not "predictable" in the sense that
| the value could be anything.
|
| C code that reads uninitialized data is presumed to be
| buggy, because it wouldn't be doing that unless it
| thought the memory was initialized. But _the read itself
| is not unsafe_.
|
| Rust is just confused, and is preventing all reads from
| uninitialized data a-priori instead of relying on its
| perfectly working type system to tell it whether the
| uninitialized data is safe to use. And that has
| performance impact, as described in the linked article,
| which has then resulted in some terrible API choices to
| evade.
| gpm wrote:
| > All RAM access is deterministic in the sense that the
| value will not change until written.
|
| Again, the article literally points to how this is not
| true given modern allocators. The memory that Linux
| exposes to processes will change without being written to
| prior to being initialized given how allocators manage
| it. This isn't a fiction of the C-standard or rust
| reference, it's what actually happens in the real world
| on a regular basis.
|
| Rust is not confused, it is correctly observing what is
| allowed to actually happen to uninitialized memory while
| the process does nothing to it.
|
| You could change the C/Rust specification of that memory.
| You could in your C/rust implementation declare that the
| OS swapping out pages of uninitialized memory counts as a
| write just like any other, and that it's the programmers
| (allocators) responsibility to make sure those writes
| obey the normal aliasing rules. Doing so would be giving
| _up_ performance though, because the fact that writing to
| memory has the side-effect of cancelling collection of
| freed pages is a powerful way for processes to quickly
| communicate with the OS. (You 'd probably also cause
| other issues with memory mapped IO, values after the end
| of the stack changing, and so on, but we can just focus
| on this one issue for now).
| vlovich123 wrote:
| I don't understand your point and you're wrong on a
| couple of things.
|
| > C code that reads uninitialized data is presumed to be
| buggy, because it wouldn't be doing that unless it
| thought the memory was initialized. But the read itself
| is not unsafe. Rust is just confused, and is preventing
| all reads from uninitialized data a-priori instead of
| relying on its perfectly working type system to tell it
| whether the uninitialized data is safe to use
|
| Reads of uninitialized memory is unsafe full stop. That's
| literally what Rust's memory safety is about. If you give
| that up you're not in safe land and you can always use
| unsafe & all the risks that come with that to try to
| write more optimal code / abstractions.
|
| This article is literally about the mechanisms Rust is
| trying to stabilize how you use the type system to go
| from a block of uninitialized memory & is aware of writes
| occuring so that it can take a &[MaybeUninit<T>] and give
| you back a &[T] after a call to read which wrote into the
| slice. But reading uninitialized memory by definition is
| tautologically unsafe. It doesn't mean that the computer
| will pull out a knife and kill you, but it does mean
| you're no longer memory safe.
| paulddraper wrote:
| > All RAM access is deterministic in the sense that the
| value will not change until written.
|
| No, this is directly addressed in the article.
|
| RAM access to _uninitialized_ memory is not deterministic
| and can change. See MADV_FREE.
| ajross wrote:
| That's a VM feature. I mean, yes, if you change your
| process's memory space between accesses you break
| whatever the compiler might have assumed. You don't need
| fancy flags, either! Just, y'know, munmap() will do.
|
| This is yet another unrelated tangent. Can you explain
| why it it you think the presence of MADV_FREE disallows
| Rust from allowing writes to uninitialized memory?
| db48x wrote:
| You're missing the point. It doesn't matter that this is
| a VM feature and not a hardware feature. The fact that it
| exists at all means that reading from uninitialized
| memory is unsound. You can write to it all you want, but
| not read from it.
| ajross wrote:
| Once more, the use case in question is _WRITING_ to
| uninitialized memory, not reading from it. Rust is
| applying the constraints of the latter (and thus
| requiring "unsafe") to the former, which is not an
| unsafe operation.
| gpm wrote:
| No. Rust is only applying any constraints to reading from
| it.
|
| The entire discussion is how to make an API which defines
| which portions of the memory can be read from because
| they have been written to, so we can expose an API that
| cannot be misused. There have never been any constraints
| on writing to uninitialized memory.
| hansvm wrote:
| > All RAM access is deterministic in the sense that the
| value will not change until written
|
| That's correct at a hardware level. It's not correct from
| a userspace program's perspective when interacting with
| Linux if the memory in question is uninitialized.
| GrantMoyer wrote:
| > That's correct at a hardware level.
|
| Not even there, due to memory caches.
| GrantMoyer wrote:
| You have some misconceptions about C and undefined
| behavior.
|
| > C code that reads uninitialized data is presumed to be
| buggy, because it wouldn't be doing that unless it
| thought the memory was initialized. But the read itself
| is not unsafe.
|
| The read itself _is very much unsafe_ because it 's
| undefined behavior. The compiler is allowed to assume the
| programmer doesn't allow reads to uninitialized memory to
| happen, so if the programmer does allow a read from
| uninitialized memory, any false conclusion can follow
| from the false assumption.
|
| This is a problem even in trivial cases; see [1] and try
| commenting out and switching arround the calls to foo and
| bar. The behavior is very unintuitive, because reads from
| uninitialized memory is unsafe.
|
| [1]: https://godbolt.org/z/71vbPYT6G
| ajross wrote:
| > You have some misconceptions about C and undefined
| behavior.
|
| The discussion is about RAM and Rust, not C. And the
| particular use of uninitialized data in C that
| corresponds to the linked article (as a target buffer for
| a read call) is clearly not undefined behavior.
|
| This is a classic HN tangent, basically. You're making
| the discussion worse and not better.
|
| The question is "Why can't Rust act like C when faced
| with empty buffers?", and the answer has nothing to do
| with undefined behavior or unsafe. They just got it
| wrong.
| GrantMoyer wrote:
| Okay, I think I see. In the linked article, due to the
| behavior of `read` uninitialized memory is never read.
| The same would be true in equivalent C code.
|
| However, in C, the programmer doesn't need to prove to
| the compiler that uninitialized memory is never read,
| they are just expected to prevent it from happening. In
| this case, it's clear to the programer that there's no
| undefined behavior.
|
| In Rust though, the compiler must be able to statically
| verify no undefined behavior can occur (except due to
| unsafe sections). It's not possible to statically verify
| this in either the Rust or C case, because not enough
| information is encoded into the type signature of `read`.
| The article discusses a couple of ways that information
| might be encoded so that Rust can be more like C, and
| discusses their trade-offs. C explicitly sidesteps this
| by placing the responsibility entirely on the programmer.
|
| So to directly answer your question "Why can't Rust act
| like C when faced with empty buffers?", it's because the
| Rust compiler cannot yet statically verify there's no
| undefined behavior in this case, even though there is in
| fact no undefined behavior, and one of the primary design
| goals of Rust is to statically prevent undefined
| behavior.
|
| And to what's perhaps the initial question, this is
| discussed using the term "safety" simply because Rust
| defines things which can't be statically verified to not
| invoke undefined behavior as "unsafe". Perhaps a better
| term would be "not yet statically provable as safe", but
| it's a bit of a mouthful.
| ajross wrote:
| > it's because the Rust compiler cannot yet statically
| verify there's no undefined behavior in this case
|
| Uh... yes it can. It's a memory write to the
| uninitialized region. Writes are not undefined, nor
| unsafe, and never have been. They aren't in C, they
| aren't in hardware. Writes are _fine_.
|
| The bug here is API design, not verification constraints.
| GrantMoyer wrote:
| The issue isn't writes to uninitialized memory, it's
| reads from uninitialzed memory. The compiler doesn't know
| how much of the buffer `read` writes. The docs say it
| returns a unsigned integer with how many bytes it wrote,
| so a programmer can know the later read from
| `buffer[0..num_bytes_written]` is valid, but the compiler
| doesn't know what the number returned from `read`
| represents, so from the compiler's point of view, the
| whole buffer needs to be initialized regardless of what
| read does for reads from it to be valid. That means it
| has to be initialized before it's passed to read,
| otherwise the compiler can't prove the elements which are
| later read from the buffer are initialized.
| MrBuddyCasino wrote:
| What does safe mean here? Everything can be interpreted as a
| [u8], right?
| lidavidm wrote:
| https://www.ralfj.de/blog/2019/07/14/uninit.html perhaps
| (the OP also talks about this when linking to a talk about
| jemalloc)
| gpm wrote:
| [u8] guarantees to the compiler that two reads through the
| array at the same location without any intervening writes
| return the same value.
|
| Turns out that's not the case on freshly returned
| uninitiated allocations. The first read could return old
| data (say "1"), and the second read could return a freshly
| zeroed page ("0").
| leni536 wrote:
| Do some Rust types have invalid object representation or trap
| representation? On SysV x86_64 bool only has two valid
| representations in memory, the rest are trap representations.
|
| So for an array of bools (if Rust matches SysV) freeze wouldn't
| be sound, even without the madvise problem.
| gpm wrote:
| Yes.
|
| That's what "// SAFETY: u8 has no invalid bit patterns." is
| discussing. That while types in general can u8 specifically
| does not (none of the u*/i* integer types do) so freezing a
| buffer of u8s is sound.
| kmeisthax wrote:
| Yes; and furthermore LLVM also has undef, which is sort of a
| trap representation, but it only exists in the optimizer.
| (There's also poison for overflow, which is a strictly less
| defined value than undef.)
|
| I _suspect_ even reading an array of uninitialized u8s would
| cause havoc just from LLVM miscompiles alone.
| queuebert wrote:
| The more pertinent question to me is can we implement some new
| static analysis that understands buffer re-use and can hoist
| buffer initialization outside the loop? Rather than make the
| programmer write obfuscated code for efficiency, it is usually
| better to have the compiler do the heavy lifting.
|
| P.S. Also, folks, don't re-use buffers without zeroing unless you
| absolutely need the performance and know what you're doing.
| vlovich123 wrote:
| I like that direction better but it requires the ability to
| declare data-flow based contracts whereas Rust's tools are only
| lifetime and type contracts. Is there a language that has data-
| flow based contracts?
| queuebert wrote:
| That would be easier but is not required. There are no
| compiler hints these days to unroll loops or hoist
| invariants, even though if done incorrectly it could change
| the result. It would take some complicated analysis, but I
| think it could be done safely in some cases.
| gpm wrote:
| I was going to make this argument, but I actually don't
| think it's true in almost any case.
|
| Most functions could be inferred, but the ultimate source
| of basically all of these write only APIs is FFI functions,
| which in turn call systemcalls.
|
| You're at least going to need a way to annotate the FFI
| calls and systemcalls to describe to the compiler how they
| access data.
| queuebert wrote:
| If you're calling FFIs in an inner loop, you have bigger
| issues than the time it takes to clear the buffer, right?
| gpm wrote:
| No? It depends on your definition of inner loop I guess.
|
| If you're doing some sort of zero-copy IO, the time to
| clear the buffer might be non-trivial (not huge, but non-
| trivial). It's true that you need a large enough buffer
| that syscall/ffi overhead doesn't dominate, but that's
| not unrealistic.
|
| It's rare that we care about this, that's true, that's
| why generally rust has been fine with "just zero
| buffers". There are definitely domains that care though.
| vlovich123 wrote:
| The loop unrolling & invariant hoisting is a static
| transformation. What the "read" function does semantically
| isn't captured _today_ within that and the compiler
| wouldn't be able to automatically infer it. It would have
| to be told that information and there would need to be
| unsafe annotations for things like syscalls and FFI
| boundaries. The other approach is to change the API which
| is what BorrowedBuf is.
|
| If you can think of a different approach of how the
| compiler can figure out automatically what memory has
| become initialized by a random function call I'm all ears.
| queuebert wrote:
| That's what I glossed over as "complicated analysis". In
| my mind, if a compiler can understand register and stack
| use (required for static transformations), it can
| (theoretically, and with some effort) understand heap
| use. Am I wrong?
| ijustlovemath wrote:
| what do you mean by this?
| vlovich123 wrote:
| There would need to be contractual declarations on the read
| method that the compiler is able to enforce that tells it
| that the input &mut slice has N elements clobbered based on
| the returned length. That's basically what BorrowedBuf is
| accomplishing via the type system and runtime enforcement
| of the contract. Using a non-existent syntax:
| fn read<T, N: size_t>(&mut self, buf: &mut [MaybeUninit<T>]
| becomes &[T; N] after call) -> N { ... enforces
| the body initializes N elements out of buf }
|
| and then rules that &mut [T] can _also_ be supplied to such
| functions that today could only accept a &mut
| [MaybeUninit<T>] transparently.
|
| A more likely interface you could write today would look
| like: fn read_uninit<T>(&mut self, buf:
| &mut [MaybeUninit<T>]) -> (&[T], &[MaybeUninit<T>]) {
| ... enforces the body initializes N elements out of buf
| }
|
| You still have to cast &[T] into &[MaybeUninit<T>] somehow.
| Someone wrote:
| I think an ergonomic way to do that would to have _read_
| return not an integer, but a slice of that integer's
| length.
|
| Problem would be: how do you express "you can only access
| the buffer you sent me through the read-only slice I
| returned, but you have to free that same buffer when
| you're done calling me?
|
| I think that can be done using a function creating a read
| buffer for a given input stream that
|
| - during calls to _read_ is 'owned for writing' by that
| stream (so, it has to borrow a capability that the
| creator of the buffer doesn't have. I don't think Rust
| currently supports that)
|
| - where _stream.read_ returns a read only slice whose
| lifetime is bound to that of the buffer
|
| So, the creator of the buffer can only pass it to _read_
| to get a slice back that contains precisely the data
| read.
|
| The stream can write to the entire buffer.
| gpm wrote:
| > You still have to cast &[T] into &[MaybeUninit<T>]
| somehow. unsafe{
| std::mem::transmute(slice) }
|
| This is probably the only way that will ever exist,
| because let slice: &mut [NonZeroU8] =
| ...; let slice_uninit: &mut
| [MaybeUninit<NonZeroU8>] = ...; let
| nonzero_uninit: &mut MaybeUninit<NonZeroU8> = &mut
| slice_uninit[0]; *nonzero_uninit =
| MaybeUninit::zeroed(); slice[0]; // Undefined
| behavior for sure by now.
|
| Is all safe except for the cast.
|
| I.e. MaybeUninit<T> allows you to write invalid bit-
| patterns to T, so you can't safely cast a reference to T
| to it (and if you do unsafely cast a reference to T to it
| you can't soundly write an invalid bit pattern). All
| current forms of safely making a MaybeUninit take
| ownership of the value they are declaring to be
| MaybeUninit for this reason.
|
| I guess at some point we might get methods for this on
| types that can take on all bit patterns - if/when that's
| encoded as a trait.
| mrpf1ster wrote:
| Why would re-using a buffer be bad? Assuming you write to it
| with the contents of the file/stream before it is read.
| kohbo wrote:
| You just answered your own question
| rendaw wrote:
| Why is it particularly more dangerous or likely than other
| logic errors?
| benschulz wrote:
| Because the compiler optimizes based on the assumption
| that consecutive reads yield the same value. Reading from
| uninitialized memory may violate that assumption and lead
| to undefined behavior.
|
| (This isn't the theoretical ivory tower kind of UB.
| Operating systems regularly remap a page that hasn't yet
| been written to.)
| fnordpiglet wrote:
| I think they implied you would prevent that.
| phkahler wrote:
| And that's not something you should be depending on a
| compiler to verify.
| jvanderbot wrote:
| Fair, but note there is a significant subset of Rust-targeted
| programmers who dislike the compiler doing things like that.
| They _also_ dislike the compiler doing things like auto-
| initializing every loop iteration, but two wrongs wouldn 't
| make it right, just less wrong.
| DannyBee wrote:
| This seems not even close to the worst feature of rust - this
| seems like it needs some more design work and baking. Like lots
| of things.
|
| The amount of hyperbole in this article makes it a bit hard to
| take the author all that seriously.
|
| Is there evidence more baking won't happen? While i have my loves
| and hates about rust, it definitely always felt like they had a
| pretty thorough/careful process for additions like this. If you
| go constructively into the threads and offer some concerns, you
| will usually get some reasonable response.
|
| (All processes of course, fail, so this is not always true, but
| it's mostly true)
|
| While i think it's fine to write rants on blogs, and don't feel
| like everyone has a responsibility to file bugs or whatever
| before they write a rant about it, if you actually want to see
| this "worst feature" fixed, this probably won't help very much.
|
| (IE You don't _have_ to be constructive, or even helpful, but if
| you _want_ to be constructive or helpful, this ain 't how you do
| it)
| Xunjin wrote:
| I'm not as much low-level development as the author seems to
| be, but the hyperbole made me think "you have a point, which
| is/can be valid, but aren't you stretching the reasons to fit
| in your sentiment/PoV?"
|
| Being honest, plenty of times we throw "Ergonomics" as an
| argument, however, are ergonomics more a feeling of how good
| are the API usage instead of actually prove with examples and
| design choices?
| dathinab wrote:
| > Is there evidence more baking won't happen?
|
| No, actually there is a lot of evidence that it will still be
| worked on.
|
| Normally I would just say look at the issue linked in the
| nightly docs but due to an overlap of tacking PR and moving it
| from std to core PR it's not supper useful.
|
| Tracking Issue: https://github.com/rust-lang/rust/issues/78485
|
| If the author has constructive critique they should probably
| mention it there (after skimming through the discussion to make
| sure this wasn't already considered and not done due to
| subtleties they overlooked in the blog post (like e.g. that
| it's a standard/core feature which has to work across all
| targets and as such can't rely on anything being initialized to
| 0 by the OS, or that depending on the global allocator used you
| definitely can't rely on things being zeroed even if the OS
| only hands out zeroed memory, etc. etc.))
| Zagitta wrote:
| Maybe Rust needs another type of reference that's exclusive write
| only? Right now there's RO (&T) and exclusive RW (&mut T) but WO
| is missing.
|
| Having a WO reference would allow these read_buf APIs to express
| they only write and never read so the uninitialized memory is
| safe to pass directly.
| gpm wrote:
| In some sense that's exactly what a `&mut MaybeUninit<T>` is?
| mmastrac wrote:
| Probably more once https://doc.rust-
| lang.org/beta/std/mem/union.MaybeUninit.htm... is no longer
| nightly-only.
| 1oooqooq wrote:
| everyone just tell you to use mpsr in this case
| breaker-kind wrote:
| what does the term "nightly" mean in this context?
| seanhunter wrote:
| Rust has "nightly" and "stable" toolchain variants called
| "release channels". The "nightly" version is literally just the
| compiler and all the other tools built in the CI/CD pipeline on
| a nightly basis with the latest branch including various
| features that need further development before they go to
| stable.
|
| The process and reasoning are described here https://doc.rust-
| lang.org/book/appendix-07-nightly-rust.html
| camdenreslink wrote:
| It is the version of rust that is under active development, and
| also contains experimental features that aren't in the "stable"
| rust compiler.
| jasonthorsness wrote:
| I am not yet a Rust programmer but - is it not typical to have a
| small collection of unsafe functions, carefully reviewed, that in
| this case seem like they might be easier to maintain than some of
| these convoluted type-based workarounds?
| steveklabnik wrote:
| It's a tradeoff! You have to explore both options to know which
| side of the tradeoff to take.
| Matthias247 wrote:
| If this is mainly useful for working with plain/uninterpreted
| byte arrays, then I wonder why we can't just do `[u8;
| N]::with_noinit()` method instead of doing the multi-line plus
| unsafe things listed in the article.
|
| Is the main point that things like `slice_freeze_mut` could also
| be used for slices of e.g. `struct Coordinate { x: u32, y: u32,
| z: u32 }`?
|
| It would obviously not work for f64 things, since there also not
| all bit-patterns are valid.
| orlp wrote:
| All f64 bit patterns are valid.
| umanwizard wrote:
| > While replacing the array of zeros by an array of uninitialised
| values may work in specific circumstances, the code is unsound.
| Change to the compiler, its options, modification of unrelated
| parts of the code or using the function for a different Read
| trait implementation may break the program in unpredictable ways.
|
| Why? It seems the only thing on that list that will cause UB is
| using the function with a different reader (one that inspects the
| uninitialized bytes). Why would any of the other listed possible
| changes break it?
| steeeeeve wrote:
| The selling point of Rust was that it protects programmers from
| doing dangerous things.
| steveklabnik wrote:
| This is a good first approximation, but it misses something.
| It's actually that it protects programmers from _accidentally_
| doing dangerous things. There 's a lot of support in the
| language for doing dangerous things, you just have to
| explicitly say "hey I know I'm doing something dangerous, and I
| promise I'm right here."
| pnathan wrote:
| I think the answer is that in a case when you need that speed,
| you hoist the stack allocation & zeroing and unsafe that buffer
| in the loop if need be. Test well. I am a huge Rust fan but also
| it is actually possible to write correct unsafe code.
|
| If I am interacting with from IO space, I would much rather write
| the interaction code myself for the machine at hand than farm it
| out to an array of third party crates. ::shrug::
|
| getting the machinery to let it properly be hoisted smoothly and
| safely would be nice, but it isn't required.
|
| personally I think rust macros are very painful and the "worst
| feature", but that's speaking as someone who did a fair bit of
| Common Lisp.
___________________________________________________________________
(page generated 2025-01-30 23:02 UTC)