[HN Gopher] There is no memory safety without thread safety
___________________________________________________________________
There is no memory safety without thread safety
Author : tavianator
Score : 225 points
Date : 2025-07-24 15:35 UTC (7 hours ago)
(HTM) web link (www.ralfj.de)
(TXT) w3m dump (www.ralfj.de)
| jchw wrote:
| This comes up now and again, somewhat akin to the Rust soundness
| hole issue. To be fair, it is a legitimate issue, and you could
| definitely cause it by accident, which is more than I can say
| about the Rust soundness hole(s?), which as far as I know are
| basically incomprehensible and about as likely to come across
| naturally as guessing someone's private key.
|
| That said in many years of using Go in production I don't _think_
| I 've ever come across a situation where the exact requirements
| to cause this bug have occurred.
|
| Uber has talked a lot about bugs in Go code. This article is
| useful to understand some of the practical problems facing Go
| developers actually wind up being, particularly the table at the
| bottom summarizing how common each issue is.
|
| https://www.uber.com/en-US/blog/data-race-patterns-in-go/
|
| They don't have a specific category that would cover this issue,
| because most of the time concurrent map or slice accesses are on
| the same slice and this needs you to exhibit a torn read.
|
| So why doesn't it come up more in practice? I dunno. Honestly
| beats me. I guess people are paranoid enough to avoid this
| particular pitfall most of the time, kind of like the Technology
| Connections theory on Americans and extension
| cords/powerstrips[1]. Re-assigning variables that are known to be
| used concurrently is obvious enough to be a problem and the
| language has atomics, channels, mutex locks so I think most
| people just don't wind up doing that in a concurrent context (or
| at least certainly not on purpose.) The race detector will
| definitely find it.
|
| For some performance hit, though, the torn reads problem could
| just be fixed. I think they should probably do it, but I'm not
| losing sweat over all of the Go code in production. It hasn't
| really been a big issue.
|
| [1]: https://www.youtube.com/watch?v=K_q-xnYRugQ
| bombela wrote:
| It took months to finally solve a data race in Go. No race
| detector would see anything. Nobody understood what was
| happening.
|
| It ultimately resulted in a loop counter overflowing, which
| recomputed the same thing a billion of time (but always the
| same!). So the visible effect was a request would randomly take
| 3 min instead of 100ms.
|
| I ended up using perf in production, which indirectly lead me
| to understand the data race.
|
| I was called in to help the team because of my experience
| debugging the weirdest things as a platform dev.
|
| Because of this I was exposed to so many races in Go, from my
| biased point of view, I want Rust everywhere instead.
|
| But I guess I am putting myself out of a job? ;)
| jchw wrote:
| I think the true answer is that the moment you have to do
| tricky concurrency in Go, it becomes less desirable. I think
| that Go is still better at tricky concurrency than C, though
| there are some downsides too (I think it's a bit easier to
| sneak in a torn read issue in Go due to the presence of fat
| pointers and slice headers everywhere.)
|
| Go is _really_ good at easy concurrency tasks, like things
| that have almost no shared memory at all, "shared-nothing"
| architectures, like a typical web server. Share some
| resources like database handles with a sync.Pool and call it
| a day. Go lets you write "async" code as if it were sync with
| no function coloring, making it decidedly nicer than
| basically anything in its performance class for this use
| case.
|
| Rust, on the other hand, has to contend with function
| coloring and a myriad of seriously hard engineering tasks to
| deal with async issues. Async Rust gets better every year,
| but personally I still (as of last month at least) think it's
| quite a mess. Rust is absolutely _excellent_ for traditional
| concurrency, though. Anything where you would 've used a
| mutex lock, Rust is just way better than everything else.
| It's beautiful.
|
| But I struggle to be _as_ productive in Rust as I am in Go,
| because Rust, the standard library, and its ecosystem gives
| the programmer so much to worry about. It sometimes reminds
| me of C++ in that regard, though it 's nowhere near as
| extremely bad (because at least there's a coherent build
| system and package manager.) And frankly, a lot of software I
| write is just _boring_ , and Go does fine for a lot of that.
| I try Rust periodically for things, and romantically it feels
| like it's the closest language to "the future", but I think
| the future might still have a place for languages like Go.
| zozbot234 wrote:
| > And frankly, a lot of software I write is just boring,
| and Go does fine for a lot of that. I try Rust periodically
| for things, and romantically it feels like it's the closest
| language to "the future", but I think the future might
| still have a place for languages like Go.
|
| It's not so much about being "boring" or not; Rust does
| just fine at writing boring code once you get familiar with
| the boilerplate patterns (Real-world experience has shown
| that Rust is not really at a disadvantage wrt. productivity
| or iteration speed).
|
| There is a case for Golang and similar languages, but it
| has to do with software domains where there literally is no
| viable alternative to GC, such as when dealing with
| arbitrary, "spaghetti" reference graphs. Most programs
| aren't going to look like that though, and starting with
| Rust will yield a higher quality solution overall.
| jchw wrote:
| Rust _can_ yield a higher quality solution, but we can 't
| make a perfect solution, we can only approach perfection.
| If we want to go further, we could introduce formally-
| proven code, too. Personally I'm interested in the
| intersection of proof assistants and Rust, like creusot-
| rs, and have been investigating it.
|
| But as much as I love LARPing about correctness (believe
| me I do,) it's just simply the case that we won't right
| perfect software and it's totally OK. It's totally OK
| that our software will have artificial limitations, like
| with Go, only accepting filenames that are valid UTF-8,
| or taking some unnecessary performance/latency hits, or
| perhaps even crashing in some weird ass edge case. There
| are _very_ few domains in which correctness issues _can
| 't_ be tolerated.
|
| I don't deal with domains that are truly mission
| critical, where people could die if the code is
| incorrect. At worst, people could lose some money if my
| code is incorrect. I still would prefer not to cause that
| to happen, but those people are generally OK with taking
| that risk if it means getting features faster.
|
| That's why Go has a future really. It's because for most
| software, some correctness issues are not the end of the
| world, and so you can rely on not fully sound approaches
| to finding bugs, like automated testing, race detection,
| and so on.
|
| Rust can also make some types of software more productive
| to write, but it is unlikely to beat Go in terms of
| productivity when it comes to a lot of the stuff SaaS
| shops deal with. And boy, the software industry sure is
| swamped in fucking SaaS.
| sophacles wrote:
| A lot of sass people i know are more and more choosing
| rust for boring code. This includes several people who
| said things like "go is good enough, i don't want to deal
| with all the rust completely".
|
| Once your sass products get enough users, and you're
| dealing with millions or billions of requests per day,
| those rare bugs start showing up quite often... And it
| turns out programming towards correctness is desirable,
| if for no other reason than to keep pagerduty quiet.
| Tolerating correctness issues isn't cost-free... People
| having to respond during off hours costs money and
| stress. I think most people would rather pay the costs at
| dev time, when they aren't under the pressure of an
| incident, than during an outage.
| zozbot234 wrote:
| This is also in line with everything we know about good
| software engineering. Putting out fires in production is
| extremely costly, hence potential issues should be
| addressed at the earliest feasible stage.
| jchw wrote:
| But correctness is not binary, it's more like a
| multidimensional spectrum. Your choice of programming
| language has some influence, as does standards and
| conventions, the ecosystem of your programming language,
| use of automated tooling like linting and testing, or
| even just ol' unreliable, discipline. Being a relatively
| greenfield language, Go is not in a terrible place when
| it comes to most of those things. Tons of automated
| tooling, including tools like the Checklocks analyzer or
| the many tools bundled with golangci-lint. Uber has done
| a pretty good job enumerating the challenges that remain,
| and even working at improving those issues too, such as
| with NilAway.
|
| The question isn't "wouldn't you prefer more
| correctness?" it's "how much would you pay for how much
| of an improvement in correctness?".
|
| Rust is still growing rapidly though, whereas Go is
| probably not growing rapidly anymore, I think Go has at
| least saturated it's own niche more than 50% and is on
| the other end of the curve by now. Last I checked Rust is
| the trendiest language by far, the one that people most
| wish they were writing, and the one that you want to be
| able to say your project is written in. So it would be
| extremely surprising to hear if there _wasn 't_ a growing
| Rust presence basically everywhere, SaaS's included.
| Mawr wrote:
| > (Real-world experience has shown that Rust is not
| really at a disadvantage wrt. productivity or iteration
| speed).
|
| I don't believe that for a second. Even just going from
| Python to Go drops my productivity by maybe about 50%.
| Rust? Forget it.
|
| Sure, if you have a project that demands correctness
| _and_ high performance that requires tricky concurrency
| to achieve, something like Rust may make sense. Not for
| your run-of-the-mill programs though.
| Yoric wrote:
| Hey, going from Rust to Go drops my productivity by maybe
| about 50% :)
|
| But more seriously, yeah, Rust doesn't make sense for
| trivial programs. But these days, I write Python for a
| living, and it doesn't take long to stumble upon bugs
| that Rust would have trivially detected from within the
| comfort from my IDE.
| bombela wrote:
| It wasn't really tricky concurrency. Somebody just made the
| mistake of sharing a pointer across goroutines. It was
| quite indirect. Boils down to a function takeing a param
| and holds onto it. `go` is used at some point closing over
| this pointer. And now we have a data race in the waiting.
| norir wrote:
| It is very unfortunate that we use fixed width numbers by
| default in most programming languages and that common ops
| will silently overflow. Smarter compilers can work with
| richer numeric primitives and either automatically promote
| machine words to big numbers or throw an error on overflow.
|
| People talk a lot about the productivity gains of ai, but
| fixing problems like this at the language level could have an
| even bigger impact on productivity, but are far less
| sensational. Think about how much productivity is lost due to
| obscure but detectable bugs like this one. I don't think rust
| is a good answer (it doesn't check overflow by default), but
| at least it points a little bit in the vaguely correct
| direction.
| recursivecaveat wrote:
| The situation with numbers in basically every widely used
| programming language is kind of an indictment of our
| industry. Silent overflow for incorrect results, no
| convenient facilities for units, lossy casts everywhere.
| It's one of those things where standing in 1975 you'd think
| surely we'll spend some of the next 40 years of performance
| gains to give ourselves nice, correct numbers to work with,
| but we never did.
| tomp wrote:
| "nice, correct numbers" end somewhere between 1/3 and
| sqrt(2)
|
| so in reality, it's just "pick your own poison" to
| various degrees...
| devnullbrain wrote:
| Rust checks overflow by default in debug builds
| Thaxll wrote:
| Rust does have loop counter overflow.
| swiftcoder wrote:
| Theoretically you can construct a loop counter that
| overflows, but I don't that there is any reasonable way to
| do it accidentally?
|
| Within safe rust you would likely need to be using an
| explicit .wrapping_add() on your counter, and explicitly
| constructing a for loop that wasn't range-based...
| tekne wrote:
| Well, in debug mode yes, but in release mode overflow is
| wrapping by default unless you explicitly set a flag to
| make it panic.
| jason_oster wrote:
| This is irrelevant on 64-bit platforms [^1] [^2]. For
| platforms with smaller `usize`, enable overflow-checks in
| your release builds.
|
| [^1]: https://www.reddit.com/r/ProgrammerTIL/comments/4tsps
| n/c_it_...
|
| [^2]: https://stackoverflow.com/questions/69375375/is-it-
| safe-to-a...
| wavemode wrote:
| > It ultimately resulted in a loop counter overflowing, which
| recomputed the same thing a billion of time (but always the
| same!). So the visible effect was a request would randomly
| take 3 min instead of 100ms.
|
| This means that multiple goroutines were writing to the same
| local variable. I've never worked on a Go team where code
| that is structured in such a way would be considered normal
| or pass code review without good justification.
| qcnguy wrote:
| What do Uber mean in that article when they say that Go
| programs "expose 8x more concurrency compared to Java
| microservices"? They're using the word concurrency as if it
| were a countable noun.
| wmf wrote:
| If the Java version creates 4 concurrent tasks (could be
| threads, fibers, futures, etc.) but the Go version creates 32
| goroutines, that's 8x the concurrency.
| ameliaquining wrote:
| I think it's also worth noting that Rust's maintainers
| acknowledge its various soundness holes as bugs that need to be
| fixed. It's just that some of them, like
| https://github.com/rust-lang/rust/issues/25860 (which I assume
| you're referring to), need major refactors of certain parts of
| the compiler in order to fix, so it's taking a while.
| advisedwang wrote:
| Wow that's a really big gotcha in go!
|
| To be fair though, go has a big emphasis on using its
| communication primitives instead of directly sharing memory
| between goroutines [1].
|
| [1] https://go.dev/blog/codelab-share
| zozbot234 wrote:
| Real-world golang programs share memory all the time, because
| the "share by communicating" pattern leads to pervasive logical
| problems, i.e. "safe" race conditions and "safe" deadlocks.
| jrockway wrote:
| I am not sure sync.Mutex fixes either of these problems.
| Press C-\ on a random Go server that's been up for a while
| and you'll probably find 3000 goroutines stuck on a Lock()
| call that's never going to return. At least you can time out
| channel operations: select { case
| <-ctx.Done(): return context.Cause(ctx)
| case msg := <-ch: ... }
| TheDong wrote:
| Even if you use channels to send things between goroutines, go
| makes it very hard to do so safely because it doesn't have the
| idea of sendable types, ownership, read-only references, and so
| on.
|
| For example, is the following program safe, or does it race?
| func processData(lines <-chan []byte) { for line :=
| range lines { fmt.Printf("processing line: %v\n",
| line) } } func main() {
| lines := make(chan []byte) go processData(lines)
| var buf bytes.Buffer for range 3 {
| buf.WriteString("mock data, assume this got read into the
| buffer from a file or something") lines <-
| buf.Bytes() buf.Reset() } }
|
| The answer is of course that it's a data race. Why?
|
| Because `buf.Bytes()` returns the underlying memory, and then
| `Reset` lets you re-use the same backing memory, and so
| "processData" and "main" are both writing to the same data at
| the same time.
|
| In rust, this would not compile because it is two mutable
| references to the same data, you'd either have to send
| ownership across the channel, or send a copy.
|
| In go, it's confusing. If you use
| `bytes.Buffer.ReadBytes("\n")` you get a copy back, so you can
| send it. Same for `bytes.Buffer.String()`.
|
| But if you use `bytes.Buffer.Bytes()` you get something you
| can't pass across a channel safely, unless you also never use
| that bytes.Buffer again.
|
| Channels in rust solve this problem because rust understands
| "sending" and ownership. Go does not have those things, and so
| they just give you a new tool to shoot yourself in the foot
| that is slower than mutexes, and based on my experience with
| new gophers, also more difficult to use correctly.
| hu3 wrote:
| That code would never pass a human pull request review. It
| doesn't even pass AI code review with a simple "review this
| code" prompt: https://chatgpt.com/share/68829f14-c004-8001-ac
| 20-4dc1796c76...
|
| "2. Shared buffer causes race/data reuse You're writing to
| buf, getting buf.Bytes(), and sending it to the channel. But
| buf.Bytes() returns a slice backed by the same memory, which
| you then Reset(). This causes line in processData to read the
| reset or reused buffer."
|
| I mean, you're basically passing a pointer to another thread
| to processData() and then promptly trying to do stuff with
| the same pointer.
| tptacek wrote:
| This is a canard.
|
| What's happening here, as happens so often in other situations,
| is that a term of art was created to describe something
| complicated; in this case, "memory safety", to describe the
| property of programming languages that don't admit to memory
| corruption vulnerabilities, such as stack and heap overflows,
| use-after-frees, and type confusions. Later, people uninvolved
| with the popularization of the term took the term and tried to
| define it from first principles, arriving at a place different
| than the term of art. We saw the same thing happen with "zero
| trust networking".
|
| The fact is that Go doesn't admit memory corruption
| vulnerabilities, and the way you know that is the fact that there
| are practically zero exploits for memory corruption
| vulnerabilities targeting pure Go programs, despite the
| popularity of the language.
|
| Another way to reach the same conclusion is to note that this
| post's argument proves far too much; by the definition used by
| this author, most other higher-level languages (the author
| exempts Java, but really only Java) _also_ fail to be memory
| safe.
|
| Is Rust "safer" in some senses than Go? Almost certainly. Pure
| functional languages are safer still. "Safety" as a general
| concept in programming languages is a spectrum. But "memory
| safety" isn't; it's a threshold test. If you want to claim that a
| language is memory-unsafe, POC || GTFO.
| kllrnohj wrote:
| > in this case, "memory safety", to describe the property of
| programming languages that don't admit to memory corruption
| vulnerabilities, such as [..] type confusions
|
| > The fact is that Go doesn't admit memory corruption
| vulnerabilities
|
| Except it does. This is exactly the example in the article.
| Type confusion causes it to treat an integer as a pointer &
| deference it. This then trivially can result in memory
| corruption depending on the value of the integer. In the
| example the value "42" is used so that it crashes with a nice
| segfault thanks to lower-page guarding, but that's just for
| ease of demonstration. There's nothing magical about the choice
| of 42 - it could just as easily have been any number in the
| valid address space.
| dboreham wrote:
| Everyone knows that there's something very magical about the
| choice of 42.
| Sharlin wrote:
| > to describe the property of programming languages that don't
| admit to memory corruption vulnerabilities, such as stack and
| heap overflows, use-after-frees, and type confusions.
|
| And data races allow all of that. There cannot be memory-safe
| languages supporting multi-threading that admit data races that
| lead to UB. If Go does admit data races it is not memory-safe.
| If a program can end up in a state that the language
| specification does not recognize (such as termination by
| SIGSEGV), it's not memory safe. This is the only reasonable
| definition of memory safety.
| tptacek wrote:
| If that were the case, you'd be able to support the argument
| with evidence.
| chowells wrote:
| You mean like the program in the article where code that
| never dereferences a non-pointer causes the runtime to
| dereference a non-pointer? That seems like evidence to me.
| tptacek wrote:
| An exploit against a real Go program that relies on
| memory corruption.
| afdbcreid wrote:
| It should be _possible_ to construct an exploit for such
| programs. But even for truly unsafe languages,
| vulnerabilities just from data races are very rare,
| because they are much harder to exploit.
|
| You could argue Go is safe from memory vulnerabilities,
| and that'll be 99% correct (we can't know what will
| happen if some very strong organization (e.g. a nation-
| state actor) will heavily invest in exploiting some Go
| program), but it still isn't memory safe, as per the
| definition in Wikipedia:
|
| > Memory safety is the state of being protected from
| various software bugs and security vulnerabilities when
| dealing with memory access, such as buffer overflows and
| dangling pointers.
| tptacek wrote:
| There's enormous incentive to construct those exploits.
| Why don't they exist?
| qualeed wrote:
| This seems to be operating on the premise that everyone
| knows of every exploit that exists. There's also an
| enormous incentive to hide working exploits.
| tptacek wrote:
| _Any_. Not _every_.
| qcnguy wrote:
| Go programs are normally running on the server side and
| are often proprietary, so you can't see the bugs exist to
| exploit them. It's not something like Chrome where
| someone can spend weeks finding a race and exploiting it
| to get a bug bounty, with full visibility of the source
| code and ability to develop an exploit in the lab.
| afdbcreid wrote:
| I'm not sure that's correct.
|
| Yes, this is an enormous effort to construct exploits,
| but constructing exploits for C/C++ code is much much
| easier and gives not less, or even more, benefit.
| Therefore it makes sense the efforts are focused on that.
|
| If/when most C/C++ code in the world will be gone, I
| assume we'll see more exploits of Go code.
| lossolo wrote:
| I can show you a trivial POC in C/C++ where someone opens
| a socket and ends up with a buffer overflow or UAF, both
| cases leading to memory corruption due to sloppy
| programming, and both easily exploitable for RCE.
|
| Can you show me any reasonable proof of concept (without
| using unsafe etc.) in Go that leads to similar memory
| corruption and is exploitable for RCE?
| ameliaquining wrote:
| https://blog.stalkr.net/2022/01/universal-go-exploit-
| using-d...
|
| This example hardcodes the payload, but (unless I've
| badly misunderstood how the exploit works) that's not
| necessary, it could instead be input from the network
| (and you wouldn't have to pass that input to any APIs
| that are marked unsafe). The payload is just hardcoded so
| that the example could be reproduced on the public Go
| Playground, which sandboxes the code it runs and so can't
| accept network input.
|
| Note that what tptacek is asking for is more stringent
| than this; he wants a proof-of-concept exploitation of a
| memory safety vulnerability caused by the data-race
| loopholes in the Go memory model, _in a real program that
| someone is running in production_. I do think it 's
| interesting that nobody has demonstrated that yet, but
| I'm not sure what it tells us about how sure we can be
| that those vulnerabilities don't exist.
| lossolo wrote:
| Yeah, it looks like CTF like POC, not what I would call
| reasonable code by any measure:
|
| https://github.com/StalkR/misc/blob/master/go/gomium/expl
| oit...
|
| The tight goroutine loop that flips one variable between
| two different struct types just to win a race is not
| something a typical developer writes on purpose. The
| trick to "defeat" compiler optimizations by assigning to
| a dummy variable inside an inline function. Carefully
| computing the address difference between two slices to
| reach out of bounds, then using that to corrupt another
| slice's header. I mean calling mprotect and jumping to
| shellcode is outright exploit engineering, not business
| logic and it's not part of the attackers payload.
|
| Chances of exact PoC pattern showing up in the wild by
| accident is basically zero.
| comex wrote:
| You're a cryptography person. So you know that most
| theoretically interesting cryptography vulnerabilities,
| even the ones that are exploitable in PoCs, are too
| obscure and/or difficult to get used by actual attackers.
| Same goes for hardware vulnerabilities. Rowhammer and
| speculative execution attacks are often shown to be able
| to corrupt and leak memory, respectively, but AFAIK there
| are no famous cases of them actually being used to attack
| someone. Partly because they're fiddly; partly out of
| habit. Partly because if you're in a position to satisfy
| the requirements for those attacks - you have copies of
| all the relevant binaries so you know the memory layout
| of code and data, you have some kind of sandboxed
| arbitrary code execution to launch the attack from - then
| you're often able to find better vulnerabilities
| elsewhere. And the same is also true for certain types of
| software vulnerabilities...
|
| Honestly, forget about Go: when was the last time you
| heard of a modern application backend being exploited
| through memory corruption, in any language? I know that
| Google and Meta and the like use a good amount of C++ on
| the server, as do many smaller companies. That C++ code
| may skew 'modern' and safer, but you could say the same
| about newly-developed client-side C++ code that's
| constantly getting exploited. So where are the server-
| side attacks? Part of the answer is probably that they
| exist, but I don't know about them because they haven't
| been disclosed. Unlike client-side attacks, server-side
| attacks usually target a single entity who has little
| incentive to publish deep dives into how they were
| attacked. That especially applies to larger companies,
| which tend to use more C++. But we do sometimes see those
| deep dives written anyway, and the vulnerabilities
| described usually aren't memory safety related. So I
| think there is also a gap in actual exploitation. Which
| probably has a number of causes, but I'd guess they
| include attackers (1) usually not having ready access to
| binaries, (2) not having an equivalent to the browser as
| a powerful launching point for exploits, and (3) not
| having access to _as much_ memory-unsafe code as on the
| client side.
|
| This is relevant to Go because of course Go is usually
| used on the server side. There is some use of Go on the
| client side, but I can't think offhand of a single
| example of it being used in the type of consumer OS or
| client-side application that typically gets attacked.
|
| Meanwhile, Go is of course much safer than C++. To make
| exploitation possible in Go, not only do you need a race
| condition (which are rarely targeted by exploits in any
| language), you also need a very specific code pattern.
| I'm not sure exactly how specific. I know how a
| stereotypical example of an interface pointer/viable
| mismatch works. But are there other options? I hear that
| maps are also thread-unsafe in general? I'd need to dig
| into the implementation to see how likely that is to be
| exploitable.
|
| Regardless, the potential exists. If memory safety is a
| "threshold test" as you say, then Go is not memory-safe.
|
| I agree though that the point would best be proven with a
| PoC of exploiting a real Go program. As someone with
| experience writing exploits, I think I could probably
| locate a vulnerability and create an exploit, if I had a
| few months to work on it. But for now I have employment
| and my free time is taken up by other things.
| amluto wrote:
| > when was the last time you heard of a modern
| application backend being exploited through memory
| corruption, in any language?
|
| It happens all the time, but it's a bit hard to find
| because "modern application backend[s]" are usually
| written in Go or Python or Rust. Even so, you'll find
| plenty of exploits based on getting a C or C++ library on
| the backend to parse a malformed file.
| comex wrote:
| Are these exploits publicly documented?
| tptacek wrote:
| There's lots of clientside Go, too!
| comex wrote:
| Where? Within, as I said, "the type of consumer OS or
| client-side application that typically gets attacked". It
| has to be a component of either a big application or a
| big OS, or something with comparable scope. Otherwise it
| would not likely be targeted by real-world memory
| corruption attacks (that we hear about) no matter the
| language. At least that's my impression.
| tptacek wrote:
| I'm sure I could come up with a bunch of examples but the
| first thing that jumps into my head is the Docker
| ecosystem.
| amluto wrote:
| https://github.com/golang/go/issues/34902
|
| https://www.cloudfoundry.org/blog/cve-2020-15586/
|
| I don't see any evidence that anyone wrote an RCE exploit
| for this, but I also don't see any evidence of anyone
| even trying to rule it out.
| tptacek wrote:
| What about this particular bug do you think makes it
| likely to be exploitable? I'm not asking you to write an
| RCE POC, just to tell a story of the sequence of events
| involving this bug that results in attacker-controlled
| code. What does the attacker control here, and how do
| they use that control to divert execution?
| amluto wrote:
| As a general heuristic, a corrupted data structure in a
| network server results in RCE. This is common in
| languages like C and C++.
|
| On first glance, it looks like the bug can (at least)
| result in the server accessing a slice object where the
| various fields don't all come from the same place. So the
| target server can end up accessing some object out of
| bounds (or as the wrong type or both), which can easily
| end up writing some data (possibly attacker controlled)
| to an inappropriate place. In standard attack, the
| attacker might try to modify the stack or a function
| pointer to set up a ROP chain or something similar, which
| is close enough to arbitrarily code to eventually either
| corrupt something to directly escalate privileges or to
| do appropriate syscalls to actually execute code.
| tptacek wrote:
| No, that doesn't work. Lots of (maybe even most)
| corrupted data structures aren't exploitable (past DOS).
| _Where does the attacker-controlled data come from_.
| _What path does it take to get to where the attacker
| wants it to go_. You have to be able to answer those two
| questions.
| amluto wrote:
| The Internet is full of nice articles of people bragging
| about their RCE exploits that start with single-byte
| overruns or seemingly-weak type confusions, etc.
|
| > Where does the attacker-controlled data come from.
|
| The example I gave was an HTTP server. Attackers can
| shove in as much attacker-controlled data as they want.
| They can likely do something like a heap by using many
| requests or many headers. Unless the runtime zeroes freed
| memory (and frees it immediately, which GC languages like
| Go often don't do), then lots of attacker controlled data
| will stick around. And, for all I know, the slice that
| gets mixed up in this bug is fully attacker controlled!
|
| In any event, I think this whole line of reasoning is
| backwards. Developers should assume that a memory safety
| error is game over unless there is a very strong reason
| to believe otherwise -- assume full RCE, ability to read
| and write all in-process data, the ability to issue any
| syscall, and the ability to try to exploit side channels.
| _Maybe_ very strong mitigations like hardware-assisted
| CFI will change this, and maybe not.
| gf000 wrote:
| Hide the same program into some dependency of a
| dependency and you have a nice little security
| vulnerability in your prod app. It's actually very easy
| to hide such a vulnerability as an innocent bug.
| ameliaquining wrote:
| If you're stipulating deliberately inserted
| vulnerabilities then there are much easier ways, e.g.,
| with a plausibly-deniable logic bug in code that calls
| os/exec or reflect (both of which can execute arbitrary
| code by design).
| gf000 wrote:
| If you see `exec`, that's an obvious point where you want
| to pay extra attention.
|
| Compare to an innocent looking map operation, and it's
| not even in the same league.
| Sharlin wrote:
| That's called "moving the goal posts".
|
| A definition of memory safety that permits unsoundness as
| long as nobody has exploited said unsoundness is not a
| definition that anyone serious about security is going to
| accept. Unsoundness is unsoundness, undefined behavior is
| undefined behavior. The conservative stance is that once
| execution hits UB, anything can happen.
| tialaramex wrote:
| It's just a little airborne, it's still good
|
| https://www.youtube.com/watch?v=1XIcS63jA3w
| sophacles wrote:
| I think your security background is coloring your
| perception of the term memory safety. Specifically the
| requirement that the various issues lead to exploitation.
| These issues can lead to many other issues that are not
| vulnerability in the security sense, e.g. data
| corruption, incorrect (but not insecure) behavior,
| performance issues, and more. I don't think any of those
| were ever dismissed or excluded from memory safety
| discussion. Infosec circles tend to evaluate most ideas
| in the context of (anti)exploitation, and the rest of
| programming tends to focus on what the cool kids argue
| (that is they often weigh security concerns higher than
| other issues as well), so the other problems caused by
| double-free or buffer overruns (etc) just may not have
| been given as much weight in your mind.
| tptacek wrote:
| "Memory safety" is a security term, not a PLT term.
| nemothekid wrote:
| This is no true Scotsman for programming languages.
|
| I could also argue C is memory safe and all the exploits
| that have been made weren't real C programs
| weinzierl wrote:
| _" What's happening here, as happens so often in other
| situations, is that a term of art was created to describe
| something complicated; [..] Later, people uninvolved with the
| popularization of the term took the term and tried to define it
| from first principles, arriving at a place different than the
| term of art."_
|
| Happens all the time in math and physics but having centuries
| of experience with this issue we usually just slap the name of
| a person on the name of the concept. That is why we have
| Gaussian Curvature and Riemann Integrals. Maybe we should speak
| of Jung Memory Safety too.
|
| Thinking about it, the opposite also happens. In the early 19th
| century _" group"_ had a specific meaning, today it has a much
| broader meaning with the original meaning preserved under the
| term _" Galois Group"_.
|
| Or even simpler: For the longest time seconds were defined as
| fraction of a day and varied in length. Now we have a precise
| and constant definition and still call them seconds and not ISO
| seconds.
| lenkite wrote:
| How does Java "fail" to be memory safe by the definition used
| by the author ? Please give an example.
| johnnyjeans wrote:
| This is a good post and I agree with it in full, but I just
| wanted to point out that (safe) Rust is safer from data races
| than, say, Haskell due to the properties of an affine type
| system.
|
| Haskell in general is a much safer than Rust thanks to its more
| robust type system (which also forms the basis of its
| metaprogramming facilities), monads being much louder than
| unsafe blocks, etc. But data races and deadlocks are one of the
| few things Rust has over it. There are some pure functional
| languages that are dependently typed like Idris, and thus far
| safer than Rust, but they're in the minority and I've yet to
| find anybody using them industrially. Also Fortnite's Verse
| thing? I don't know how pure that language is though.
| chowells wrote:
| I don't think it's true that Rust is safer, using the
| terminology from the article. Both languages prevent you from
| doing things that will result in safety violations unless you
| start mucking with unsafe internals.
|
| Rust absolutely does make it easier to write high-performance
| threaded code correctly, though. If your system depends on
| high amounts of concurrent mutation, Rust definitely makes it
| easier to write correct code.
|
| On the other hand, a system like STM in Haskell can make it
| easier to write complex concurrency logic correctly in
| Haskell than Rust, but it can have very bad performance
| overhead and needs to be treated with extreme suspicion in
| performance-sensitive code. It's a huge win for simple
| expression of complex concurrency, but you have to pay for it
| somewhere. It can be used in ways where that overhead is
| acceptable, but you absolutely need to be suspicious in a way
| that's never a concern in Rust.
| empath75 wrote:
| > Another way to reach the same conclusion is to note that this
| post's argument proves far too much; by the definition used by
| this author, most other higher-level languages (the author
| exempts Java, but really only Java) also fail to be memory
| safe.
|
| Yes I mean that was the whole reason they invented rust. If
| there were a bunch of performant memory safe languages already
| they wouldn't have needed to.
| jstarks wrote:
| > If you want to claim that a language is memory-unsafe, POC ||
| GTFO.
|
| There's a POC right in the post, demonstrating type confusion
| due to a torn read of a fat pointer. I think it could have just
| as easily been an out-of-bounds write via a torn read of a
| slice. I don't see how you can seriously call this memory safe,
| even by a conservative definition.
|
| Did you mean POC against a real program? Is that your bar?
| tptacek wrote:
| You need a non-contrived example of a memory-corrupting data
| race that gives attackers the ability to control memory,
| through type confusion or a memory lifecycle bug or something
| like it. You don't have to write the exploit but you have to
| be able to tell the story of how the exploit would actually
| work --- "I ran this code and it segfaulted" is not enough.
| It isn't even enough for C code!
| danbruc wrote:
| Nope. You can have programs without undefined behavior and still
| not have thread safety. In .NET, for example, writes to variables
| that are wider then the machine width or not aligned properly,
| are not guaranteed to be atomic. So if you assign some value to
| an Int128 variable, it will not be updated atomically - how could
| it, that is just beyond the capabilities of the processor - and
| therefore a different thread can observe a state where only half
| of the variable has been updated. No undefined behavior here but
| also sharing this variable between threads is not thread safe.
| And having the language synchronize all such writes - just in
| case some other thread might want tot look at it - is a
| performance disaster. And disallowing anything that might be a
| potential thread safety issue will give you a pretty limited
| language.
| tialaramex wrote:
| > disallowing anything that might be a potential thread safety
| issue will give you a pretty limited language.
|
| Safe Rust doesn't seem that limited to me.
|
| I don't think _any_ of the C# work I do wouldn 't be possible
| in Rust, if we disregard the fact that the rest of the team
| don't know Rust.
|
| Most of the programs you eliminate when you have these
| "onerous" requirements like memory safety are nonsense, they
| either sometimes didn't work or had weird bugs that would be
| difficult to understand and fix - sometimes they also had scary
| security implications like remote code execution. We're better
| off without them IMNSHO.
| kibwen wrote:
| The statement "there is no memory safety without thread safety"
| does not suggest that memory safety is sufficient to provide
| thread safety. Instead, it's just saying that if you want
| thread safety, then memory safety is a requirement.
| minitech wrote:
| > Instead, it's just saying that if you want thread safety,
| then memory safety is a requirement.
|
| It's saying the opposite - that if you want memory safety,
| thread safety is a requirement - and Java and C# refute it.
| zozbot234 wrote:
| > Java and C# refute it.
|
| No, they don't. They're using a different meaning for
| "thread safety" that's more useful in context since they do
| ensure data race safety - which is the _only_ kind of
| thread safety OP is talking about. By guaranteeing data
| race safety as a language property, Java and C# are proving
| OP 's point, not refuting it.
| kibwen wrote:
| _> It 's saying the opposite_
|
| Indeed, you're correct, I interpreted the implications in
| reverse.
| kllrnohj wrote:
| Critically to the authors point that type of data race does not
| result in UB and does not break the language and thus does not
| create any memory safety issues. Ergo, it's a memory safe
| language.
|
| Go (and previously Swift) fails at this. There data races _can_
| result in UB and thus break memory safety
| ameliaquining wrote:
| See the article's comments on Java, which is "thread safe" in
| the sense of preventing undefined behavior but not in the sense
| of preventing data-race-related logic bugs. .NET is precisely
| analogous in this respect.
| tialaramex wrote:
| I can buy that claim for the .NET CLR but I've never seen it
| nailed down properly the way Java did which gives me pause.
|
| I worry about the Win95-era "Microsoft Pragmatism" at work
| and a concrete example which comes to mind is nullability. In
| the nice modern software I often work on I can say some
| function takes a string and in that program C# will tell me
| that's not allowed to be null, it has to be an actual string
| - a significant engineering benefit. But, the CLR does not
| enforce such rules, so that function may still receive a null
| instead e.g. if called by some ten year old VB.NET code which
| has no idea about "nullability" and so just fills out a null
| for that parameter anyway.
|
| Of course the CLR memory model might really be set in stone
| and 100% proof against such problems, but I haven't seen
| anything to reassure me as I did for Java and I fear that if
| it were convenient for Windows to not quite do that work they
| would say eh, good enough.
| ameliaquining wrote:
| There's a documented memory model (https://github.com/dotne
| t/runtime/blob/main/docs/design/spec...), does that not
| address this concern?
| actionfromafar wrote:
| It depends on what it says?
| kazinator wrote:
| This is false as a generality.
|
| A memory safe, managed language doesn't become unsafe just
| because you have a race condition in a program.
|
| Like, say, reading and writing several related shared variables
| without a mutex.
|
| Say that the language ensures that the reads and writes
| themselves of these word-sized variables are safe without any
| lock, and that memory operations and reclamation of memory are
| thread safe: there are no low-level pointers (or else only as an
| escape hatch that the program isn't using).
|
| The rest is your bug; the variable values coming out of sync with
| each other, not maintaining the invariant among their values.
|
| It could be the case that a thread-unsafe program breaks a
| managed run-time, but not an unvarnished truth.
|
| A managed run-time could be built on the assumption that the
| program will not create two or more threads such that those
| threads will invoke concurrent operations on the same objects.
| E.g. a managed run time that needs a global interpreter lock, but
| which is missing.
| qcnguy wrote:
| The author knows that. His point is that Go doesn't work that
| way because it uses greater-than-word-sized values that can
| suffer torn writes leading to segfaults in some cases.
| munificent wrote:
| _> A memory safe, managed language doesn 't become unsafe just
| because you have a race condition in a program._
|
| The author's point is that Go is not a memory safe language
| according to that distinction.
|
| There are values that are a single "atomic" write in the
| language semantics (interface references, slices) that are
| implemented with multiple non-atomic writes in the
| compiler/runtime. The result is that you can observe a torn
| write and break the language's semantics.
| dodobirdlord wrote:
| If the variables are word-sized, sure. But what if they are
| larger? Now a race condition between one thread writing and
| another thread reading or writing a variable is a memory safety
| issue.
| kazinator wrote:
| Don't have such things, if you know what's good for you, or
| else don't have threads.
| zozbot234 wrote:
| > Now a race condition between one thread writing and another
| thread reading or writing a variable is a memory safety
| issue.
|
| No it isn't, because the torn write cannot have _arbitrary_
| effects that potentially break the program. It only becomes
| such if you rely on such a variable to establish an invariant
| about memory that 's broken if a torn write occurs (such as
| by encoding a ptr+len in it), which is just silly. Don't do
| that!
| gpderetta wrote:
| > which is just silly. Don't do that!
|
| tell that to the Go runtime, which relies on slices always
| being valid and not being able to create invalid ones.
| gpderetta wrote:
| race condition != data race. Specifically, in go, a race
| condition can cause application level bugs but won't affect,
| directly, the runtime consistency; on the other hand a data
| race on a slice can cause torn writes and segfaults in the best
| case, and fandango on core in the worst case.
| qcnguy wrote:
| The point being made is sound, but I can never escape the feeling
| that most concurrency discussion in programming language theory
| is ignoring the elephant in the room. The concurrency bugs that
| matter in most apps are all happening inside the database due to
| lack of proper locking, transactions or transactional isolation.
| PL theory ignores this and so things like Rust's approach to race
| freedom ends up not mattering much outside of places like
| kernels. A Rust app can avoid use of unsafe entirely and still be
| riddled with race conditions because all the data that matters is
| in an RDBMS and someone forgot a FOR UPDATE in their SELECT
| clause.
| munificent wrote:
| I agree with the author's claim that you need thread safety for
| memory safety.
|
| But I don't agree with:
|
| _> I will argue that this distinction isn't all that useful, and
| that the actual property we want our programs to have is absence
| of Undefined Behavior._
|
| There is plenty of undefined behavior that can't lead to
| violating memory safety. For example, in many languages, argument
| evaluation order is undefined. If you have some code like:
| foo(print(1), print(2));
|
| In some languages, it's undefined as to whether "1" is printed
| before "2" or vice versa. But there's no way to violate memory
| safety with this.
|
| I think the only term the author needs here is "memory safety",
| and they correctly observe that if the language has threading,
| then you need a memory model that ensures that threads can't
| break your memory safety.
|
| Go lacks that. It seems to be a rare problem in practice, but if
| you want guarantees, Go doesn't give you them. In return, I guess
| it gives you slightly faster execution speed for writes that it
| allows to potentially be torn.
| zozbot234 wrote:
| That's "unspecified" not "undefined". "Undefined behavior"
| literally means "anything goes", so any program that invokes it
| is broken by definition.
| bigstrat2003 wrote:
| That is not true, that is a very specific definition of UB
| which C developers (among others) favor. That doesn't mean
| that another language can't say "this is undefined behavior"
| without all the baggage that accompanies the term in C.
| zozbot234 wrote:
| It's literally how the term "UB" is defined, and understood
| by experts. Why would anyone want to say "undefined" when
| they really mean "unspecified"? That's just confusing.
| bigstrat2003 wrote:
| No, it's how one _very specific_ community of experts
| understands it. It is not some kind of universal law of
| definition that it must mean that always and everywhere.
| As far as what is confusing, that is a matter of
| perspective. I think it is confusing (to put it mildly)
| that the C community has chosen to use "undefined
| behavior" to mean "it must never happen, and anything
| goes if it does". That is _extremely_ counterintuitive,
| and only makes sense to those who live and breathe that
| world. So if the standard is to be "avoiding confusion",
| then we better change the definition used by the C
| community ASAP.
| ameliaquining wrote:
| I agree that the term "undefined behavior", when used as
| in C/C++/Rust/Swift/.NET, isn't very good at
| communicating to non-experts what's at stake, not least
| because it doesn't sound scary enough (the security
| community remains indebted to whoever coined the term
| "nasal demons"). That said, is there a specific other
| community of practice where there's a shared
| understanding that the term "undefined behavior" means
| something different?
| uecker wrote:
| It is also not what the C community has chosen. It is
| what was imposed on us by certain optimizing compilers
| that used the interpretation that gave them maximum
| freedom to excel in benchmarks, and it was then endorsed
| by C++. The C definition is that "undefined behavior" can
| have arbitrary concrete behavior, not that a compiler can
| assume it does not happen. (that form semantic people
| prefer the former because it makes their life easier did
| not help)
| bakugo wrote:
| "Undefined behavior" is not a meaningless made up term that
| you can redefine at will.
|
| The word "undefined" has a clear meaning: there is no
| behavior defined at all for what a given piece of code will
| do, meaning it can literally do anything. If the language
| spec defines the possible behaviors you can expect (even if
| the behavior can vary between implementations), then by
| definition it's not undefined.
| bigstrat2003 wrote:
| > "Undefined behavior" is not a meaningless made up term
| that you can redefine at will.
|
| Sure, I agree with that.
|
| > The word "undefined" has a clear meaning: there is no
| behavior defined at all for what a given piece of code
| will do...
|
| That is true, but...
|
| > ...meaning it can literally do anything.
|
| This is _not at all_ true! That is a different (but
| closely related) matter, which is "what is to be done
| about undefined behavior". Which is certainly something
| one has to take a stance on when working to a language
| spec that has undefined behavior, but that does _not_
| mean that "undefined" automatically means your preferred
| interpretation of how to handle undefined behavior.
| zozbot234 wrote:
| The original question is how UB is _defined_ , not about
| the preferred way of dealing with it in a practical
| sense. And the definition of UB is behavior for which the
| language definition imposes no requirements, and
| explicitly leaves open the possibility of ignoring the
| situation altogether with unpredictable results.
| gliptic wrote:
| The author is using the term in the way that everyone else
| understands it. They are not aware of your unusual
| definition.
| joaohaas wrote:
| Your example does not classify as 'undefined behavior'.
| Something is 'undefined behavior' if it is specified in the
| language spec, and in such case yes, the language is capable of
| doing anything including violating memory safety.
| gliptic wrote:
| The evaluation order is _unspecified_, not undefined behaviour.
| gpderetta wrote:
| Interestingly, at least in C++, this was changed in the
| recent past. It used to be that evaluation of arguments was
| not sequenced at all and if any evaluation touched the same
| variable, and at least one was a write, it was UB.
|
| It was changed as part of the C++11 memory model and now, as
| you said, there is a sequenced-before order, it is just
| unspecified which one it is.
|
| I don't know much about C, but I believe it was similarly
| changed in C11.
| gliptic wrote:
| Yes, but that's just a subset of expressions where
| unspecified sequencing applied. For instance, the example
| with two `print()` as parameters would have a sequence
| point (in pre-C++11 terminology) separating any
| reads/writes inside the `print` due to the function calls.
| It would never be UB even though the order in which the
| prints are called is still unspecified.
| gpderetta wrote:
| IIRC the point was that there was no sequence point
| between argument evaluation, so for example f(++i, ++i)
| was UB. Or maybe it was only for builtin operators?
|
| Cppreference is not authoritative[1], but seems to
| support my recollection. In fact it states that the
| f(++i, ++i) was UB till C++17.
|
| [1] https://en.cppreference.com/w/cpp/language/eval_order
| .html, Pre C++11 Ordering Rules, point (2).
| gliptic wrote:
| `f(++i, ++i)` is/was indeed UB, but the example in
| munificent's comment was `foo(print(1), print(2))` which
| as far as I know is not even if both `print` calls
| read/write the same memory.
| gpderetta wrote:
| (5) in the paragraph I mentioned earlier seems to prevent
| interleaving of function calls, which admittedly would
| make the language hard to use. So I think you are right.
| tialaramex wrote:
| Sure, prior to the C++ 11 memory model there just isn't a
| memory ordering model in C++ and all programs in either C
| or C++ which would need ordering for correctness did not
| have any defined behaviour in the language standard.
|
| This is very amusing because that means _in terms of the
| language standard_ Windows and Linux, which both
| significantly pre-date C++ 11 and thus its memory model,
| were technically relying on Undefined Behaviour. Of course,
| as operating systems they 're already off piste because
| they're full of raw assembly and so on.
|
| Linux has its own ordering model as a result, pre-dating
| the C++ 11 model. Linus is writing software for multi-
| processor computers more than a _decade_ before the C++ 11
| model so obviously he can 't wait around for that.
|
| [Edit: Corrected Linux -> Linux when talking about the man]
| gpderetta wrote:
| It is not so much that windows and linux were relying on
| UB, but that these platforms, with their compilers,
| provided guarantees beyond the standard. e.g. GCC not
| only aims for C/C++ standard compliance, but also POSIX.
|
| Of course these guarantees were often not fully written
| down nor necessarily self consistent (but then again,
| neither is the current standard).
| nromiun wrote:
| I bet not even 5% of all programs are multi-threaded, or even
| concurrent.
|
| Memory safety is a much bigger problem.
| norir wrote:
| The sad thing is that most languages with threads have a default
| of global variables and unrestricted shared memory access. This
| is the source of the vast majority of data corruption and races.
| Processes are generally a better concurrency model than threads,
| but they are unfortunately too heavyweight for many use cases. If
| we defaulted to message passing all required data to each thread
| (either by always copying or tracking ownership to elide
| unnecessary copying), most of these kinds of problems would go
| away.
|
| In the meantime, we thankfully have agency and are free to choose
| not to use global variables and shared memory even if the
| platform offers them to us.
| zozbot234 wrote:
| Message passing can easily lead to more logical errors (such as
| race conditions and/or deadlocks) than sharing memory directly
| with properly synchronized access. It's not a silver bullet.
| umpalumpaaa wrote:
| 100%.
|
| Some more modern languages - eg. Swift - have "sendable"
| value types that are inherently thread safe. In my experience
| some developers tend to equate "sendable" / thread safe data
| structures with a silver bullet. But you still have to think
| about what you do in a broader sense... You still have to
| assemble your thread safe data structures in a way that makes
| sense, you have to identify what "transactions" you have in
| your mental model and you still have to think about data
| consistency.
| kibwen wrote:
| _> The sad thing is that most languages with threads have a
| default of global variables and unrestricted shared memory
| access. This is the source of the vast majority of data
| corruption and races. Processes are generally a better
| concurrency model than threads_
|
| Modern languages have the option of representing thread-safety
| in the type system, e.g. what Rust does, where working with
| threads is a dream (especially when you get to use structured
| concurrency via thread::scope).
|
| People tend to forget that Rust's original goal was not "let's
| make a memory-safe systems language", it was "let's make a
| thread-safe systems language", and memory safety just came
| along for the ride.
| tialaramex wrote:
| _Originally_ Rust is something altogether different. Graydon
| has written about that extensively. Graydon wanted tail
| calls, reflection, more "natural" arithmetic with Python
| style automatic big numbers, decimal for financial work and
| so on.
|
| The Rust we have from 1.0 onwards is not what Graydon wanted
| at all. Would Graydon's language have been broadly popular?
| Probably not, we'll never know.
| nine_k wrote:
| While at it, I suppose it's straightforward to implement
| arbitrary-precision integers and decimals in today's Rust;
| there are several crates for that. There's also a
| `tailcall` crate that apparently implements TCO [1].
|
| [1]: https://docs.rs/tailcall/latest/tailcall/
| tialaramex wrote:
| Oh, I do know you can have arbitrary precision. I'm the
| author of realistic, which isn't "just" arbitrary
| precision it's an approximation of the computable reals
| as well, which is sometimes just enough more power than
| you'd hardly notice you have arbitrary precision too.
|
| https://crates.io/crates/realistic
| kibwen wrote:
| Even in pre-1.0 Rust, concurrency was a primary goal;
| there's a reason that Graydon listed Newsqueak, Alef,
| Limbo, and Erlang in the long list of influences for proto-
| Rust.
| fmajid wrote:
| And yet it ignored the primary lesson of Erlang: no
| shared memory access whatsoever, which is what makes it
| so robust.
| littlestymaar wrote:
| Because the other teams members (IIRC brson and pcwalton)
| wanted Rust to be as performant as C++, which means you
| must have a way to have shared memory.
| chadaustin wrote:
| Every time this conversation comes up, I'm reminded of my team at
| Dropbox, where it was a rite of passage for new engineers to
| introduce a segfault in our Go server by not synchronizing writes
| to a data structure.
|
| Swift has (had?) the same issue and I had to write a program to
| illustrate that Swift is (was?) perfectly happy to segfault under
| shared access to data structures.
|
| Go has never been memory-safe (in the Rust and Java sense) and
| it's wild to me that it got branded as such.
| junebash wrote:
| Swift is in the process of fixing this, but it's a slow and
| painful transition; there's an awful lot of unsafe code in the
| wild that wasn't unsafe until recently.
| cosmic_cheese wrote:
| One of the biggest hurdles is just getting all the
| iOS/macOS/etc APIs up to speed with the thread safety
| improvements. It won't make refactoring all that application
| code any easier, but as things stand even if you've done
| that, you're going to run into problems anywhere your code
| makes contact with UI code because there's a lot of AppKit
| and UIKit that have yet to make the transition.
| RetpolineDrama wrote:
| Swift 6 is only painful if you wrote a ton of terrible Swift
| 5, and even then Swift 5 has had modes where you could
| gracefully adopt the Swift 6 safety mechanisms for a long
| time (years?)
|
| ~130k LoC Swift app was converted from 5 -> 6 for us in about
| 3 days.
| shadowgovt wrote:
| Mostly because it was a remarkable improvement over what came
| before (and what came before was hilariously fragile).
| pjmlp wrote:
| Only for those not paying attention outside mainstream, or
| too young to remember former languages.
| the_plus_one wrote:
| > or too young to remember former languages.
|
| Do you have any good examples? Not trying to argue, just
| genuinely curious as someone who hasn't been in this field
| for decades.
| LtWorf wrote:
| Basically go was designed ignoring all the research and
| progress that had been made in programming languages
| until then.
|
| It was designed with contempt for developers, for example
| disallowing developers to create generic data structures,
| or lacking a decent way of error checking that is not
| extremely error prone and verbose.
| shadowgovt wrote:
| I'm certainly not disagreeing, but I will note that by
| definition, most people are in the mainstream, so something
| being a remarkable improvement over what came before (in
| the mainstream) is a remarkable improvement (for most
| people).
| Mawr wrote:
| Safety isn't binary, so your comment makes no sense.
| kstrauser wrote:
| I'd argue that unsafety is binary. If a normal eng doing
| normal things can break it without going out of their way to
| deliberately fool the compiler or runtime, I'd call it
| unsafe.
| Calavar wrote:
| By that definition Rust also counts as unsafe. Even managed
| languages like C# and Java would be unsafe.
| kstrauser wrote:
| My impression of the Rust devs is that they'd agree with
| you about any easy-to-trigger calamities. So would Java
| contributors. C# might not because MS is institutionally
| not good about admitting mistakes, but I bet the
| individual devs would agree over a beer.
| dcminter wrote:
| What kinds of breakage do you have in mind though? The
| number of times I've segfaulted the JVM is tiny.
| ackfoobar wrote:
| Do you have some examples? I think JDK developers make a
| lot of effort to make sure users bugs will not corrupt
| the runtime.
| gpm wrote:
| There's a reason why rust devs qualify it as "memory
| safe" so frequently, we tend to agree that rust is, like
| virtually every current programming language, unsafe in
| other ways.
|
| Memory safety is just the source of bugs that we've
| figured out how to eliminate. It's a significant source
| of really bad (hard to debug due to action at a distance,
| high impact, etc) bugs so that's worth a lot, but it's
| not perfect. And even then we have a more frequently used
| escape hatch to the memory-unsafe world than would be
| ideal from a safety perspective for practical reasons.
|
| A more complete version of safety would be achieved with
| a language that proves code correct to arbitrary
| specifications. We aren't there yet for there being such
| a language that is practical for every day use.
| Personally I'm increasingly optimistic we'll get there
| sooner rather than later (say, within 20 years). Even
| then there will probably be specification level bugs that
| prevent a claim of complete safety...
| pjmlp wrote:
| It is kind of wild that for a 21st century programming
| language, the amount of stuff in Go that should have been but
| never was, but hey Docker and Kubernetes.
| 9rx wrote:
| On the flip side, what would be the point? There are already
| a million other languages that have everything and the
| kitchen sink.
|
| Not going down the same road is the only reason it didn't end
| up on the pile of obscure languages nobody uses.
| pjmlp wrote:
| The only reason it didn't end on pile of obscure languages
| nobody uses, it called Google, followed by luck with Docker
| and Kubernetes adoption on the market, after they decided
| to rewrite from Python and Java respectively into Go, after
| Go heads joined their teams.
|
| Case in point, Limbo and Oberon-2, the languages that
| influenced its design, and authors were involved with.
| 9rx wrote:
| _> The only reason it didn 't end on pile of obscure
| languages nobody uses, it called Google_
|
| Dart ended up on the pile of languages nobody uses. And
| Carbon? What's Carbon? Exactly!
|
| _> Case in point, Limbo and Oberon-2, the languages that
| influenced its design_
|
| Agreed. Limbo and Oberon-2, as primitive as they may look
| now, had the kitchen sinks of their time. Why wouldn't
| they have ended up on the pile of languages nobody uses?
| pjmlp wrote:
| People love to bring those as counter examples, without
| actually knowing a single fact about them.
|
| Dart was a victim of internal politics between the Chrome
| team, Dart team, AdWords moving away from GWT wanting
| AngularDart (see Angular documentary), and the Web in
| general.
|
| Had Chrome team kept pushing DartVM, it might have been
| quite different story.
|
| Carbon, good example of failure to actually know what the
| team purposes are. It is officially a research project
| for Google themselves, where the team is the first to
| advise using Rust or another MSL.
|
| One just needs to actually spend like a couple of minutes
| on their wiki, but I guess that is asking too much on
| modern times.
|
| Limbo and Oberon-2 were definitely not kitchen sinks of
| their time, their failure was that neither Bell Labs in
| 1996, nor ETHZ in 1992, were that relevant for the
| programming language community in the industry.
| 9rx wrote:
| _> Had Chrome team kept pushing DartVM, it might have
| been quite different story._
|
| Trouble with that line of thinking is that Google never
| pushed Go either. It didn't even bother to use it
| internally (outside from the occasional side project here
| and there). Google paid some salaries. I'll give you
| that. But it has paid salaries for a lot of different
| languages. That is not some kind of secret sauce.
|
| _> It is officially a research project for Google
| themselves_
|
| It's not just a research project. It is officially "not
| ready for use", but its roadmap has a clear "ready for
| use" plan in the coming months. Rust was also "not ready
| for use" when it hit the streets, it officially being a
| Mozilla research project, but every second discussion on
| HN was about it and what is to come. And that was without
| Google backing. If what you say is true, why isn't Carbon
| being shouted from every rooftop right now?
|
| I know you're struggling to grasp at straws here, but
| let's just be honest for a moment: If it hasn't caught
| attention already, it isn't going to. Just another
| language to add to the pile.
| Animats wrote:
| The strength of Go is not the language. It's that the
| libraries you need for web back-end stuff are written,
| maintained, and used in production by Google. All the
| obscure cases get exercised in production due to sheer
| volume of internal usage.
|
| At one time, Go maps were not thread-safe. Was that
| fixed?
| Yoric wrote:
| I'd be surprised if the JSON module was used within
| Google, though. It's neither particularly fast nor
| particularly convenient nor particularly suited to
| properly handle edge cases. But it's still in the stdlib
| for compatibility reasons.
| 9rx wrote:
| _> At one time, Go maps were not thread-safe. Was that
| fixed?_
|
| sync.Map was added, but isn't intended to be a general
| purpose map.
|
| ----
|
| _The Map type is specialized. Most code should use a
| plain Go map instead, with separate locking or
| coordination, for better type safety and to make it
| easier to maintain other invariants along with the map
| content.
|
| The Map type is optimized for two common use cases: (1)
| when the entry for a given key is only ever written once
| but read many times, as in caches that only grow, or (2)
| when multiple goroutines read, write, and overwrite
| entries for disjoint sets of keys. In these two cases,
| use of a Map may significantly reduce lock contention
| compared to a Go map paired with a separate Mutex or
| RWMutex. _
| pjmlp wrote:
| Exactly, by Google.
| Yoric wrote:
| Well, that and the slight fact that it bears Google's brand
| name.
|
| I personally appreciate Go as a research experiment. Plenty
| of very interesting ideas, just as, for instance, Haskell.
| I don't particularly like it as a development language, but
| I can understand why some people do.
| 9rx wrote:
| _> Plenty of very interesting ideas_
|
| Is there? When you get down to it, it is really just a
| faster Python. Which is exactly what it was said to be
| when it was released. Their goal was to create a
| "dynamically-typed" language that was more performant. It
| is likely that it wouldn't have had a static type system
| at all if they figured out how to achieve on the
| performance end without needing types.
|
| You can tell who is clueless when you hear someone say
| its type system is lacking. I mean, technically it is,
| but it is supposed to be. Like saying Javascript or
| Ruby's type system is lacking.
| Yoric wrote:
| Two examples of interesting ideas:
|
| - using zero values as an optimization mechanism;
|
| - (non-)pointers and passing self by copy.
|
| I mean, I hate both mechanisms, but intellectually, I
| find them quite interesting.
|
| Also, I'd not classify it as a faster Python. It's more
| of a cousin of Obj-C if the authors of Obj-C had fallen
| in love of Erlang instead of Smalltalk.
| pjmlp wrote:
| Faster Python with a very small set of its capabilities.
| stouset wrote:
| Having the weight of Google behind it is the primary reason
| it didn't end up on the pile of obscure languages nobody
| uses.
| 9rx wrote:
| Oh...? Dart never gained much steam. And let's not forget
| about Carbon! Can you name even just one person who has
| tried Carbon? Have more than a handful of people even
| heard of Carbon?
|
| I will grant you that Carbon is still in its infancy, but
| when Rust was in the same youthful stage we never heard
| an end to all the people playing with it. You, even if
| not tried it yourself, definitely knew about it.
|
| You've made up a fun idea, but reality doesn't support
| it. Google has not shown its weight carries anything.
| They have really struggled to get any for-profit business
| units off the ground since they gained the weight, never
| mind their hobbies! If anything, Google is detrimental to
| a project.
| pjmlp wrote:
| Already explained in another thread, learn the politics
| of Dart, and Carbon is still on the drawing board.
| 9rx wrote:
| Already...? Said "explanation" was posted over an hour
| after the comment replied to here.
|
| If only Google put their weight into a watch, maybe you'd
| have one?
|
| Oh wait. They did! Google can't successfully turn their
| weight into much of anything. Go's success, if we can
| call it that, clearly happened in spite of Google.
| geodel wrote:
| So failures are some deep valid reasons whereas success
| is developers don't know any better language.
| potato-peeler wrote:
| I am curious. Generally basic structures like map are not
| thread safe and care has to be taken while modifying it. This
| is pretty well documented in go spec. In your case in dropbox,
| what was essentially going on?
| maxlybbert wrote:
| I thought the same thing. Maybe the point of the story isn't
| "we were surprised to learn you had to synchronize access"
| but instead "we all thought we were careful, but each of us
| made this mistake no matter how careful we tried to be."
| adamwk wrote:
| Crashing on shared access is the safe thing to do
| mirashii wrote:
| An intentional exit by a runtime is a safe crash. A segfault
| is not, and is here a clear sign that memory safety has been
| violated.
| adamwk wrote:
| I guess I was thinking specifically of the swift case where
| values have exclusive access enforcement. Normally caught
| by a compiler, they will safely crash if the compiler
| didn't catch it. I think the only way to segfault would be
| by using Unsafe*Pointer types, which are explicitly marked
| unsafe
| Gibbon1 wrote:
| Yeah it's not the segfault that's bad, it's when it's when
| the write to address 0x20001854 succeeds and now some
| hapless postal clerk is going to jail.
| LtWorf wrote:
| a segfault is completely unintentional. Had the kernel been
| older it could be used to execute code.
| tptacek wrote:
| Right, the issue here is that the "Rust and Java sense" of
| memory safety is not the actual meaning of the term. People
| talk as if "memory safety" was a PLT axiom. It's not; it's a
| software security term of art.
|
| This is just two groups of people talking past each other.
|
| It's not as if Go programmers are unaware of the distinction
| you're talking about. It's literally the premise of the
| language; it's the basis for "share by communicating, don't
| communicate by sharing". Obviously, that didn't work out, and
| modern Go does a lot of sharing and needs a lot of
| synchronization. But: everybody understands that.
| Ygg2 wrote:
| > People talk as if "memory safety" was a PLT axiom. It's
| not; it's a software security term of art.
|
| It's been in usage for PLT for at least twenty years[1]. You
| are at least two decades late to the party.
| Software is memory-safe if (a) it never references a memory
| location outside the address space allocated by or that
| entity, and (b) it never executes intstruction outside code
| area created by the compiler and linker within that address
| space.
|
| [1]https://llvm.org/pubs/2003-05-05-LCTES03-CodeSafety.pdf
| shadowgovt wrote:
| This is, in my mind, the trickiest issue with Rust right now as a
| language project, to wit:
|
| - The above is true
|
| - If I'm writing something using a systems language, it's because
| I care about performance details that would include things like
| "I want to spawn and curate threads."
|
| - Relative to the borrow-checker, the Rust thread lifecycle
| static typing is _much_ more complicated. I think it is because
| it 's reflecting some real complexity in the underlying problem
| domain, but the problem stands that the description of resource
| allocation across threads can get very hairy very fast.
| chc4 wrote:
| This is one of the things that I'm also looking on at Zig like a
| slow moving car crash about: they claim they are memory safe (or
| at least "good enough" memory safe if you use the safe
| optimization level, which is it's own discussion), but they don't
| have the equivalent to Rust's Send/Sync types. It just so happens
| that in practice no one was writing enough concurrent Zig code to
| get bitten by it a lot, I guess...except that now they're working
| on bringing back first-class async support to the language, which
| will run futures on other threads and presumably a lot of feet
| are going to be fired at once that lands.
| ameliaquining wrote:
| IIUC even single-threaded Zig programs built with ReleaseSafe
| are not guaranteed to be free of memory corruption
| vulnerabilities; for example, dereferencing a pointer to a
| local variable that's no longer alive is undefined behavior in
| all optimization modes.
| Thaxll wrote:
| Go is memory safe by the most common definition, does not matter
| if you have segfault in some scenario.
|
| How many exploits or security issues have there been related to
| data race on dual word values? I work with Go for the last 10
| years and I never heard of such issues. Not a single time.
| zozbot234 wrote:
| The most common definition of memory safe is literally "cannot
| segfault" (unless invoking some explicitly unsafe operation -
| which is not the case here unless you think the "go" keyword
| should be unsafe).
| Thaxll wrote:
| I don't know the NSA with their white house paper about
| memory safe language mentioned Go, maybe you should tell that
| there are wrong.
| Sesse__ wrote:
| https://en.wikipedia.org/wiki/Argument_from_authority
| Thaxll wrote:
| So NSA does not have the relevant authority to qualify a
| language as memory safe, is it what you're saying?
|
| The document is backed by foreign government as well.
|
| https://media.defense.gov/2023/Dec/06/2003352724/-1/-1/0/
| THE...
| SkiFire13 wrote:
| TBH segfaults are not necessarily a sign of memory unsafety,
| but _unexpected_ segfaults are.
|
| For some examples, Rust (although this is not specific to it)
| uses stack guard pages to detect stack overflows by _forcing_
| a segfault (as opposed to reading/writing arbitrary memory
| after the usual stack). Some JVMs also expect and handle
| segfaults when dereferencing null pointers, to avoid always
| paying the cost for checking them.
| dylnuge wrote:
| I've never heard anyone define memory safety that way. You
| can segfault by overflowing stack space and hitting the guard
| page or dereferencing a null pointer. Those are possible in
| languages that don't even expose their underlying pointers
| like Java. You can make Python segfault if you set the
| recursion limit too high. Meanwhile a memory access bug or
| exploit that does not result in a segfault would still be a
| memory safety issue.
|
| Memory safe languages make it harder to segfault but that's a
| consequence, not the primary goal. Segfaults are just another
| memory protection. If memory bugs only ever resulted in
| segfaults the instant constraints are violated, the hardware
| protections would be "good enough" and we wouldn't care the
| same way about language design.
| jlouis wrote:
| The definition has to do with certain classes of spatial and
| temporal memory errors. Ie., the ability to access memory
| outside the bounds of an array would be an example of a
| spatial memory error. Use-after-free would be an example of a
| temporal one.
|
| The violation occurs if the program keeps running after
| having violated a memory safety property. If the program
| terminates, then it can still be memory safe in the
| definition.
|
| Segfaults has nothing to do with the properties. There's some
| languages or some contexts in which segfaults is part of the
| discussion, but in general, the theory doesn't care about
| segfaults.
| zozbot234 wrote:
| Both spatial and temporal memory unsafety can lead to
| segfaults, because that's how memory protection is intended
| to work in the first place. I don't believe it's feasible
| to write a language that manages to _provably_ never trip a
| memory protection fault in your typical real-world system,
| yet still fails to be memory safe, at least in _some_ loose
| sense. For example, such a language could never be made to
| execute arbitrary code, because arbitrary code can just
| trip a segfault. You 'd be left with the sort of type
| confusion logical error that happens all the time anyway in
| all sorts of "weakly typed" languages - that's not what
| "memory safety" is about.
| Yoric wrote:
| Segfaults are just the simplest way of exposing a memory issue.
| It's quite easy to use a race condition to reproduce a state
| that isn't supposed to be reachable, and that's much worse than
| a segfault, because it means memory corruption.
|
| Now the big question, as you mention, is "can it be exploited?"
| My assumption is that it can, but that there are much lower-
| hanging fruits. But it's just an assumption, and I don't even
| know how to check it.
| Mawr wrote:
| There is no house safety without nuclear warhead detonation
| safety.
|
| There is no pedestrian safety without mandatory helmet laws.
|
| There is no car safety without driving a tank.
| FiloSottile wrote:
| I have never seen real Go code (i.e. not code written
| purposefully to be exploitable) that was exploitable due to a
| data race.
|
| This doesn't prove a negative, but is probably a good hint that
| this risk is not something worth prioritizing for Go applications
| from a security point of view.
|
| Compare this with C/C++ where 60-75% of real world
| vulnerabilities are memory safety vulnerabilities. Memory safety
| is definitely a spectrum, and I'd argue there are diminishing
| returns.
| stouset wrote:
| Maintenance in general is a burden much greater than CVEs.
| Exploits are bad, certainly, but a bug not being exploitable is
| still a bug that needs to be fixed.
|
| With maintenance being a "large" integer multiple of initial
| development, anything that brings that factor down is probably
| worth it, even if it comes at an incremental cost in getting
| your thing out the door.
| 9rx wrote:
| _> but a bug not being exploitable is still a bug that needs
| to be fixed._
|
| Do you? Not every bug needs to be fixed. I've never see a
| data race bug _in documented behaviour_ make it past initial
| development.
|
| I have seen data races _in undocumented behaviour_ in
| production, but as it isn 't documented, your program doesn't
| have to do that! It doesn't matter if it fails. It wasn't a
| concern of your program in the first place.
|
| That is still a problem if an attacker uses undocumented
| behaviour to find an exploit, but when it is benign... Oh
| well. Who cares?
| LtWorf wrote:
| I have! What do i win?
| crawshaw wrote:
| Memory safety is a big deal because many of the CVEs against C
| programs are memory safety bugs. Thread safety is not a major
| source of CVEs against Go programs.
|
| It's a nice theoretical argument but doesn't hold up in practice.
| nine_k wrote:
| A typical memory safety issue in a C program is likely to
| generate an RCE. A thread-safety issue that leads to a segfault
| can likely only lead to a DoS attack, unpleasant but much less
| dangerous. A race condition can theoretically lead to more
| powerful attacks, but triggering it should be much harder.
| stouset wrote:
| A CVE is worse, but a threading bug resulting in corrupted data
| or a crash is still a bug that needs someone to triage,
| understand, and fix.
| crawshaw wrote:
| But it's not why I stopped writing C programs. It's just a
| bug and I create and fix a dozen bugs every day. Security is
| the only argument for memory safety that moves mountains.
| loeg wrote:
| Are we still have semantic fights about what exactly memory
| safety means? Why?
| pizlonator wrote:
| False.
|
| Java got this right. Fil-C gets it right, too. So, there is
| memory safety without thread safety. And it's really not that
| hard.
|
| Memory safety is a separate property unless your language chooses
| to gate it on thread safety. Go (and some other languages) have
| such a gate. Not all memory safe languages have such a gate.
| glowcoil wrote:
| I would recommend reading beyond the title of a post before
| leaving replies like this, as your comment is thoroughly
| addressed in the text of the article:
|
| > At this point you might be wondering, isn't this a problem in
| many languages? Doesn't Java also allow data races? And yes,
| Java does allow data races, but the Java developers spent a lot
| of effort to ensure that even programs with data races remain
| entirely well-defined. They even developed the first
| industrially deployed concurrency memory model for this
| purpose, many years before the C++11 memory model. The result
| of all of this work is that in a concurrent Java program, you
| might see unexpected outdated values for certain variables,
| such as a null pointer where you expected the reference to be
| properly initialized, but you will never be able to actually
| break the language and dereference an invalid dangling pointer
| and segfault at address 0x2a. In that sense, all Java programs
| are thread-safe.
|
| And:
|
| > Java programmers will sometimes use the terms "thread safe"
| and "memory safe" differently than C++ or Rust programmers
| would. From a Rust perspective, Java programs are memory- and
| thread-safe by construction. Java programmers take that so much
| for granted that they use the same term to refer to stronger
| properties, such as not having "unintended" data races or not
| having null pointer exceptions. However, such bugs cannot cause
| segfaults from invalid pointer uses, so these kinds of issues
| are qualitatively very different from the memory safety
| violation in my Go example. For the purpose of this blog post,
| I am using the low-level Rust and C++ meaning of these terms.
|
| Java is in fact thread-safe in the sense of the term used in
| the article, unlike Go, so it is not a counterexample to the
| article's point at all.
| pizlonator wrote:
| > I would recommend reading beyond the title of a post before
| leaving replies like this, as your comment is thoroughly
| addressed in the text of the article:
|
| The title is wrong. That's important.
|
| > Java is in fact thread-safe in the sense of the term used
| in the article
|
| The article's notion of thread safety is wrong. Java is not
| thread safe by construction, but it is memory safe.
| dwattttt wrote:
| If a language is "memory safe", by some definition we
| expect safety from memory faults (for example, not
| accessing memory incorrectly).
|
| If a language is "memory safe" but not "thread safe", is
| the result "the language is free from 'memory faults',
| unless threads are involved"?
|
| Or to put it another way; when used however the term of art
| is intended, "memory safety" is meant to provide some
| guarantees about not triggering certain erroneous
| conditions. "not thread safe" seems to mean that those same
| erroneous conditions can be triggered by threads, which
| seems to amount to '"memory safety" does not guarantee the
| absence of erroneous memory conditions'.
| dwattttt wrote:
| I guess to also elaborate the point; it's also entirely
| correct to say "Rust is guaranteed to be memory safe
| unless 'unsafe' is involved".
| pizlonator wrote:
| Yeah and Rust is guaranteed to be thread safe unless
| 'unsafe' is involved, I think
| pizlonator wrote:
| > If a language is "memory safe" but not "thread safe",
| is the result "the language is free from 'memory faults',
| unless threads are involved"?
|
| Yes.
|
| If a language is memory safe but not thread safe, then
| you can race, but the outcome of those races won't be
| memory corruption or the violation of the language's type
| system. It will lead to weird stuff, however - just a
| different kind of weirdness than breaking out of the
| language's sandbox
| kiitos wrote:
| > To see what I mean by this, consider this program written in
| Go, which according to Wikipedia is memory-safe:
|
| The Wikipedia definition of memory safety is not the Go
| definition of memory safety, and in Go programs it is the Go
| definition of memory safety that matters.
|
| The program in the article is obviously racy according to the Go
| language spec and memory model. So this is all very much tilting
| at windmills.
___________________________________________________________________
(page generated 2025-07-24 23:02 UTC)