commented: If I continue to have say in this, wordexp() will never be in OpenBSD's libc. Perhaps if there is one open source project which refuses to have it, coders will continue avoiding use of wordexp(). —Theo de Raadt (2010) commented: Oh wow, it’s an actual quote! commented: Additionally: No kidding. I agree with Ted; thanks for the effort but the code you borrowed is bad, and it is was designed by people who the unabomber should have gotten to first. —Theo de Raadt (2010) commented: I have seen this in libc, and I wonder why it is there at all. No shells use it to implement anything. Tilde expansion can be implemented with a much more straightforward algorithm. And if it just forks a shell, yes that is incredibly dangerous !! (Not to mention making it impossible for shells to use it :-) ) I think the problem is that the specification of such a function involves the entire shell, because the entire shell language can appear inside command subs: $ echo $( f() { echo 'defined function'; }; f; f && f; f | f ) defined function defined function defined function defined function Nobody is going to implement an entire shell inside libc ... that would be crazy. It seems like it should be removed from libc. Not sure who put it there, and why. commented: Yeah, there are quite a few POSIX APIs that directly or indirectly assume the presence of a full sh lexer, a full C lexer, etc. But very few (approachably) POSIX systems are built such that the sh or C lexer is reusable from libc, so we end up with these crazy hacks instead. Not sure who put it there, and why. The post I linked to has an interesting explanation (quoting): It seems the problem was that people kept demanding more and more features for glob, so it seems like the committee threw up their hands and added the far-too-broad wordexp in attempt to cover all possible bases. From: https://www.cipht.net/2023/11/21/worried-by-wordexp.html commented: Yeah, there are quite a few POSIX APIs that directly or indirectly assume the presence of a full sh lexer, a full C lexer, etc. But very few (approachably) POSIX systems are built such that the sh or C lexer is reusable from libc, so we end up with these crazy hacks instead. At least nowadays there are C-frontend-as-libraries (i.e. libclang); I'm not aware of a POSIX shell that's usable as a library. The Unix distinction between executables and libraries (and lack of an omnipresent shell like DCL) is unfortunate. commented: I think the problem is that the specification of such a function involves the entire shell, because the entire shell language can appear inside command subs FWIW, glibc uses the system shell for command substitution, but implements all of the other word expansion logic itself. commented: A few other things that make wordexp(3) dangerous: You can’t use it in a multithreaded program, because it usually forks a shell. On glibc wordexp is built in, but shell expansions that set variables call setenv() which is not thread safe, and which also modifies the caller’s environment which traditional forky wordexp() does not. (libc basically never calls setenv which is why this is a notable oddity.) It really is a lunatic function that should be documented with a big “do not use this” health warning. commented: It really is a lunatic function that should be documented with a big “do not use this” health warning. Sadly, not something that makes it unusual in POSIX. commented: A while back I came to the same conclusion when researching wordexp, because we were having discussions about adding a wordexp:: module to the Hare standard library. I was extremely against it, both because it's impossible to use safely on untrusted input (as you noted), and also because pretty much no one fully implements it correctly: implementations like musl's are hacks where it's trivial to find stuff like shell bypasses, and implementations like glibc which don't use the system shell (except for command substitution) can't handle edge cases like $(case 0 in *) echo hi; esac) The compromise we ended up on was to ignore POSIX altogether. This means that Hare's wordexp always uses the system shell (even if it isn't a POSIX shell), and, more importantly, we don't implement WRDE_NOCMD at all; we instead document that the interface is only suitable for trusted inputs: Note that, by design, this module runs arbitrary shell commands from user-supplied inputs. It must only be used in a trusted environment. IMO providing a flag like WRDE_NOCMD which doesn't actually make the interface safe to use is worse than providing nothing at all, since it gives a false illusion of security. The main motivator for us to implement wordexp was for the upcoming hare.ini configuration format, to declare dependencies and linker flags and stuff (that way the user can use pkg-config, for example). Although wordexp wouldn't have been my first choice, I'm happy with the compromise we ended up with. commented: implementations like musl's are hacks where it's trivial to find stuff like shell bypasses Yes I am going to start using the term "parser differential" for this. It is often used in the context of say web firewalls trying to parse HTML incorrectly and block stuff, but I see it extremely often in relation to shell -- simply because shell is hard to parse, and people write a second, crappy, and incomplete shell parser. https://nhimg.org/glossary/parser-differential/ And even bash can't parse shell correctly in many cases, after 30 years of trying :-) That is one reason I started https://oils.pub/ I don't have the bugs handy, but simply parsing nested () {} $() ${} in shell is extremely difficult, and the unbalanced case 0 in *) is one reason, but not the only one. commented: Oh and now I remember the reason I was looking into parser differentials. I noticed that there were two cases of longstanding design bugs in the Go stdlib that can be thought of as parser differentials: (1) One them was a motivation for JSON v2 - https://go.dev/blog/jsonv2-exp (Sept 2025) e.g. two parsers handling {"admin": false, "admin": true} differently An example (not involving Go) is https://nvd.nist.gov/vuln/detail/cve-2017-12635 - Due to differences in the Erlang-based JSON parser and JavaScript-based JSON parser (2) The other parser differential is in contextual auto-escaping in Go's template packages https://nvd.nist.gov/vuln/detail/cve-2023-24538 - Templates do not properly consider backticks (`) as Javascript string delimiters, and do not escape them as expected. Backticks are used, since ES6, for JS template literals Basically Go has a crazy HTML/CSS/JS incomplete parser / state machine, in order to do automatic escaping. But the difference between that parser and the real one can lead to missed escaping of untrusted variables, and thus a vulnerability. If shell and HTML and other "crappy languages" weren't already entrenched enough, then LLMs really put the nail in the coffin. We will be using shell and HTML "forever". But maybe we can do better with LLMs ... i.e. nobody wants to do all this work to migrate toward stricter languages (e.g. OSH vs bash, which is easy in 99.9% of cases), but we really should. commented: With bash being as widespread as it is, by doing it "correctly" in oils, aren't you creating just another parser differential? commented: I'd define parser differential as a security issue -- and in that case, the bug is that wordexp() exists, and it has a flag that implies parsing shell, which it does incorrectly. (The problem is not even well-posed, because Parsing bash is undecidable, and bash is /bin/sh on Red Hat machines). So even if everyone switched to OSH, wordexp() would STILL be bad and unsafe. That's just fundamental. On the other hand, It's true there are differences between bash and OSH. Some of the differences I see as a feature, rather than a bug. The most obvious one is that OSH lacks this security issue that bash, ksh, and zsh have: https://lobste.rs/s/mla0ns/til_some_surprising_code_execution (now I see this post is also from @yossarian !) At first, OSH was indeed stricter than it should have been by default (more differences with bash). Over time, more of the differences have moved under shopt --set strict:all. So OSH is extremely compatible with bash, but you can opt into a better behavior. One thing we could do to reduce "parser differences" is subsume the functionality of shfmt (parser in Go) and ShellCheck (parser in Haskell). However I am only going to experiment along those lines, not promise it will be done, because it's a huge amount of work. :-) Probably the experiment will need to involve ways to make the problem more tractable -- formatting bash is harder than formatting Python or JavaScript, simply because it has so much more syntax. commented: Honestly, even without command expansion, I think it's an iffy function. I went through different implementations a decade ago on Twitter, and many just shelled out directly (as other people here have mentioned). It doesn't seem safe to use in many circumstances. https://www.openwall.com/lists/oss-security/2015/02/10/11 (caveat here is that this is from 2015, and clearly some implantations have changed since) commented: Digging through https://github.com/apple-oss-distributions/, it looks like standard utilities shipped with macOS actually use wordexp themselves here and there, though I imagine most of this is inherited from upstream sources. For example, it's used in the :e command in less. There's also some special-case handling in sudo, but this appears to be defensive in case a different program calls wordexp via sudo. commented: One person’s dangerous is another person’s useful ;) i3’s “include” config file directive uses wordexp and the userguide includes examples like wildcard expansion (include ~/.config/i3/config.d/*.conf) and command substitution (include ~/.config/i3/hostname.conf): https://i3wm.org/docs/userguide.html#include commented: Using it for things like i3 config files where any attacker who can write to them can own you anyway is fine; the problem is that sometimes people just want to parse tildes, variables, and maybe globs the "safety" mechanism is basically impossible to implement correctly So it'd be better if there was a separate function that was guaranteed to only parse tildes, basic $VAR substitution, and maybe glob expansion (ideally with knobs for controlling which of these), and it was required to be safe on untrusted input and not just secretly shell behind your back. commented: There is glob() which does glob expansion. Linux allows one to do tilde expansion but that's not POSIX. No mention of environment substitution though. commented: There's a bit of a footgun if you don't turn off features you don't need you get features you don't want. That's poor API design. It sounds like more of the issue is that there are some buggy libcs so that if you want shell expansion. I'm certainly not eager to rely on a libc with lines of code that look like: if (flags & WRDE_NOCMD) for (i=0; s[i]; i++) switch (s[i]) { .