[HN Gopher] Rustaceans at the border
___________________________________________________________________
Rustaceans at the border
Author : bitcharmer
Score : 246 points
Date : 2022-04-15 06:56 UTC (16 hours ago)
(HTM) web link (lwn.net)
(TXT) w3m dump (lwn.net)
| jstimpfle wrote:
| In my eyes, the problem isn't precisely whether some components
| are _downloaded_ separately. What I find problematic is the idea
| of giving up responsibility by dependending on separately
| developed components. Components written with more than just one
| particular project in mind are tough to change. Integrating them
| into a particular project often requires workarounds and some
| imperfect abstractions to make things fit, and there tends to be
| a proliferation of blind spots about "global optimization
| potential".
|
| Not to drag on Rust, but I can see a "global optimization"
| problem when I try to cargo build a moderate Rust project and
| have to download hundreds of transitive dependencies, and have a
| good chance of ending up not building a thing (for reasons that
| probably include my own incompetence, but still).
|
| There are of course tons of C projects that have similar problems
| in a way - the number of dependencies might be lower but I have
| to hunt them manually. But well-engineered projects are at most a
| handful of clearly identifiable dependencies, and everything will
| generally build without fuss. This is how it is with the Linux
| kernel today. The main dependency of the Linux kernel is gcc, and
| as far as I perceive, communication between Linux and gcc
| projects is alive.
|
| I think there is a lesson that can be learned in particular from
| C development: There is value in growing a system vs mainly
| integrating existing stuff (junk or not). The larger a project
| gets, the more sense it makes for it to bring its own tools and
| implementations so it can continue to build and be maintained as
| a whole without a lot of complications.
|
| This might be the difference between a huge project and an
| ecosystem, and the question is where are people aiming with Rust
| support? If it's just about drivers I can see some limited value
| in the ecosystem approach but it will be a tough sell for any
| Rust driver that wants its 'M' in the kernel config.
| kibwen wrote:
| _> Components written with more than just one particular
| project in mind are tough to change._
|
| Even in the very worst-case scenario, you can fork the code.
| That would still leave you far ahead of having to write
| everything you need from scratch. And my experience is that
| plenty of libraries are happy to support general use cases when
| at all possible.
|
| _> and have to download hundreds of transitive dependencies,
| and have a good chance of ending up not building a thing_
|
| Can you give an example of what you mean by this? I'm unclear
| what the concern is regarding the idea of "not building a
| thing".
|
| _> as far as I perceive, communication between Linux and gcc
| projects is alive_
|
| The Rust developers also have open and active lines of
| communication to the Linux developers.
|
| _> The larger a project gets, the more sense it makes for it
| to bring its own tools and implementations so it can continue
| to build and be maintained as a whole without a lot of
| complications._
|
| Certainly, but even very large projects benefit from sharing
| code in foundational areas.
| jstimpfle wrote:
| >> Even in the very worst-case scenario, you can fork the
| code.
|
| You can fork the code that you never knew well enough to
| write yourself.
|
| >> and have to download hundreds of transitive dependencies,
| and have a good chance of ending up not building a thing
|
| > Can you give an example of what you mean by this? I'm
| unclear what the concern is.
|
| My concern is of course that as someone who wants, or is
| supposed to, mess with a project, I can easily get depressed
| if I have to invest significant to infinite amounts of energy
| to bring that project to build. And if I get it to build,
| have a hard time figuring out how it works because what it
| does is all over the place, hidden in hundreds of libraries.
|
| > Even very large projects benefit from sharing code in
| foundational areas.
|
| "Other crates that I'd like: anyhow, bincode, byteorder, log,
| once_cell, pin-project, rand, serde, slab, static_assertions,
| uuid plus some more esoteric ones."
| kibwen wrote:
| _> You can fork the code that you never knew well enough to
| write yourself._
|
| Certainly. It is hard to envision any circumstance where
| one is worse off for having the opportunity to stand on the
| shoulders of giants rather than having to invent the
| universe from scratch.
|
| _> I can easily get depressed if I have to invest
| significant to infinite amounts of energy to bring that
| project to build_
|
| Can you give an example of a Rust project where you have
| had this experience? Nearly every Rust project is as simple
| to build as `git clone && cargo build`. The exceptions are
| those with C dependencies, where you will need to turn to
| your C package manager (apt, etc.) to first install the C
| dependencies, and it's hard to claim this as a weakness of
| Rust relative to C when it's no worse than what you'd get
| in a pure C project.
|
| _> Other crates that I 'd like:_
|
| Can you be more specific about which of these crates
| provides functionality which you do not think qualifies as
| generally foundational?
| jstimpfle wrote:
| > Can you give an example of a Rust project where you
| have had this experience?
|
| Trying to make a better experience I just randomly picked
| "Weylus" from Github's rust/trending list and wasted more
| than 1 hour building it on Windows 10 and Debian
| Bullseye, ultimately quitting both. Yes, it was a lot of
| C library related issues, but also other executables
| missing, like tsc, or in case of Windows, make. On both
| systems, the number of transitive dependencies was about
| 300, and so I guess it was inevitable that there were
| going to be some problems.
|
| I've had similar problems before that trying to build
| other, less ambitious programs than Weylus using cargo.
|
| > it's hard to claim this as a weakness of Rust relative
| to C when it's no worse than what you'd get in a pure C
| project.
|
| It seems to me that making it more reliable to depend on
| other modules (like cargo does) does not lead to easier
| to use projects, since it doesn't change the amount of
| bullshit that most developers and users are willing to
| endure. This is an instance of Parkinson's law and I
| think it was already a learning from the node.js
| ecosystem.
|
| As a consequence, with a system like cargo programs end
| up having far more dependencies (which is empirically
| true), but the programs aren't easier to build by a
| random user. For developers, the situation is changed
| compared to e.g. C, in that it is easier to add more
| dependencies before the software becomes unmaintainable.
| This might shift the development situation to a place
| where the average developer in this ecosystem is more
| competent in plumbing things than understanding the
| problems and coming up with reliable solutions that are
| easy to maintain without a system like cargo.
|
| I'm not sure if this is good or bad - probably it's an
| evolution that can't be stopped, and which creates new
| types of developers. What I'm saying is I'm not positive
| that this attitude is compatible with a central piece of
| infrastructure, like the Linux kernel is.
|
| > Can you be more specific about which of these crates
| provides functionality which you do not think qualifies
| as generally foundational?
|
| I'm not a Rust developer and I don't know any of these,
| but looking at those names I'm wondering: which is not
| some arbitrary helper thing, where it would be easy to
| maintain an alternative in-tree? At least that's better
| than each developer bringing their own slightly different
| preference, resulting in more bloat and maintainability
| problems.
| proto_lambda wrote:
| I can sympathize with the need to have all required source code
| in the repository and not having to fetch a bunch of dependencies
| at build time. Thankfully, cargo already offers a solution here:
| `cargo vendor` will download all the specified dependencies once
| into a local directory, which can then be checked into the source
| tree.
|
| This maintains cargo's dependency resolution/update checking/etc,
| but also allows for all dependency code to be kept alongside the
| kernel code and audited accordingly.
| humanrebar wrote:
| Will there be a massive review process every time someone
| updates the vendored dependencies? Or will each dependency
| change be reviewed one release increment at a time?
|
| What happens if a dependency adds a system call something?
| Libraries intended for kernel-friendly use cases really need
| that scope to be an intentional goal.
| kibwen wrote:
| Of course, they would certainly review any updates to
| vendored code. Just because most companies are too lazy to
| audit their dependencies doesn't mean the kernel needs to be.
|
| As for syscalls, Rust has a whole ecosystem of no_std crates
| for kernel and microcontroller development that already
| assume the lack of an OS. We use (and contribute to) such
| crates extensively in our product (which is already developed
| jointly with the Linux Foundation, though we're not working
| on the Linux kernel (well we are a little bit, but all of
| that work is still in C :P )) and can vouch for their
| quality.
| jupp0r wrote:
| > Just because most companies are too lazy to audit their
| dependencies
|
| They are not "too lazy". It's not economically viable for
| them to do so for various reasons.
| the8472 wrote:
| This is more a coordination/incentive/game theory problem
| than a cost one. If all the companies that use open
| source libraries contributed resources to pooled audits
| then individually they'd have to pay far less than each
| reviewing dependencies on their own. Maybe that could be
| incentivized by penalizing data breaches caused by
| negligence and using non-audited code = negligence. But I
| suspect this would just result in people running some
| random static analysis tool and calling it a day rather
| than doing for proper code reviews.
| scns wrote:
| Penalizing does not work, period. Religion tried with
| scare tactics for thousands of years. Sharing costs for
| auditing open source libraries depended on would need a
| platform to share the load, something like a Patreon for
| businesses.
| geodel wrote:
| Ah this makes total sense. Other day I read how some
| factories are dumping pollutant chemicals in rivers. But
| it turned out to be all fine as there was no wrong
| intentions it just was not economically viable for them
| to properly dispose waste.
| notriddle wrote:
| Being "not lazy" isn't the same thing as being "fine."
| What it means is that the problem is with the game, not
| the players.
| joshmarlow wrote:
| > What happens if a dependency adds a system call something?
|
| AFAIK, in Haskell, any function that returns IO must indicate
| it in the type signature.
|
| If you could find a way to take that idea, expand upon it
| with much more fine-grained types, then you could build an
| ecosystem where any system call, external network call, use
| of env variables, etc is baked into the type signature of
| every function and package. If you could do that, you could
| build pretty trivial checks to ensure that a given package
| doesn't perform any kind of system call.
|
| So sort of like the types of permissions we assign to apps on
| Android, but represented (and enforced) directly in the type
| system.
|
| It would be difficult to make something like this ergonomic
| but could be pretty cool.
| shirogane86x wrote:
| I feel like effect systems like the ones in unison and koka
| (or haskell with extensible effects / freer libraries like
| polysemy or eff) get pretty close to that. I feel like
| there's a lot of effort needed to make the idea
| "mainstream" and to get a good implementation going though
| (I think a lot of implementations use delimited
| continuations, but I'm not sure so don't quote me on that.
| At least it's the design for haskell's new primops to make
| that kind of library better)
| steeleduncan wrote:
| This works in Haskell because it is a purely functional
| language, and the IO monad is an intrusion into that to
| allow for useful computation. Rust is not purely
| functional, so there is no way to enforce something like
| this in the compiler. You could add an IO monad, but it
| would be easy for someone further down the chain of
| packages to ignore it and make a syscall.
|
| Anything similar in rust would have to be enforced through
| code auditing tools, either forking the compiler, using
| some of its code as a basis or starting from scratch.
| joshmarlow wrote:
| Oh you're totally right; I think it would have to be a
| purely functional framing - I don't think we could shoe-
| horn this into Rust.
|
| I should have prefaced my comment with a note that this
| was way more speculative :)
| steeleduncan wrote:
| Yes, it is a shame that no version of that purely
| functional Haskell ideal has been created that could
| reasonably be used for kernel development. Using the type
| system to constrain side effects of code in the way you
| suggest would eliminate massive classes of security
| vulnerabilities and crashes.
| Quekid5 wrote:
| Copilot is used for hard real time systems.
|
| Now, Copilot is probably more specialized than even
| kernel dev, but AFAICT there's not really any hard
| barrier there. "Just" a question of effort.
| staticassertion wrote:
| IO is _necessary_ because of purity, but it 's in no way
| enabled by it. You don't need purity to have the IO
| monad, and you don't need purity to enforce that IO is
| defined in a type signature.
| jupp0r wrote:
| counterexample: trace
| Quekid5 wrote:
| If you're going to pick a "counterexample" just say
| unsafePerformIO, or the aptly named
| accursedUnutterablePerformIO[0]. (There are quite a few
| variants.)
|
| Everybody knows that there are escape hatches because
| they _are_ actually sometimes absolutely required for
| asymptotics or just plainly because it may be too hard to
| "prove" that your code is safe to the compiler.
|
| That isn't a gotcha -- you know exactly what code you
| need to audit extra carefully.
|
| [0] That one's actually in ByteString to technically not
| standard Haskell, but I just like the name.
| akira2501 wrote:
| I'm suspicious of someone who says they need to download and
| run /complex math/ libraries inside of a kernel module. I
| seriously wonder what they're planning on developing, and if a
| hybrid approach with most of that code being in user space
| wouldn't be a better idea here.
|
| I wonder if this is a real and considered need, or a knee-jerk
| response to try to put everything in rust and directly in ring
| 0 just for the sake of it. I see no reason to try to break down
| the "divide between kernel and user space" in this way, and I
| wonder what's actually driving it here.
|
| I wish this article went a little deeper into _what
| specifically_ these Rust users are trying to make modules
| _for_.
| jerf wrote:
| I don't think it's even remotely likely the kernel will end up
| supporting Cargo in any form.
|
| As said in the article, "the world is changing". However, the
| way the world is changing is that more and more people are
| aware of supply-chain attacks. While I acknowledge that Cargo
| isn't npm, it is still the case that of all the software that
| can not afford to just sorta grab things from the internet, the
| Linux kernel is arguably #1, straight up. There is no chance
| that the kernel developers are ever going to accept "well, I
| wanted some async stuff so I grabbed a v0.5.23 of a package
| that I found appealing". Even if they pull some stuff in, it
| will be through a review process and it won't be through cargo
| in general.
|
| The argument against cargo or any equivalent being used in the
| kernel is stronger today than it was 10 years ago, and the
| first derivative is also positive. Probably the second one too,
| honestly. This isn't about kernel developers being old fogey
| sticks in the mud, this is about the kernel being such a high-
| assurance environment, the biggest, fattest target in the
| world, that things that make sense for most software packages
| don't make sense for it. And it has nothing to do with cargo
| specifically or Rust. It's just that the entire workflow
| afforded by cargo, or npm, or go modules, or the half-dozen
| Python package managers, or anything else resembling those
| things is simply not appropriate for the Linux kernel. The only
| way to make such a thing work would be to pin the versions so
| hard that you're effectively only using them as a downloader,
| not a package manager, and you might as well just have vendored
| code in the kernel repo anyhow.
| jacoblambda wrote:
| > There is no chance that the kernel developers are ever
| going to accept "well, I wanted some async stuff so I grabbed
| a v0.5.23 of a package that I found appealing". Even if they
| pull some stuff in, it will be through a review process and
| it won't be through cargo in general.
|
| > The only way to make such a thing work would be to pin the
| versions so hard that you're effectively only using them as a
| downloader, not a package manager, and you might as well just
| have vendored code in the kernel repo anyhow.
|
| This is the exact thing the person you replied to said should
| be done. `cargo vendor` fetches the sources for the package @
| the version stated and embeds them in the repo. After that
| all deps would be sourced from the repository itself.
|
| Nothing in the review process would have to change beyond
| adding a dependencies directory to the repo with a dedicated
| set of CODEOWNERS to handle reviewing patches with new
| dependencies.
| jerf wrote:
| Yes, I meant it as amplification, not disagreement, but
| upon review I see I did bad job. The post got away from me
| in edits. Sorry.
| pjmlp wrote:
| Android and Fuchsia are already using Rust, and they also
| abstain to use cargo directly.
| kibwen wrote:
| Just to be clear, because I see this misconception
| frequently, using Cargo does not require using crates.io. You
| can set up your own private registry and configure Cargo to
| use it, if you like. You can even use Cargo entirely offline.
| lodovic wrote:
| Requiring packages to build the kernel source feels like a
| circular dependency to me. The kernel should contain all of its
| own source code.
| nrabulinski wrote:
| So now a kernel cannot depend on anything? What about all the
| tooling that's required like python, perl and many others or
| the fact that you have to have a working machine with the
| kernel already on it to build the kernel?
| yjftsjthsd-h wrote:
| Obviously you have to bootstrap from something. A more
| interesting case is NetBSD, which can bootstrap from just
| about anything; take a NetBSD src checkout, run ./build.sh,
| and it will first use the local tooling to build its own
| dependencies, then use _those_ to build the system. So in
| _some_ OSs, _yes_ , you can more or less vendor in the
| universe. But that only works because *BSDs are developed
| as full systems, not just a kernel; Linux probably can't
| (and arguably shouldn't) do that.
| spaintech wrote:
| OMG, here comes a bunch of bloat into the kernel. I like rust
| (well, the idea and principle) but when you are doing low level
| stuff, something as trivial as a structure to bind an address
| then manipulate those bits by writing to them (enabling and
| disabling features) it becomes a no go. If you even consider how
| exhaustive it is to manage OBMR at this level of code, it becomes
| counter productive, not to mention unreadable. Now for the bloat
| part, take a simple code (one of their examples) which is a
| showcase of their capabilities and "zero cost" abstraction for
| something like embedded, their blinky app is like 100KB versus a
| initialize the clock/ set GPIO to read mode, and write to it that
| come out shy of 4K in C++ with all the hall form STM for example.
| Object lifetime analysis is really cool at compile time, but any
| un defines behavior is still present when you compile the code...
| Show me someone doing a baremetal implementation of blinky (that
| is, no HAL) on rust or a struct for bit banging and I'll be a
| believer that you can do "some" of the stuff needed for kernel
| development but not all.
| PoignardAzur wrote:
| "Please show me some amazing features of your language but I'll
| still believe your language sucks".
|
| That's, uh, not a very compelling request.
| varajelle wrote:
| > their blinky app is like 100KB
|
| I just compiled the blinky example for a cortex m4 board (the
| example in the rust hal crate for that board, not using any
| unsafe), and the striped binary is 420 bytes.
|
| So you probably want to review your stence, because you're
| about 3 orders of magnitude off.
| spaintech wrote:
| Yes its 420 bytes, which is why I'm commenting as my fingers
| played me on my thoughts. Do the same for one writer in C++
| and report back on the orders of magnitudes... Cheers!
| AnyTimeTraveler wrote:
| I am currently working with the ATTiny85 and the stripped
| blinky binary is 275 bytes. I was worried that the ATTiny85
| wouldn't have enough flash space (6K) for Rust and now I
| don't know what to do with all that free space :)
| ralls_ebfe wrote:
| You can use rust on an attiny85? What a time to be alive. I
| wonder where mine are.
| fmajid wrote:
| Rust's cavalier attitude to language and compiler stability,
| their absurd bootstrapping situation and limited platform
| support, not to mention their belief that "curl something|bash"
| is acceptable procedure are all reasons why I've avoided it
| despite the many good qualities of the language.
| jopsen wrote:
| Aren't many of those things stuff that can be fixed as the
| ecosystem matures?
|
| Is C ecosystems much better?
| humanrebar wrote:
| I wouldn't phrase my concerns that way, but it is concerning
| that Rust doesn't do LTS releases (someone correct me if I
| missed something here).
|
| Because C compilers, make, etc. are really stable, existing
| Linux source tarballs have a habit of building fine for decades
| after initial release.
| tialaramex wrote:
| In what sense is Rust 1.0 not already an LTS release ?
|
| If you wrote some Rust 1.0 code back in 2015, put it away in
| a drawer (maybe on a USB stick) but now get it out today it
| will still build with current Rust tooling -- except for some
| very narrow cases where you might have done something
| inherently _unsound_ and subsequently the compiler was
| corrected to fix that so it 's an error.
|
| Rust's language shifted slightly in those years. However the
| Editions system - even though it hadn't yet been invented in
| 2015 - allows for that. Your 2015 code lacks any Edition
| metadata, so, the modern tools understand that as Rust 2015
| edition, and will compile accordingly, while still
| interoperating correctly with modern Rust.
|
| Suppose your 2015 code named a variable await. That's a
| keyword in modern Rust. But it isn't a keyword in Rust 2015
| edition, so your code compiles just fine. This would not work
| in C of course, which is why its newer keywords are ugly
| stuff like _Bool but in Rust it's fine.
| humanrebar wrote:
| Modern releases of GCC still support C89. If rustc, at
| least, will keep supporting 1.0 features for all future
| releases, I guess that's fine.
|
| But that also means that crates used by the kernel would
| need to likewise be conservative about updating to new
| Editions so as to not break expected support surfaces.
| tialaramex wrote:
| So, firstly the editions system means it doesn't even
| matter about crates using a different edition. Remember
| that await variable in the Rust 1.0 code above? My Rust
| 2021 code can still talk about that variable, even though
| it's from a different edition and the word "await" is now
| a keyword, it just calls that variable r#await meaning
| "the identifier named await" - which is a bit ugly but
| gets the job done for interoperability purposes.
|
| But also, all previous versions of published crates are
| kept indefinitely. If Linux wants serde v1.0.240 then
| that's fine, even if subsequently serde shipped v1.0.241,
| v1.1.14 and v2.0.1 the repository holds on to everything.
| humanrebar wrote:
| It matters if certain drivers require certain toolchains.
| C89 is portable across all sorts of C toolchains. If the
| Linux kernel added C17 features, only toolchains that
| support C17 could compile the kernel.
|
| You're correct that the kernel codebase could pin older
| versions of crates when it is appropriate, but it's never
| quite that simple at scale, especially if the kernel
| pulls in more than a handful of crates.
| cesarb wrote:
| > C89 is portable across all sorts of C toolchains. If
| the Linux kernel added C17 features, only toolchains that
| support C17 could compile the kernel.
|
| Funny you should mention that. The Linux kernel recently
| switched from C89 to C11 (https://git.kernel.org/pub/scm/
| linux/kernel/git/torvalds/lin...), so only toolchains
| that support C11 can compile the kernel.
|
| Even before that, the minimum version of GCC required to
| compile the kernel has risen several times. The current
| minimum is GCC 5.1, so the GCC 4.9 which you could use to
| compile the kernel one year ago is no longer good enough.
| roblabla wrote:
| Wat?
|
| Rust has been a very stable language since Rust 1.0. They have
| a stellar record of keeping things working - with most code
| breaking being due to said code invoking UB. The edition system
| is a brilliant invention that allows evolving the language
| _without_ causing an ecosystem split. Thanks to this, Rust ends
| up having a much better stability story than even C++ (for whom
| you can't really mix and match different C++ versions).
|
| The bootstrapping situation is really not that bad? We have
| mrustc (A C++ rust->C transpiler) which allows compiling modern
| versions of rustc (latest supported rustc version being 1.54),
| which we can then iteratively bootstrap from up to the latest
| version. And things are getting better, with gccrs[0] in
| particular promising a rust frontend for GCC, written in C.
|
| As for the "curl something|bash", I suppose you're talking
| about rustup. You're free to download the script, and review it
| before installing it. And rust is also distributed many
| different ways. At least `curl something|bash` does not require
| root account, unlike `sudo apt install`, which can be very
| convenient. Like all things: Multiple options are generally
| better.
|
| [0] https://github.com/Rust-GCC/gccrs
| fmajid wrote:
| Right, so you basically have to replay nearly the entire
| history of Rust versions since 1.54 (that's what, 6 or 7
| stages?) to bootstrap. Compare this to Go, where there is a
| stable version of Go 1.4 for bootstrapping, and the current
| Go 1.18 compiles on it, as opposed to only the previous
| version of Rust in rustc, so 2 stages, or Zig which can be
| bootstrapped in a single phase, I believe. That is what I
| call lack of stability.
| kouteiheika wrote:
| ...and why should I care that it's somewhat more
| inconvenient to bootstrap the compiler from scratch? No,
| seriously, why? What I care about is that the code I wrote
| on Rust 1.0 still compiles on Rust 1.60. And I _do_ still
| have code from back then (I started writing Rust just
| before 1.0 hit) and I can confirm that it still compiles.
|
| Yes, I know it sucks for all of the distribution
| maintainers who want to bootstrap every package from
| scratch, and I do feel for them, but that's a very niche
| thing to do which the vast majority of people will _not_
| do.
| kldx wrote:
| What do you mean by different C++ versions can't mix?
| Language, standard library implemention or something else?
| tialaramex wrote:
| Try it. Here's a completely reasonable line of C++ 17 code:
| int concept = 4;
|
| Now here's a completely reasonable line of C++ 20 code:
| template <class T> concept delicious = true;
|
| Huh. You can't have those in the same project because C++
| 20 believes you can't name a variable "concept" and C++ 17
| believes "concept" is an identifier and so you can't write
| your delicious concept template.
|
| These are both valid C++, it's just that they aren't
| simultaneously valid in any of the half dozen distinct
| versions of standard C++.
| kldx wrote:
| Thanks for the example. Is it considered UB to compile
| the first example with -std=c++17, the second one with
| c++2a and link them together?
| zozbot234 wrote:
| C fixes this by using _Concept as the keyword and
| sticking a #define concept _Concept in a separate
| stdconcept.h header file. Or maybe it was about complex
| or whatever, but same deal.
| parentheses wrote:
| Rust is in an interesting position, where like C or C++, it's
| capable of solving very low-level problems. At the same time,
| it's got a lot of "modern" language features.
|
| The kernel being literally the "bottom" of a system presents a
| challenge as this space rejects complexity and tooling variety
| that is intrinsic to "higher-level" languages.
| samhw wrote:
| Could you say in concrete terms what you think is not going to
| work? This just feels like a jumble of spatial metaphors. I
| don't see why it matters whether Rust or C are 'high' or 'low'
| level languages, or how that relates to the kernel being at the
| 'bottom' of the operating system.
| oxff wrote:
| I don't think people should be looking at adding another language
| to the kernel, mostly because every OS build on it has been a
| disaster. We just need approach and definition for operating
| systems.
| samhw wrote:
| > I don't think people should be looking at adding another
| language to the kernel, mostly because every OS build on it has
| been a disaster. We just need approach and definition for
| operating systems.
|
| What does "we need approach and definition for operating
| systems" _actually mean_? (If you know, that is.)
|
| Also, it's the Linux kernel, not just _the kernel_. There are
| more than one, and every operating system has one[0]. You might
| be interested in looking at the Mach kernel (Apple[1]) or
| Google 's new Zircon kernel (in Fuchsia).
|
| Both of those are microkernels, and as such minimise the amount
| of work that the kernel does, as well as removing drivers to
| userspace. Inasmuch as I can extract any meaning from your
| comment, it seems like your problem might be with the size and
| scope of Linux's kernel, in which case those might appeal to
| you.
|
| [0] I'm sure this absolute claim will summon someone to point
| out some recondite 1980s operating system that doesn't have a
| kernel.
|
| [1] Yes, yes, _used by_ Apple.
| jrimbault wrote:
| > The kernel has to stand alone
|
| Why ? (un-opionated "why")
|
| To reuse the migrant melting pot metaphor used at the end of the
| article, the "kernel has to stand alone" seems to mirror the
| states wanting a "strong border" (while the idea of borders and
| states has a beginning and will surely end).
| humanrebar wrote:
| It makes it easier to get a kernel working on a bare bones
| system for one. If things get complex enough, step one of
| getting Linux working on a new architecture will always involve
| getting a cross compiler working.
| rocqua wrote:
| I can imagine there are plenty of cases where people build a
| linux kernel that was moved to an air-gaped server over
| sneaker-net. Or someone who wants to build linux on some SoC
| that doesn't even really have an internet connection.
|
| I think in your view in melting pot metaphore 'Kernel has to
| stand alone' might be a lot like christian countries holding
| the view "Sunday is a forced day of because it is the day of
| the lord". Whereas I think it might be more like e.g. the
| Netherlands having the cultural idea of "we should actually
| work together to prevent the entire country flooding". Or maybe
| the Spanish "taking a break during midday is smart, not lazy".
| Its a cultural preference that arises as an adaption to the
| specific geographic reality of the country.
| BlueTemplar wrote:
| Using this metaphor, wouldn't the kernel be more akin to a
| military base ?
| geijoenr wrote:
| I believe Rust will benefit from the reality check that kernel
| development represents.
|
| Kernel development is hard, and bullshit doesn't go very far in
| that context. Success for Rust in that environment (with some
| changes along the way) will be a proof of value.
| hardwaregeek wrote:
| You have a valid point although I wouldn't frame it as
| adversarial as much as mutually beneficial. There will
| certainly be some bullshit eliminated from Rust, but I would
| not be surprised if there is a similar quantity eliminated from
| the kernel. Even in the most scrutinized C codebase in the
| world, there are likely memory usage bugs. If Rust can find and
| eliminate them, while also improving its capabilities, we all
| benefit.
| GolDDranks wrote:
| I think Rust is a good position in the sense that the language
| and community has a track record and culture of going and
| solving problems instead of sitting on them forever. I agree
| that many challenges of kernel development are going to end up
| strengthening and evolving Rust as a language.
| [deleted]
| pjmlp wrote:
| Solving async traits, having a proper concurrency runtime
| story, reducing the reliance on third party crates to ease
| error handling does seem to take out forever.
| scns wrote:
| I like your input, as critical as it is. Rust is developed
| by a community/foundation in contrast to C# (Java+). Anders
| Heilsberg, a brilliant language designer/BDFE can ponder
| and decide, after careful consideration, top down.
| Consensus takes time.
| ragnese wrote:
| > Solving async traits, having a proper concurrency runtime
| story
|
| Something something let the Java guy cast the first stone.
|
| > reducing the reliance on third party crates to ease error
| handling does seem to take out forever.
|
| I've written a ton of Rust and I've never used any of these
| third party error convenience libraries. As far as I'm
| concerned there is no issue in need of solving around error
| handling in Rust.
| bobbylarrybobby wrote:
| The lack of context is a pain. For instance, trying to
| open a file that doesn't exist just gives the error "file
| not found" without telling you the path you tried to
| open.
| GolDDranks wrote:
| While I agree that these are problems (and, they are being
| addressed), none of these problems are ones that have to do
| with Rust in Kernel. The developers actually keep a
| wishlist of potential or unstable
| Rust/libstd/libcore/tooling features they'd consider
| helpful: https://github.com/Rust-for-
| Linux/linux/labels/prio%3A%20met...
| FridgeSeal wrote:
| > Solving async traits
|
| Pretty sure async traits are coming soon (next couple of
| versions?) which is pretty speedy considering what I
| understand to be a semi-thorny problem.
|
| > having a proper concurrency runtime story
|
| Do you mean the language/stdlib shipping an async runtime?
|
| > reducing the reliance on third party crates to ease error
| handling
|
| I for one don't rely on 3rd party crates for error handling
| often? Anyhow is the most common one I use, but mostly out
| of habit and laziness, not actual hard requirement....
| the_duke wrote:
| > Pretty sure async traits are coming soon (next couple
| of versions?)
|
| No. There are ideas for how to do it, but as far as I
| know they haven't been tested out yet.
|
| It also relies on pretty large features that are not
| proposed for stabilization yet (GAT and existential
| types). When that is done and the implementation strategy
| is chosen there will also need to be a RFC cycle, a phase
| of ironing out bugs and finally stabilization.
|
| It'll be ... quite ... a while... I can't see it
| happening this year.
| tialaramex wrote:
| GATs are apparently very close to stabilization, I think
| I've seen Rust 1.62 suggested as plausible. So that's a
| big step towards the likely design of async traits. But
| sure, async traits is not likely to be this year.
| Fortunately although for some reason async traits are on
| pjmlp's must-have list, they're nowhere close to the
| critical path for Linux, which again is written today in
| C.
| mcronce wrote:
| I usually use `thiserror` for errors, but all that is is
| a shortcut macro for `impl Error` and `impl Display`
| jupp0r wrote:
| Async error handling has been a nightmare last time I
| tried a few months ago. Has that improved lately?
| lijogdfljk wrote:
| Pardon my ignorance, but what do you mean? Handling
| errors with the async runtime..?
|
| If you just mean handling errors from funcs .. it's no
| different than non-async, no?
| throwaway82652 wrote:
| >Kernel development is hard, and bullshit doesn't go very far
| in that context.
|
| I don't know what it is about Linux that makes people say this.
| Kernel development has most of the same constraints as any
| other embedded context, which Rust has plenty of focus on. No
| it's not as mature as C, but few languages are.
|
| Plus if you go looking in the kernel you can still find plenty
| of bullshit hacky code. It's not special, it's just another
| random open source software. The quality very much depends on
| the individual maintainer of that subsystem and how much has
| been invested in that area.
| noobermin wrote:
| It's ironic that the framing of some parts in the article is
| that the kernel mindset is arrogance and self-assuredness and
| that somehow isn't applied to the rust developers approaching
| an ecosystem that isn't familiar. It reminds me of tourists who
| visit another country and try to get locals to do things the
| way they are familiar with as if something is wrong with the
| locals and their way of life. I generally agree with gkh's
| response here. It avoids the arrogance in the other way ("why
| can't these kids roll their own leftpad") and presents a more
| valid concern, that of the kernel's actual constraints.
| oscargrouch wrote:
| Your example is quite funny and also represents how i would
| approach the matter of becoming a linux kernel developer.
|
| Even if C is not my primary language of choice, i would
| definitely try adapt myself to the ecosystem and not the
| other way around.
|
| You have all the knowledge of other peers, manual, books, all
| the libraries, the whole ecosystem.. this cant be replaced.
|
| Also there's something else about C nowadays, is the lingua-
| franca, the latim (or english) of programming languages. We
| use it to expose api's to others in any other language that
| want to consume it as a library.
|
| There's something about culture that people often forget in
| tech.. it's the real backbone of any project that it's on its
| own feet.. and when you want to enter in a community you will
| be better of if you learn and adapt yourself into this
| community culture instead of creating cultural clashes into
| the community and try to overtake it (be it hostile or not).
|
| People should be aware that this effort will make it possible
| to create rust-based kernel drivers and that's it. the RIIR
| folks are delusional and hype fueled and its better if the
| sane Rust community get away from them or start to get them
| back into reality as i bet they are not willing to expend 10
| or 15 years of their lives rewriting big and complex piece of
| software for a likely no return as people will tend to keep
| using the software the have more community and that are
| stronger.
|
| It's a much better approach for Rust or any other programming
| language to become research darlings and eventually become
| the primary ecosystem of a research OS that went well and is
| the thing that will replace Linux. The language alone wont do
| it, it must be able to be a contender to UNIX and POSIX, and
| whatever language that is in such a system will probably be
| the one that will become the dominant one in such a
| ecosystem.
|
| Also another good approach is to virtualize the Linux Api
| like gvisor does is userspace or the fuchsia OS(and even
| FreeBSD) does in the kernelspace. So that you can create your
| OS and kernel in the best way you can looking ahead, and have
| this Linux compat layer where applications dont even need to
| be aware they are not actually running in Linux.
| kaba0 wrote:
| You are fighting an imaginary opponent. The one proposing
| it is a long-time kernel contributor.
| oscargrouch wrote:
| > You are fighting an imaginary opponent.
|
| Yes, i'm sure there's an imaginary opponent downvoting my
| comment.
|
| Also there's a lot into my comment, and people are not
| even noticing it in the whole context.
|
| If there's no reason, why people get so upset? just move
| on if you are not being mentioned, as it will clearly be
| the case if i'm talking about "imaginary opponents"..
|
| > The one proposing it is a long-time kernel contributor.
|
| I'm not saying anything about people working on it
| specifically, if you read my comment, there's a clear
| separation between serious people and the hype crowd
| (which is not just RIIR, but now also cryptocurrency
| fellows, etc).. i can't say where the people working on
| this fits, i don't know them. Don't know from which part
| of my comment you took that conclusion.
|
| There are quiet, clever, serious people doing the work,
| like Hoare, Matsakis and the people that are real
| enginners, i have all the respect for them (and im pretty
| sure Rust have tons of such a people). To be fair, all
| languages have all kinds of people but i don't know what
| happen to some of them that tend to attract a certain
| type of people more than others, like the feeling a got
| from Haskell community more often than others (but given
| the community was much smaller)..
|
| For instance talking about culture, C succeeded exactly
| because it was a pragmatic language very simple and
| efficient like their founders to get things done. With
| this culture, things happened to be done around the
| language and we have the ecosystem we have today.. it's a
| great hacker spirit of more humble, hard-working, behind-
| the-cameras sort of people which i sincerely miss in the
| days of instagram, tick-tock and tech celebritism.
| GolDDranks wrote:
| Just a note, I am an avid long-time Rust user and
| contributor, but I have never met these "RIIR folks". They
| seem to exist only in anecdotes.
| oscargrouch wrote:
| it just took me to scroll down on this same thread to
| find one sample
|
| > * 11 hours ago | parent | prev | next [-] > Rust will
| quickly replace C in the kernel, I have no doubt about it
|
| I've have seen tons of such a comment in all sort of
| products when the matter is discussed around here and
| elsewhere (twitter, reddit, you name it)
|
| If you want to really get serious about this, i can feed
| this comment sections with tons of evidence over the
| course of the years.
| biorach wrote:
| Anyone who believes that Rust will quickly replace C in
| the kernel clearly knows very little about Rust or the
| kernel, and definitely should not be taken as a
| spokesperson for either.
|
| I suspect that this "RIIR" that you seem to believe is
| some kind of "movement" is just a random assortment of
| clueless people posting in random places.
| samth wrote:
| The article is by Jonathan Corbet, who has been a kernel
| insider for a very very long time, and is generally a strong
| proponent of the way Linux does things as a project.
| lmkg wrote:
| The person who is leading this project and making it happen,
| Miguel Ojeda, is a long-term kernel developer. Whatever your
| experience might be of other RIIR evangelism, this project
| initiative is coming from an insider rather than an outsider.
| oxff wrote:
| The issue with Kernel development is that it's a dying
| profession. Nobody is interested in it. That's why there's an
| attempt with sexy new language, to bring more developers to
| work on it.
|
| Probably less than 30 people in the world understand the thing
| as a holistic entity. From evolutionary terms, there's a giant
| risk of losing relevant technical knowledge to keep it up in
| the future.
|
| I do not support adding it to the Kernel, I think we should
| just throw away the kernel entirely, but I understand why
| they're looking at Rust.
| jerf wrote:
| The kernel isn't dying, it's niche. It always has been and it
| always will be.
|
| Fortunately for the kernel, despite being niche, it has a
| rock-solid onramp forcing people to get into it. There will
| always be companies interested in the n'th degree of
| performance, both generally, and for some specific hardware.
| _Someone_ has to go do the relevant kernel work for those
| things. So while you or I may never touch it, it is
| effectively impossible for a kernel like Linux to just rot
| away because nobody cares. It would require first a multi-
| year, if not multi- _decade_ process of fading first.
| biorach wrote:
| nice?
|
| It's on most of the smartphones on the planet. Plus it is
| the dominant server OS. Plus in the top 3 in many embedded
| categories (a highly diverse set of technologies)
| teddyh wrote:
| > _It 's on most of the smartphones on the planet._
|
| Sadly enough, probably not for long:
| https://en.wikipedia.org/wiki/Fuchsia_(operating_system)
| samhw wrote:
| Haha, I heard another good one about a nun going into a
| bar..
| teddyh wrote:
| Even if Fuchsia is crapware, Google has the money to keep
| polishing that ball of mud, and eventually force it as
| the new version of Android, quality be damned. The rest
| of the world will have no recourse but to switch. It will
| be painful for quite a while, but people will survive. I
| mean, people survive using _Windows_ , and not enough
| people switch away from that, either.
|
| I'm not saying that Fuchsia is necessarily bad, I'm
| saying that Google will do _anything_ to get away from
| GPL code, including, if necessary, forcing Android to
| Fuchsia. It doesn't actually matter if Fuchsia is any
| good.
| throwaway82652 wrote:
| If Linux developers are actually afraid of that, then
| they should simply switch the license away from the GPL,
| or dual license it, or do anything else than what they're
| currently doing.
| samhw wrote:
| From what I can see, the Fuchsia _kernel_ is actually
| quite interesting. I like the foci on (1) capabilities
| and (2) message passing. It 's not the most innovative
| thing in the known universe - in fact both of those
| concepts are of pretty late-80s-to-early-90s vintage,
| from the OOP boom when programmers were misspending their
| ill-gotten performance gains[0] - but they make a degree
| of sense. The _userspace bits_ I 'm less sure about. Like
| you say, it seems to be a non-GPL-ed clone of Linux. It's
| the kind of thing I'd expect of some cheap Chinese
| company. This kind of fragmentation is emphatically not a
| good thing for our industry and Google knows it, and I
| very much hope they don't get away with it, but I suspect
| its being a clone is exactly why it'll be a very easy
| transition to force on end-users. Programmers will never
| in a million years use it on the server side, though.
|
| [0] https://en.wikipedia.org/wiki/Andy_and_Bill%27s_law
| jerf wrote:
| In context, it should be clear that kernel _development_
| is a niche.
| pjmlp wrote:
| Android Linux !== Linux.
| camgunz wrote:
| I think you're talking about users and they're talking
| about engineers.
| unmole wrote:
| > That's why there's an attempt with sexy new language, to
| bring more developers to work on it.
|
| At no point did any active kernel developer express any such
| sentiment. Nobody is worried about not being unable to
| attract new blood.
| pohl wrote:
| It's only spoken of in knowing glances, like Voldemort.
| biorach wrote:
| > The issue with Kernel development is that it's a dying
| profession. Nobody is interested in it.
|
| That's a pretty wild assertion and files in the face of the
| highly active kernel development process.
|
| e.g. > Linus has released 5.18-rc1 and closed the merge
| window for the 5.18 release... 13,207 non-merge changesets
| were merged during this merge window.
|
| https://lwn.net/Articles/890119/
|
| > I think we should just throw away the kernel entirely
|
| That's the dumbest thing I've see on HN in some months. The
| kernel is deployed in hundreds of millions of devices
| worldwide and continues to be the dominant OS in many many
| sectors.
| staunch wrote:
| > _The kernel is deployed in hundreds of millions of
| devices..._
|
| There are _billions_ of Android phones alone. Not to
| mention the huge numbers of servers, IoT devices, embedded
| computers, and all the SBCs in my closet.
| pjmlp wrote:
| That Linux kernel is full of Google specifics, uses a
| microkernel like architecture since Project Treble, can
| be compiled since years with clang, now uses Rust for
| Bluetooth drivers,...
| samhw wrote:
| To be charitable, they may have meant something more like
| 'personal devices which people use directly'. (Though
| maybe not - it would be odd to exclude servers, which are
| one of Linux's biggest 'clients'. Perhaps they meant to
| exclude the more mundane types: routers, lightbulbs,
| etc.)
| oxff wrote:
| The issue is about the # of new developers joining the
| development if you look at the Linux kernel development as
| an organization, not about # of non-merge changesets
| merged.
| matheusmoreira wrote:
| > The issue with Kernel development is that it's a dying
| profession. Nobody is interested in it.
|
| That's simply false. I'd love to work on the kernel. I have
| immense respect for the people who work on it. Only reason I
| haven't tried contributing code is I don't think I'm skilled
| enough.
| sophacles wrote:
| Oh no. We have some early career devs that put up patches to
| the kernel recently. They were super excited about getting to
| do that work, and it was a big day for them - as it should be
| it's awesome.
|
| I guess I need to go let them know they are Nobodies. oxff
| said so.
| capableweb wrote:
| > Probably less than 30 people in the world understand the
| thing as a holistic entity
|
| As we only have two other "big" (popular) kernels to compare
| to this one, do you think they (Apple and Microsoft) have
| more people "holistically" understanding the entire kernel or
| less? Since the nature of those companies are closed-source,
| I'm fairly certain even less people understand those ones
| "holistically".
|
| > I do not support adding it to the Kernel, I think we should
| just throw away the kernel entirely, but I understand why
| they're looking at Rust.
|
| How would that work in reality? Re-use the existing tests to
| build a new kernel from scratch? Sounds like a very far-out
| idea that wouldn't help with any of the current problems, but
| I'm happy to entertain the idea and hear your reasoning here.
| ImprobableTruth wrote:
| Since they're complaining about maintainability, I assume
| they're advocating for microkernels?
| yjftsjthsd-h wrote:
| > How would that work in reality? Re-use the existing tests
| to build a new kernel from scratch? Sounds like a very far-
| out idea that wouldn't help with any of the current
| problems, but I'm happy to entertain the idea and hear your
| reasoning here.
|
| While I would tend to agree that a full production
| replacement would be such a massive undertaking as to be
| impractical, https://github.com/nuta/kerla does something
| very like that - Linux userspace ABI on an all-new Rust
| kernel. (And even at this small scale, I find it mind-
| blowing that this worked)
| LeFantome wrote:
| There may not be many "big and professional" operating system
| projects but, at the hobby level, it seems that there is a
| lot of interest in kernel dev actually.
|
| HaikuOS, SerenityOS, Redox, and ReactOS are all going strong.
| The BSDs continue to advance as well. Redox is written in
| Rust even.
|
| I believe Google sees Fuschia as a true Linux competitor.
|
| When you say "we should just throw away the kernel entirely",
| what are you suggesting?
| zozbot234 wrote:
| AIUI, you can still build a minimal kernel that's easily
| understood as a whole. And patches to shrink the minimal
| build even further are highly sought after because they
| expand the usability of Linux in deeply embedded
| environments.
| umanwizard wrote:
| > I do not support adding it to the Kernel, I think we should
| just throw away the kernel entirely
|
| What would you use instead?
| npigrounet wrote:
| Rust will quickly replace C in the kernel, I have no doubt
| about it.
| Bancakes wrote:
| If by "replace" you mean "nobodies rewrite existing crap into
| Rust", then yes.
| LoveMortuus wrote:
| That's a bit harsh of an approach. And I think at least
| Doom would disagree.
| nomoreusernames wrote:
| he means in the context of a kernel developer. im sure
| there is some nerd who would rewrite stuff to prove to
| themselves and shutup their inner imposter syndrome. but
| mostly its quite accurate.
| mhaberl wrote:
| > rewrite stuff to .. shutup their inner imposter
| syndrome
|
| you think that might do it?
| Bancakes wrote:
| Rust is evidently an impostor language.
| samhw wrote:
| I agree that the parent claim ("Rust will quickly replace
| C in the Linux kernel") was utterly risible, but your
| comments just seem like the mirror image of theirs. What
| on earth is an 'impostor language'? Imposture of what or
| whom?
|
| I'm tired of having to deal with this culture war crap in
| our profession. These languages are tools. C, C++, and
| Rust all compile down to the same LLVM IR (or GCC if you
| stray from rustc). There are certainly semantic and
| grammatical peculiarities that affect how each of them do
| so[0], but _by and large_ running a simple Rust program
| and a simple C program through Godbolt will do a lot to
| disabuse you of the idea that the two are irreconcilably
| different.
|
| To anyone else who wants to write performant Rust, my
| advice: (1) no_std, if only to focus the mind, (2)
| .try_foo(), not .foo(), b/c allocation is fallible, (3)
| _always_ set `opt-level` to at least 1 (1 is far further
| from 0 than 3 is from 1, ime), (4) use stack-allocated
| alternatives to heap-allocated types ( 'smallvec' or
| equiv vs Vec, 'smolstr' vs String, &c) even at the
| expense of overallocating buffer space, (5) exploit
| vectorisation where possible (e.g. SIMD), in general
| practising mechanical sympathy, and (5) parallelism is
| not a panacea, whereas cache locality usually is. Measure
| everything, but also: memorise every instruction and how
| many cycles it takes, and think in those terms - in terms
| of your assembled, perhaps-handmodified code - rather
| than unscientific laptop benchmarks. (Jeff Dean's famous
| 'numbers every programmer should know' are a good start
| but are just the very basics, and obv his exact _values_
| are long obsolete, in some areas [disk] more than others
| [CPU].)
|
| [0] These are discussed extremely soberly and
| intelligently here, for you or anyone else who may be
| interested: https://kornel.ski/rust-c-speed
| lostmsu wrote:
| I have a very strong doubt about it because Rust debugging
| still sucks. No debugger allows you to evaluate function
| calls AFAIK, which is a very strong restriction.
| umanwizard wrote:
| I have definitely evaluated function calls in rust-gdb.
| toast0 wrote:
| That doesn't seem like it would matter in the Linux kernel,
| since they don't have a kernel debugger, for better or
| worse.
| nukemaster wrote:
| saagarjha wrote:
| Running GDB against the kernel running in QEMU?
| kibwen wrote:
| Rust is great, but let's not get ahead of ourselves. :P
| ilovecaching wrote:
| I work on the kernel for a living, and I find this claim
| exceedingly dubious. We're currently talking about
| experimentally supporting modules written in Rust, which is
| an entirely different beast than replacing pieces of the
| kernel core. The barrier to entry for drivers is
| significantly lower, and driver quality can be much, much
| poorer than the quality of the core kernel. Many parts of the
| kernel have been fine tuned for decades, and many of the
| kernel developers that maintain Linux are also C experts
| (myself included) who aren't going to slow down development
| to migrate working code to Rust. It's great that we can
| experiment and see how Rust goes for driver authors, but they
| are still bus API consumers, not core kernel.
| MYEUHD wrote:
| >I work on the kernel for a living
|
| I'm interested in kernel development and I like the idea of
| working on it for a living. Can you give more details about
| your job? What does it consist in? Is it mostly code-
| review? Or are you responsible for maintaining a part of
| the kernel. Who is the entity that pays you, and what are
| the criteria they'd use to pay a new contributor to work on
| kernel full-time? Finally, can you point me to beginner-
| friendly things to work on to get started? how do I know
| which part of the kernel I should study and contribute to?
| tialaramex wrote:
| As I understand it the crucial rationale for drivers is
| that drivers were anyway necessarily platform dependent
| which undoes one argument against Rust.
|
| Today Rust does not overlap Linux in terms of platform
| support. There are (small but very much alive) communities
| doing Linux on architectures that Rust has no support for
| and in some cases has no plans ever to support. So this
| makes drivers the only case where choosing Rust doesn't
| mean some people lose out, as a platform e.g. with no PCI
| bus doesn't get to run PCI drivers even if they were
| written in C.
|
| I expect that over the next say, five to ten years, two
| things will happen to greatly improve this, maybe to the
| point where you absolutely could rewrite core Linux code in
| Rust if you wanted to. Firstly, Rust will get more platform
| support. Linux doesn't really need Rust's "Tier 1" (Linus
| doesn't check every kernel release passes tests on all real
| Linux target hardware as I understand it) but clearly you
| want Rust to at least build and take patches for every
| Linux platform some day. Secondly, some older platforms
| will "rust out". If your community is nursing 30+ year old
| hardware and increasingly more maintenance work is shared
| between fewer shoulders at some point "Linux-next" is not a
| priority and your platform will stop being supported while
| effort moves to exciting new hardware.
| rrdharan wrote:
| > Today Rust does not overlap Linux in terms of platform
| support.
|
| Nit, I believe actually the Rust and Linux platform sets
| would be considered "overlapping sets" in the
| mathematical sense :), since neither is a subset of the
| other.
|
| e.g. Rust platforms include things like the NetBSD Rump
| Kernel and Redox and I think one would be hard-pressed to
| claim that Linux supports those as platforms.
|
| https://doc.xuwenliang.com/docs/rust/1423
| sophacles wrote:
| There's active work being done on the rust gcc backend
| and it's progressing nicely. That should help with some
| of the platform concerns you (rightly) raised.
| matheusmoreira wrote:
| I have no doubt we'll be seeing Rust kernel modules but
| quickly replacing C? I don't think that's realistic.
| nightfly wrote:
| If you define quickly as "over the next 10-20 years, maybe",
| then yes
| yjftsjthsd-h wrote:
| In fairness, that _is_... if not _quick_ , then not slow
| either in kernel development time:)
| rob74 wrote:
| Just as quickly as it replaced C/C++ in Firefox?
| zozbot234 wrote:
| Firefox is getting new "oxidized" component all the time,
| Rust is the recommended language for both refactoring and
| new development. Of course the lowest-hanging fruits are
| addressed first, but that's normal and advisable.
| pjmlp wrote:
| Not after they let go everyone related to Rust, as far as
| I am aware.
|
| Hence the crazy idea of now using WebAssembly in Firefox
| modules instead.
| steveklabnik wrote:
| That's not correct. Firefox continues to gain new Rust
| code. Compare https://web.archive.org/web/20201109025230/
| https://4e6.githu..., the first version of this page
| archive.org saved after the layoffs. 9.5% Rust, 2.9
| million LOC. Today https://4e6.github.io/firefox-lang-
| stats/ 10.1% Rust, 3.4 million LOC.
| tialaramex wrote:
| There's a big difference between "We don't employ the
| language's architects" (so far as I'm aware Mozilla also
| doesn't employ many WG21 members) and "We don't have any
| engineers who know this language". In 2022 you'd probably
| have to go out of your way to hire that many engineers
| and _not_ get some people who know Rust.
|
| Not to mention how much less experience you need in Rust
| to not blow everybody's feet off by mistake. I reckon if
| you have 10 years C++ and six months Rust, any Rust you
| write is already more likely to deliver reasonable
| performance without setting everything on fire than your
| C++. Because of the constant exposure to outright
| malevolent stinking garbage (in the form of other
| people's HTML, CSS and Javascript) the browser needs to
| be exceptionally robust, and C++ just isn't very good for
| that. So Rust is often a better fit for what Mozilla do.
| devmunchies wrote:
| I'm new to programming, whats Firefox?
| colonwqbang wrote:
| The kernel today has its own complete standard library of data
| structures and threading primitives for C. In general I find
| these much more well-designed and pleasant to use than the user-
| space C standard library. Give me work queues and kfifos over
| pthreads any time!
|
| Maybe the rust devs can also learn something worthwhile from
| working with the kernel community.
| klysm wrote:
| I think this also aligns with one of the comments in the
| article that suggests we should pursue a more unified kernel
| space and user space.
| nullc wrote:
| I can't think of anything that could more effectively discredit
| the effort to use rust in the Linux kernel than suggesting using
| crates.io/cargo -- I had to check the dates just to make sure
| this wasn't a delayed April first post.
|
| But I guess some people just really want the unicode ukrainian
| flags being appended to every email they send or something and
| are too lazy to implement it themselves. :P
| OskarS wrote:
| Indeed. The comment cited in the article is pretty stunning:
|
| > Other crates that I'd like: anyhow, bincode, byteorder, log,
| once_cell, pin-project, rand, serde, slab, static_assertions,
| uuid plus some more esoteric ones.
|
| This makes Rust devs look infantile, with no real understanding
| of what it means to develop for the kernel. Or what it takes to
| develop security and performance critical software of any kind,
| frankly.
| varajelle wrote:
| Can you please elaborate? These crates are no_std and provide
| functionality which seems useful regardless of on the kernel
| or not.
|
| I don't understand why it looks bad from a performance or
| security point of view. These crate are well established and
| developed by competant devs that also care about security and
| performance.
| duckerude wrote:
| anyhow is for careless error propagation, where you know
| you won't do elaborate handling and just bubble it up to
| the user. It's suitable for (some) applications, but not
| for libraries, and a kernel falls on the "library" side of
| that divide. Errors should be handled by the kernel itself
| or reported carefully to userspace. anyhow is not built for
| that. (It also puts its errors on the heap--what do you do
| if memory allocation fails when reporting an error?)
|
| Linux has its own UUID infrastructure, so it seems weird to
| pull in a whole parallel implementation. At the very least
| you'd want to disable all the generation code, and it's
| unclear to me if the uuid crate supports that.
|
| rand is large. Does Linux need all the probability
| distributions it supports? (ripgrepping for "Bernoulli"
| only turns up the name of a floppy disk system, so probably
| not.) Doesn't it already have an implementation for the
| ones it does need?
| estebank wrote:
| > It's suitable for (some) applications, but not for
| libraries, and a kernel falls on the "library" side of
| that divide.
|
| I would disagree with that assertion: the kernel is a
| service that end user applications interact with through
| an API. Using anyhow in a Rust library that Rust
| applications use _is_ a very bad idea. Using anyhow in
| the kernel _might_ be a bad idea, but not for _that_
| stated reason.
|
| > Linux has its own UUID infrastructure, so it seems
| weird to pull in a whole parallel implementation. At the
| very least you'd want to disable all the generation code,
| and it's unclear to me if the uuid crate supports that.
|
| The uuid crate can be modified to support the kernel
| infrastructure as its backend. This would allow the same
| API to be available within the kernel and in the rest of
| the ecosystem. That's beneficial to everyone involved.
| duckerude wrote:
| > the kernel is a service that end user applications
| interact with through an API.
|
| My thinking is that that API has to distinguish between
| many strictly defined error codes, while anyhow
| homogenizes errors and focuses on human-readable
| information. How would you tell whether an anyhow::Error
| means EINVAL or EFAULT?
|
| It's the same fundamental design tradeoff that makes it
| unsuitable for Rust libraries, even if anyhow wouldn't be
| visible in the public API.
|
| > The uuid crate can be modified to support the kernel
| infrastructure as its backend
|
| Fair enough, using a modified version seems reasonable.
| roca wrote:
| Actually all of those crates could potentially be implemented
| in the kernel and be useful. The current implementations
| wouldn't work as-is, in many cases, but the APIs make sense
| and the crate could use conditional compilation to work in
| the kernel.
|
| The most far-fetched crate there would be pin-project I
| think, because a lot of other work would have to be done to
| make Rust async code work in the kernel.
| Arnavion wrote:
| >The most far-fetched crate there would be pin-project I
| think, because a lot of other work would have to be done to
| make Rust async code work in the kernel.
|
| Pin's usefulness is not limited to async. Any type that
| contains a pointer relative to itself benefits from being
| wrapped in it to avoid accidental misuse.
| baq wrote:
| > static_assertions
|
| infantile?
|
| i mean, serde in the kernel is overkill, but not every crate
| is like that. Rust has been designed in the internet era and
| leverages dependencies for things older languages would ship
| with (rand!)
| sanderjd wrote:
| That one is fine, but most of the rest of them seem
| inappropriate to me.
| Hendrikto wrote:
| > Rust has been designed in the internet era and leverages
| dependencies for things older languages would ship with
| (rand!)
|
| Weird justification for missing an extremely central and
| extraordinarily important feature...
|
| How does rand being a crate make Rust "internet era" in any
| way?
| orf wrote:
| > Weird justification for missing an extremely central
| and extraordinarily important feature
|
| Can you think of any reason why decoupling it from the
| language itself would be a good idea? Why does it _need_
| to be baked in, and what problems would arise if it was?
| How do these problems trade off against having people
| just run "cargo add rand"?
| zozbot234 wrote:
| Most Rust "crates" are compile-time convenience features that
| add comparatively little binary code to the final build. The
| requests are plenty reasonable from that POV, although
| "vendoring" the deps might still introduce some complexity.
| xorvoid wrote:
| Same thought. Seriously, what a great way to trigger some
| senior kernel dev to come down and straight veto rust with some
| rant about how it's "worse C++"
|
| Rust is very slowly winning (but it's a really hard space and
| this is impressive). But some people have to come and try
| snatching defeat from the jaws of victory.
|
| (sigh)
| ratorx wrote:
| Why does a single bad idea discredit an entire language being
| included?
|
| If someone suggested including a C dependency manager in the
| kernel, does that mean that the use of the entire C language in
| kernel should be discredited?
|
| It is fair to say that the developer suggesting it (especially
| considering some of the proposed crates they want) has some
| massive misconceptions about the constraints of current Linux
| kernel development though.
| sanderjd wrote:
| Yeah, my stance on this is that it probably makes sense to
| pull some libraries into the tree that would be useful in
| kernel development and which have comprehensible dependency
| trees such that everything could be reviewed and audited. But
| then I saw the quoted list and thought "oh this doesn't seem
| like a very serious proposal".
| IYasha wrote:
| So, I assume soon there'll be a pure-C kernel fork and some
| distributions adopting it. And I'd switch to that.
| orf wrote:
| Why?
| IYasha wrote:
| Because there are many people who oppose mandatory Rust
| integration. For various reasons, not just plain dislike of
| Rust itself. And, for example, there's a Libre Kernel that is
| blob free and it has its users. There might be an audience
| for Rustless Kernel too.
| samhw wrote:
| I think they meant "why?" as in "what reason do they
| have?", rather than "do there exist people who think
| that?". I don't really care what tool was used to generate
| my assembly, and the kind of Rust that will be written in a
| kernel context[0] will - I guarantee you - produce
| virtually identical assembly to C.
|
| [0] i.e. no standard library ('no_std'), no unwinding
| panics, no dynamically sized types, &c.
| bitcharmer wrote:
| It might, and if so you will switch but a very small minority
| of Linux users ever have the need to compile it themselves and
| out of that small minority only a small fraction will migrate
| to alternatives over this.
|
| I see Rust as a replacement for C long term. From this
| perspective it's absolutely ok to have a transition period
| (years in fact) where there are multiple toolchains used to
| build the kernel.
| scoutt wrote:
| Building the kernel shouldn't require an internet connection.
| treekogreen wrote:
| Bit of a weak argument as you require and internet conneciton
| to pull the source.
| berkut wrote:
| And you can then stick that on a network drive or scp it on
| internal airgapped networks without direct internet access.
| varajelle wrote:
| And you can do the same with the Rust dependencies (with
| cargo-vendor for example) Cargo has even an --offline flag.
| IYasha wrote:
| Many people download once, use constantly and on many
| machines. My dev PC was never online since it was put
| together, all patching and updating was performed offline.
| All builds were bit-to-bit reproducible.
| easytiger wrote:
| In certain environments i have extensively worked in, the
| machines on which one builds are only allowed internet access
| on an ip:port basis after months long process involving
| dozens of people in multiple teams.
|
| Working offline is needed
| goodpoint wrote:
| Not at all. Pulling sources, reviewing and vetting them and
| finally building are completely different steps.
|
| Linux package maintainers do the vetting. Buildbots build in
| a clean room environment, without Internet access.
|
| If you mix up the steps supply chain security is gone.
| quickthrower2 wrote:
| Says who? (inserts CDROM...)
| jcelerier wrote:
| I can go to my nearest journal kiosk and I'm pretty confident
| I can find a paper Linux magazine which comes a cd with
| recent kernel source
| dunefox wrote:
| > I can go to my nearest journal kiosk and I'm pretty
| confident I can find a paper Linux magazine which comes a
| cd with recent kernel source
|
| This has to be one of the most insane threads I've read in
| a while.
| dageshi wrote:
| Do paper linux magazines actually exist any more?
|
| 10-15 years ago I'd agree with you, now... that would
| actually surprise me.
| pjmlp wrote:
| Here you go, https://www.linux-magazin.de/
| baq wrote:
| you have a cd reader...?
| BlueTemplar wrote:
| This is not particularly relevant, as it could also be an
| USB stick or a SD card.
|
| (But why wouldn't you have readers for all 3, potentially
| as peripherals if you don't have a desktop ?)
| dunefox wrote:
| The last CD or DVD I had was over ten years ago. I just
| have no use for them.
| BlueTemplar wrote:
| And you don't have any friends & family that regularly
| come for help ?
|
| When I built my current PC, I didn't bother with a 3.5"
| floppy drive, but I still had to get an USB one a couple
| of years ago when an acquaintance showed up needing to
| read some files from them...
| dunefox wrote:
| I do but they don't use them either.
| IYasha wrote:
| ...and you don't? I can lend you mine :3
| conradludgate wrote:
| If that's the case, you can do this with cargo vendor, as
| many other threads have mentioned
| dboreham wrote:
| After using your time machine to go back to 2010.
| jcelerier wrote:
| Just went to the first one I could find and there are 4
| different Linux magazines lol, here's a pic:
| https://ibb.co/djscJ94
| shp0ngle wrote:
| I am not sure if you are even serious
| pjmlp wrote:
| https://www.linux-magazin.de/
| orf wrote:
| In all likelihood, no you couldn't. Because nobody buys
| those anymore.
| pjmlp wrote:
| Funny, here in Europe there are plenty of those on sale,
| so apparently people do still buy them.
| poulpy123 wrote:
| Not in my European country at least. I haven't see a
| magazine with CD-ROM in years. And most people don't have
| a cd-dom reader anymore
| pjmlp wrote:
| Here are a couple for you,
|
| https://www.linux-magazin.de
|
| https://www.linuxformat.com/
| orf wrote:
| The argument isn't that they don't exist or are not
| published, it's that they are not published _widely_.
|
| Given that the circulation for Linuxformat was just
| 19,000 in 2014, it seems that that number backs up the
| position that you wouldn't find this at a street level
| vendor.
| pjmlp wrote:
| I would call most shopping mall surfaces and train
| station newstands, street level.
| orf wrote:
| And what do you suppose is the ratio between train
| stations + shopping malls vs actual street level tobacco
| and magazine stands/shops is in most European countries?
| 1 to 700? 1 to 1000?
|
| Any way, the whole argument is redundant. Go head to your
| nearest newsagent and see if they stock it. If you live
| on top of a train station or a supermarket then you're
| due both congratulations and commiserations.
|
| For everyone else... you can't buy them at your nearest
| kiosk.
| pjmlp wrote:
| Plenty of people can buy them, even at my tiny town back
| home, that is a little more than "everyone else".
| orf wrote:
| That wasn't my point at all. They are specialist
| magazines stocked my particular places.
|
| Because it's not 2007, no matter how hard you wish it to
| be.
| orf wrote:
| Street level magazine kiosks sell magazines with up to
| date Linux kernel CD roms? That's a lie. I've literally
| never seen that anywhere in Europe in the last decade.
| You might be able to find them in specialist shops or
| larger supermarkets, as you can in the UK, but street
| level kiosks? Get real.
| jcelerier wrote:
| I'm in france, I regularly buy one of those when I take
| the train lol.
|
| We have quite a bit of choice and they all have recent-
| ish issues:
| https://www.journaux.fr/linux_informatique_1_0_130.html
| and you can find at least a couple of them in most kiosks
| ; at least Linux Identity always comes with a physical
| disk
| pjmlp wrote:
| A DVD on a Linux magazine will do.
| sanderjd wrote:
| Why is this such a trope in this thread? It's such a
| superficial strawman. Obviously cargo can copy all the
| dependency sources into a project's own tree if that's the
| right fit for a given project, and it almost definitely would
| be for the kernel.
|
| There are lots of good arguments against pulling in lots of
| rust dependencies - it's too much code to audit, maintained by
| too many different upstream people and teams, lots more tooling
| would be needed to make sure all the dependencies are safe to
| use in the kernel, and on and on - but this one about an
| internet connection is both the most common one I've seen here
| and the most superficial and frankly silly.
| mst wrote:
| The article itself explicitly mentioning 'cargo vendor'
| (which, indeed, seems like the only logical choice for the
| kernel, just like they already do with zstd as also mentioned
| in the article) leaves me extremely confused as to why people
| keep jumping to assuming non-vendored crate usage.
| sanderjd wrote:
| It seems like people don't know that it's possible? But
| obviously it's possible to pull dependency source code into
| the tree... The only question is whether or not there is
| tooling support for it. It could only possibly be a bonus
| to have something like cargo that can automate that
| workflow, rather than being forced to do it manually.
|
| So I agree, this particular criticism is very confusing.
| Though as I said, that doesn't mean no criticism is
| warranted!
| bitcharmer wrote:
| It's possible to vendor all deps and have them embedded in the
| kernel source package.
| kibwen wrote:
| Building Rust code, and even using Cargo, doesn't require an
| internet connection.
| jvanderbot wrote:
| ah, yes it does? 'cargo build' absolutely goes and finds
| dependencies, updates them, and so on.
|
| 'rustc' does not, but we are talking about cargo here.
| miloignis wrote:
| It doesn't have to, you can "cargo vendor" all your deps (I
| do so for some of the things I work on, regularly building
| and developing without an internet connection).
| jvanderbot wrote:
| OK, I have been sufficiently reminded of a mostly-offline
| build process and punished for my flippant remark which I
| can no longer edit or delete.
|
| Thank you!
| KptMarchewa wrote:
| https://doc.rust-lang.org/cargo/commands/cargo-vendor.html
| [deleted]
| estebank wrote:
| https://doc.rust-lang.org/cargo/faq.html#how-can-cargo-
| work-...
| Bessie69 wrote:
| gquery wrote:
| zozbot234 wrote:
| > Some of them have been directly involved in cancelling non-
| native English speakers over pronouns.
|
| This is the kind of claim that could do with a source. In
| general, it seems Rust is rewarding enough and complex enough
| to master that people just don't pay much attention to the
| silliest kinds of 'activism' compared to other dev communities.
| [deleted]
| enriquto wrote:
| Rust is alright as a language, but cargo is extremely scary. I
| hope a culture of coding in rust outside of the cargo "ecosystem"
| develops. The current situation is alienating to reproducibility-
| oriented developers.
| Ar-Curunir wrote:
| I hope that never ever happens. We don't need a C-like mess
| where every library has a bespoke build process that doesn't
| work across OSes.
| enriquto wrote:
| > where every library has a bespoke build process doesn't
| work across OSes
|
| cargo sounds pretty much like a new bespoke build process
| that doesn't even work across different languages on the same
| OS
| sanderjd wrote:
| Yes. It turns out this works much better than the situation
| previously. The bespoke-per language model turns out to
| have fewer drawbacks than the bespoke-per-library-and-OS
| model. Note: _fewer_ drawbacks, not _none_. I would not
| have predicted this a priori, for what it 's worth, and I
| was very critical of and first one of these I became
| familiar with (rvm), but it turns out to work well in
| practice.
|
| I still think cross-language efforts (like bazel and I'm
| sure there are others, maybe nix?) seem generally better,
| but I suspect there is some fairly good reason they are
| less widely used.
| howinteresting wrote:
| The problem with bazel and nix is that they are
| totalizing build systems. They want _everything_ to bend
| to their worldview, and ask for an all-in commitment.
| Cargo etc require somewhat less of a commitment.
| sanderjd wrote:
| Right, but it seems like it would probably make more
| sense to pick a totalizing system, if you know you'll
| have a polyglot environment, which seems pretty
| inevitable for any business (as opposed to just like a
| side project). But I think there are probably good
| reasons this isn't the most common way to do things, and
| I just don't know them.
| howinteresting wrote:
| Many businesses do pick totalizing systems -- many
| medium-size or above corporations use bazel, and a few
| use nix. There's also some tooling to convert Cargo.toml
| to bazel and nix build files, though I don't know how
| well that works.
|
| I think the problem is that bazel and nix are just not
| that easy to get started with. If you're using them
| you're likely to have a team (or at least one person)
| working full-time on them, since bending everything to a
| singular worldview involves a lot of work.
| sanderjd wrote:
| Good to know!
| blm126 wrote:
| The reason is that defaults are powerful. Languages tend
| to have an owner in a position to declare a default for a
| language. No one is in a position to declare a default
| for polygot environments.
| sanderjd wrote:
| That seems like a good way to think of it.
| proto_lambda wrote:
| OP seems to be conflating the concepts of cargo (the rust
| build system with dependency management functionality) and
| crates.io (the main and default package repository today).
|
| It is totally possible to use cargo without ever touching
| crates.io, but if you need any dependencies you'll of course
| have to provide them through some other means (local file
| system, git repositories, or a custom package repository).
| estebank wrote:
| I could totally see the Linux Kernel setting up their own
| alternative Rust package mirror that cargo uses: you get
| the ease of use of cargo while also getting the desired
| level of curation. It could either be treated as owned
| packages, only projects written by kernel contributors, or
| it could be the linux distro packager model, where blessed
| crates at a point in time are curated, but not necessarily
| audited beyond some smoke tests.
| mort96 wrote:
| Cargo defaults to downloading stuff from crates.io. I
| certainly wish cargo was just a build system which doesn't
| care where you get your packages from, but that's not what
| it is.
| steveklabnik wrote:
| Cargo does not care where you get your packages from, but
| it does have a default place. If you don't like it, it is
| easy to change.
| mort96 wrote:
| That... is caring where I get packages from. Something
| like make or cmake doesn't care at all how packages end
| up on my drive. Almost everyone uses cargo in a way which
| makes cargo automatically download them from the web.
| steveklabnik wrote:
| If you're making that argument, you could easily turn it
| around: make or cmake do care where your pacakges are;
| they by default only build things on your local drive.
|
| The point is: all options are available for all systems,
| so suggesting any workflow doesn't work with any of these
| is just incorrect.
| mort96 wrote:
| My point is that building code coming from my local drive
| is much less problematic than automatically downloading
| the code from some random website.
|
| And I never said any workflow doesn't _work_ with Cargo.
| I said I wished it didn 't download code (by default) and
| that it didn't care how code ended up on my drive, that
| it just worked like a normal build system. That you _can_
| use Cargo in that way doesn 't help.
| nyoomboom wrote:
| How does cargo prevent reproducible builds?
| enriquto wrote:
| The default behavior of cargo is to download stuff from the
| internet. This may be the least reproducible thing ever.
|
| I'm honestly astonished that programmers of a language that
| is deemed to be "safe by default" thought that this behavior
| was acceptable in any form, not to say the default. If
| downloading things at build time is somehow necessary, it
| should be an obscure option behind a flag with a scary name,
| like --extremely-unsafe-i-know-what-i-am-doing, that prompted
| the user with a small turing test every time that it is run.
| Cargo is just bonkers, it doesn't matter at all if it is
| "convenient" or not. Convenience before basic safety and
| reproducibility is contrary to the spirit of the language
| itself.
|
| It's as if bounds checking in the language was deferred to a
| third party that you need to "trust" in order to believe that
| you won't have segmentation faults.
| [deleted]
| easytiger wrote:
| > The default behavior of cargo is to download stuff from
| the internet. This may be the least reproducible thing
| ever.
|
| Wait till you find out about java ecosystems
|
| I know investment bank dev teams pulling _whatever they
| need_ from maven central with no oversight or
| introspection.
| DrBazza wrote:
| Log4j joins the conversation.
| easytiger wrote:
| yea i'm glad i wasn't around when that was discovered.
| KronisLV wrote:
| > The default behavior of cargo is to download stuff from
| the internet.
|
| This is borderline inevitable for most modern development
| stacks, though .lock files can definitely help, even adding
| hashes to check against if you care about your dependencies
| being the same as when you first download/add them to the
| project and/or inspect the code.
|
| As for worries about the things in those URLs disappearing,
| in most cases you should be using a proxy repository of
| some sort, which i've seen leveraged often in enterprise
| environments - something like JFrog Artifactory or Sonatype
| Nexus, with repositories either globally, or on a per-
| project basis.
|
| The problem here is that _all_ of these repositories kind
| of suck and that the ecosystem around them also does:
| - for example, Nexus routinely fails to remove all of the
| proxied container images and their blobs that are older
| than a certain date, bloating disk space usage - when
| proxying npm, Nexus needs additional reverse proxy
| configuration, since URL encoded slashes aren't typically
| allowed - many popular formats, like Composer (or
| plenty more niche ones) are only community supported
| https://help.sonatype.com/repomanager3/nexus-repository-
| administration/formats (nobody will ever cover *all* of the
| formats you need, unless you limit yourself to very popular
| stacks) - many of the tech stacks that have .lock
| files may also include URLs to the registry/repository from
| which they're acquired, so some patching might be necessary
| - in technologies like Ruby, actually setting up the proxy
| isn't as easy as running something like "bundle install
| --registry=..." as it is in npm - in other
| technologies, like Java, you get into the whole SNAPSHOT vs
| RELEASE issue and even setting up publishing your own
| packages to something like Nexus can be a bit of work; the
| lack of proper code libraries for reuse and abundance of
| code being copy-pasted that i've been being a proof of this
| in my mind
|
| Of course, i'm mentioning various tech stacks here and i
| don't doubt that in the long term Rust and other
| technologies might also address their own individual
| shortcomings, but my point is that dependency management is
| just a hard problem in general.
|
| So, for most people the approach that they'll take is to
| just install stuff from the Internet that other people
| trust and just hope that the toolchain works as expected, a
| black box of sorts. I've seen plenty of people just adding
| packages without auditing 100% of the source code which
| seems like the inevitable reality when you're just trying
| to build some software with time/resource constraints.
| thecrm wrote:
| It doesn't just download random things. Cargo generates a
| Cargo.lock file with checksums and will make sure that
| those checksums match when building later on. It's about as
| safe as vendoring all dependencies while being far easier
| to work with (though tools like cargo-vendor do exist, of
| course).
|
| Edit: for things like the kernel, vendoring dependencies is
| still probably not a bad idea, of course
| humanrebar wrote:
| What prevents a given URL from disappearing? Does that
| just break a particular source version of the Linux
| kernel?
|
| What happens when a given dependency adds new kernel-
| inappropriate features? Are kernel devs going to act like
| distro maintainers and decide between forking,
| maintaining patch sets, etc.?
| sanderjd wrote:
| To the first question, obviously the sources of
| dependencies would be brought into the tree. This is easy
| and there's no reason I'm aware of not to do it for
| something like the Linux kernel.
|
| To the second set of questions, how is this any different
| than any other dependency the kernel has? If the answer
| is "the kernel has no dependencies" then yeah, I'm very
| sympathetic to the argument that bringing in rust
| libraries is not a good reason to start having
| dependencies when none previously existed at all, but is
| that the case?
| roca wrote:
| All crate sources are stored in the crates.io package
| archive, which never deletes packages.
|
| A dependency veering off in a direction you don't like is
| one of the risks of using someone else's code instead of
| writing it yourself. Cargo makes it easy to use forked
| dependencies, and forking a dependency is almost always
| less work than if you'd never used it and written the
| code yourself from the beginning. (And to be clear this
| is only a problem for future evolution; a crate author
| cannot remove or modify an already-published version of
| their crate.)
| humanrebar wrote:
| "Never" is a long time, just saying. It'll be impossible
| to beat the "availability" guarantees of a local mirror
| (like a thumb drive) of a kernel source tarball.
|
| What happens when a crate version has to be removed due
| to a critical CVE or court order (IP Law violation,
| perhaps)? There may come a day where crates.io becomes
| torn between not breaking Linux source and not hosting
| actively _bad_ source code.
|
| Note that some of those concerns do apply to vendoring
| source as well, but the additional download step also
| removes options that the kernel maintainers have as long
| as they ship all the source for the kernel in one
| tarball. Like more control over the timing of inevitable
| decisions.
| mcherm wrote:
| What happens today when a kernel module has to be removed
| due to a critical CVE or court order?
|
| That's not just a rhetorical flourish, I'm actually
| curious what the answer is. As far as I know, (1) it
| almost never happens and (2) when it does, the change is
| made in upstream repos and as a practical matter,
| everyone downloads those changes and their up-to-date
| local copies lose that code.
| humanrebar wrote:
| Fixing it in the future isn't the point. Breaking
| previous releases is.
|
| The previous tarballs still work and contain the relevant
| code. Your build wouldn't rely on hosts complying with
| court orders in countries you might not live in.
|
| If the code isn't vendored, just referenced with URLs,
| the old tarballs stop working.
| notriddle wrote:
| > What happens when a crate version has to be removed due
| to a critical CVE or court order (IP Law violation,
| perhaps)?
|
| CVE = The Yank flag. Cargo will refuse to add new yanked
| packages to a lock file, but if a yanked package is
| already in the lock file, it will still build. The
| package is not actually deleted. https://doc.rust-
| lang.org/cargo/commands/cargo-yank.html
|
| Legal = Hard delete. Nobody will go to jail just to avoid
| breaking your build. Of course, since crates.io and
| kernel.org are in the same legal jurisdiction, is there
| any actual difference here?
| 3836293648 wrote:
| Does crates.io actively host any code? I thought it was
| all just readmes and links to github and docs.rs
| conradludgate wrote:
| They do host it. It's registry info is mirrored on github
| though
| marginalia_nu wrote:
| This is still fairly short sighted. Websites shut down,
| large websites with big storage demands are especially
| vulnerable to attrition. Who wants to pay the mounting
| bill for keeping decades of revisions of historical rust
| packages online?
|
| I can grab the kernel sources from 1997 and build them
| today. Will I be able to build rust code from 2022 in
| 2047, because the 1997 kernel will still build at that
| date.
| varajelle wrote:
| > I can grab the kernel sources from 1997 and build them
| today.
|
| Can you? Do they still compiler with current compiler?
| You'll probably need to find a compiler of that time...
| And also all the interpreter for all the build scripts.
| Was that using bash or some old Perl? Maybe something
| more esoteric like m4 or tcl?
|
| The point is that it always had many external
| dependencies to bootstrap. And adding one is not such a
| big deal, it just add another thing to archive among the
| many other things. The crates.io archive is probably not
| even that big.
| marginalia_nu wrote:
| I'm not sure why that would be a problem given most of
| these languages and standards are older than the Linux
| kernel. The thing about mature technology is specifically
| that it doesn't have breaking changes every couple of
| months. This is the way it used to be for a fairly long
| time.
|
| But even if it has broken, I can just download an old
| linux distro. They effectively form a _cohesive_ snapshot
| of the state of the toolchain whenever they were
| assembled. Slackware 3.1 from 1996 might be appropriate.
| estebank wrote:
| > But even if it has broken, I can just download an old
| linux distro. They effectively form a cohesive snapshot
| of the state of the toolchain whenever they were
| assembled. Slackware 3.1 from 1996 might be appropriate.
|
| You will also need era-appropriate hardware to get that
| software to install.
| pdw wrote:
| "I can grab the kernel sources from 1997 and build them
| today."
|
| Where would you be grabbing it from? ...From a website?
| "Websites shut down, large websites with big storage
| demands are especially vulnerable to attrition. Who wants
| to pay the mounting bill for keeping decades of revisions
| of historical Linux kernels online?"
| marginalia_nu wrote:
| From <https://mirrors.edge.kernel.org/pub/linux/kernel/>?
|
| It's easy enough to keep your own website up as long as
| you want to, the liability is other projects and
| services, especially when the scope of those services is
| "archive everything for everyone forever".
| sophacles wrote:
| So your argument is you think the people who run the
| crates site don't want to do a good job but the people
| running kernel.org do? What info are you basing this
| random-seeming decision on? Do you have any actual data
| suggesting that the crates site will just disappear like
| you say?
|
| I'd like to see that data if so -- I have pretty big
| doubts that your statement has merit without some sort of
| evidence.
| marginalia_nu wrote:
| As I said in a parallel comment, there is a fairly large
| difference between archiving your own project's history
| for as long as you feel like, and archiving the complete
| history of every significant piece of code ever written
| in a particular programming language forever.
|
| Kernel.org's repository is also of major versions, not
| every minor release and patch. That really wouldn't do
| for cargo. If it has ever been released, it needs to be
| kept in storage for as long as the rust ecosystem exists.
| That's decades, maybe even centuries of passing on the
| torch and hoping the next guy accepts the responsibility.
| Hoping you can find a next guy.
| sophacles wrote:
| So vendor in the dependencies. it's a matter of cloning
| the dependencies repo and adding a handful of characters
| to your project's cargo file.
|
| Now the lifetime of the dependency is that of your
| project. There's even tooling 'cargo-vendor' to help
| manage this setup.
|
| Alternative of course is implementing it all yourself,
| which cargo doesn't prevent.
| humanrebar wrote:
| You make a copy, store it on your medium if choice, and
| put it in a filing cabinet. I gather that certain
| organizations use magnetic tape backups for especially
| important data. For some organizations and individuals,
| kernel source code could be that important.
| kibwen wrote:
| You can easily do this with Rust and Cargo as well.
| marginalia_nu wrote:
| There is a fairly large difference between archiving your
| own project's history for as long as you feel like, and
| archiving the complete history of every significant piece
| of code ever written in a particular programming language
| forever.
| kibwen wrote:
| Who claims that archiving the complete history of every
| significant piece of code ever written in Rust is
| necessary? It is easy to archive only the code that your
| project depends upon. Rust code is no different from C
| code in this regard.
| estebank wrote:
| A couple of things addressing points from different part
| in this thread:
|
| - Archiving the complete history of all crates in
| crates.io is perfectly feasible today for an individual.
| Over time that might change.
|
| - Setting up a mirror is straightforward, should you want
| to do so: https://github.com/rust-
| lang/crates.io/blob/master/docs/MIRR...
|
| - crates.io is financed by the Rust Foundation and is at
| no risk of disappearing, it is a very well funded effort.
|
| - Using cargo with an alternative repo is not difficult,
| requires some one-time configuration.
|
| - Vendoring your dependencies is supported.
|
| - cargo hits the network to look for semver compatible
| updated versions of your dependencies on specific moments
| if you don't have a Cargo.lock file.
|
| - _Not_ updating your dependencies stops you from getting
| the rug pulled from under you if an unwanted change
| happens, but it also stops you from getting any desired
| changes _including security vulnerability fixes_.
|
| - Even if you vendor all of your dependencies, you
| _still_ have to audit them the first time and every time
| you update them. Are you? Most aren 't. Code you haven't
| written yourself can't be assured not to be malicious,
| and code you've written yourself can still have
| exploitable mistakes.
| scns wrote:
| I'd rather comment than downvote. Who cares about about a
| kernel build from 1997 (25 years ago)? What was the
| hardware back then, Pentium 2? Sorry for the snark in
| advance but: Why make mountains out of molehills? Life is
| hard enough as it is.
| marginalia_nu wrote:
| You may not own a Pentium 2, but someone might. This is
| only hard if you make it hard. My that an old Linux
| kernel, by design, _can_ be built today. This is a
| feature it has for free, a consequence of not relying on
| flimsy network based dependency managers.
|
| At any rate, we are indebted to the future to preserve
| the present, as our past has been preserved for our
| benefit.
| Gwypaas wrote:
| I would imagine the kernel would use cargo vendor, or
| similar, to lock all dependencies into their chosen
| source control and quality requirements.
|
| https://doc.rust-lang.org/cargo/commands/cargo-
| vendor.html
| goodpoint wrote:
| > It doesn't just download random things.
|
| That's exactly what it does. The developer is not really
| expected to thoroughly review the codebase of every
| dependency.
|
| Just like javascript, all sort of supply chain attacks
| are made possible.
|
| A single malicious library can sneak into large
| ecosystems easily.
| yw3410 wrote:
| You're forgetting about custom build scripts. Thankfully
| most of the core ones have moved off cloning dependencies
| for ffi purposes (think cloning an alsa-lib version for
| ffi), but it used to be super common.
| CraigJPerry wrote:
| The lock file is created but is not used by default.
|
| You must specify --locked to get that behaviour
| duckerude wrote:
| That's true when running `cargo install` to install an
| application directly from crates.io, but not when running
| `cargo build` in an already checked-out repository.
| CraigJPerry wrote:
| I might be misreading this on an iphone screen but as i
| follow the logic here: https://github.com/rust-
| lang/cargo/blob/a77ed9ba87bfeaf3c275...
|
| A cargo build ends up there calling into the resolver's
| resolve_ws_with_opts() which would refresh the lockfile.
|
| Not resolve_with_previous() which would use the lock file
| as-is.
|
| The only reason this sticks in my mind is i ran into an
| issue building bat after i made some changes, i obviously
| assumed it was my changes so went through the process of
| debugging and backing out my changes until finally i was
| back to a virgin branch and still failing - passing
| --frozen --locked fixed it.
| heftig wrote:
| No, it is. Even without `--locked`, the Cargo.lock file
| is only updated when it no longer fulfills the Cargo.toml
| because the latter was edited (and then only making the
| minimal changes necessary), or explicitly using `cargo
| update`.
| CraigJPerry wrote:
| I don't follow - I'm saying the cargo.lock isn't read
| unless you specify --locked - I'm not talking about when
| it gets refreshed?
| heftig wrote:
| Yes, it's always read. If the file didn't require
| updating, a build with and without `--locked` will be
| identical. If it did require updating, `--locked` will
| make cargo exit with an error.
| CraigJPerry wrote:
| Gotcha, makes sense
| [deleted]
| roca wrote:
| If your project has a Cargo.lock file checked into its
| repo, then everyone checking that out will download _the
| same code_ for all dependencies (unless someone manages to
| compromise the crates.io package archive). That is very far
| from "the least reproducible thing ever".
| danieldk wrote:
| Cargo.lock also contains crate hashes. So, if someone
| compromises crates.io and tampers with a crate, you would
| notice.
| j-krieger wrote:
| I'd _really_ like to know where you think C++ dependencies
| and headers come from.
| humanrebar wrote:
| Downloading C++ dependencies during the build process is
| equally unacceptable for many situations. Existing C++
| build systems and package managers can be configured to
| do that and those build systems and package managers
| would be inappropriate for supporting a kernel that
| values stability and long term support.
| sophacles wrote:
| So it's a good thing that cargo can be used without
| downloading dependencies during the build! Just clone the
| repos of the dependencies (and transitive dependencies),
| just like you would for a C++ project. Then set up your
| cargo file to point at the location for your local copy
| instead of using the default download behavior.
|
| There's even a tool called cargo-vendor that does this
| for you!
| humanrebar wrote:
| I was just saying anything folks might find objectionable
| about cargo workflows happens in C and C++ workflows,
| just in ad hoc ways.
|
| The difference is that no C or C++ package management
| features are proposed for incorporation in the Linux
| kernel SDLC.
| CraigJPerry wrote:
| Is reproducability achiveable?
|
| If you preserved an immutable tag of the source code and all
| its dependencies, a copy of the compiler version used and all
| build flags, then you've still got some big holes in your
| ability to reproduce a binary: 1. OS version &
| patches installed 2. OS configuration 3. Hardware
| used (processors can have weird subtle bugs, microcode can
| affect execution behaviour, etc etc) 4. Transient issues
| - the golden copy to be reproduced for some post event
| investigation might have contained a bit flip leading to
| impossible to reproduce verification signatures
|
| Etc etc
|
| Or is reproducability just a spectrum and you try to get
| further along it with some careful attention to detail, rather
| than an absolute to be achieved?
|
| If the latter are you not cheaper just archiving binaries and
| tagging them with the source + deps + compiler + arch used to
| build them? Thats a 5 minute job to setup in your CI process
| and costs comparatively little to maintain vs wasting expensive
| human brains chasing down a futile goal.
| JonChesterfield wrote:
| It's normally taken as binary + transitive dependencies are
| bitwise identical on multiple builds. Needs a copy of a
| deterministic compiler along with your source. Whether the
| build runs the same on different machines is orthogonal to
| consistently rebuilding it.
| enriquto wrote:
| Reproducibility is certainly achievable, and an important
| point for some people.
|
| Look for example at guix, that bootstraps explicitly with
| what they call "the maxwell equations of software".
|
| Imposing a web-based build system, into the kernel of all
| places, is a kick to the face to people who care about
| reproducibility.
| sanderjd wrote:
| This would definitely be done by "vendor"-ing all the
| dependency sources into the tree, no? Unless that's not the
| proposal, which I can't imagine it isn't, then I don't see
| how "web-based" is relevant.
| [deleted]
| IshKebab wrote:
| Reproducibility is absolutely achievable. It is difficult
| with languages that have C system dependencies like C and
| Rust though. Usually you have to effectively check the
| compiler and system headers into your repo, which isn't
| ideal.
|
| For something like Go it's trivial though. If you're using
| pure Go code then reproducible builds are pretty much as
| simple as "compile using Go 1.xx".
|
| I don't understand why you think 3. and 4. are issues. Bit
| flips are very unlikely at compilation scale, and the
| hardware you use to compile something shouldn't affect the
| output. In either case you can just compile it twice and on
| different hardware and compare the result to confirm.
| fulafel wrote:
| For other repro curious readers, this seems like a good entry
| point to follow reproducibility efforts:
| https://github.com/rust-secure-code/wg/issues/28
| gspr wrote:
| I wholeheartedly agree. I _love_ Rust, it 's the most fun I've
| had with any programming language (barring perhaps Haskell).
| But I still run cargo in offline mode with crates.io disabled,
| pointing cargo to /usr/share/cargo/registry for dependencies
| (that's where Debian's librust-*-dev packages get installed,
| the only dependencies I accept in my Rust projects).
| maxerickson wrote:
| So you can change 2 settings to get the behavior you want?
|
| Sounds like they did a decent job anticipating that use case!
| gspr wrote:
| Absolutely (although see note below)! I don't have any
| gripes with the tooling from a technical perspective. I do
| have gripes from a cultural perspective, though.
|
| Addendum: Until quite recently, this was quite cumbersome.
| It also meant that _all_ cargo invocations (by the same
| user) would use that override, always. It meant that
| compiling someone else 's project became quite the hassle.
| Or compiling a project that _mostly_ uses system
| dependencies, but _some_ crates.io deps. But the situation
| is improving.
| steveklabnik wrote:
| To say it explicitly: "I don't have any internet" was a
| hard constraint on everything from the beginning. Firefox's
| build requires it. Most distros require it.
|
| Some of it was a bit awkward to actually use this way in
| the early days, but those harsh edges have been since
| sanded off.
| enriquto wrote:
| Let's just make "unsafe" the default in Rust, and add a new
| "safe" keyword to anticipate the use case of overly
| concerned users like gspr.
| pornel wrote:
| Cargo and crates.io have been specifically designed to be
| reliably reproducible, and had a chance to learn from npm's
| mistakes to not repeat them.
|
| Cargo binary projects have Cargo.lock by default with checksums
| of all dependencies used. crates.io doesn't allow changing past
| releases, and has a policy of not deleting any crates unless
| legally required (user-accesible "yank" hides, but doesn't
| delete). Crates.io index is a git repository with full history
| of all changes to the registry, so you can recreate its state
| at any point in time (in case you lost your Cargo.lock, you can
| reliably remake one from the past).
|
| And on top of that there's `cargo vendor` command that makes a
| local offline copy of everything you use, so you can fully
| archive a Cargo project and rebuild it any time later.
| nomoreusernames wrote:
| why do we have to have soooo many package managers? makes me
| cry inside a bit.
| pjmlp wrote:
| Because not everyone is using Linux distributions, and even
| those cannot agree in which system packaging they want to
| have.
| humanrebar wrote:
| Yeah, especially language specific ones. It seems like the
| world wants to be polyglot. Do we really want to have all the
| encryption algorithms reimplemented in all the languages?
|
| The answer there is often "use FFI". But if we're all going
| to use C APIs anyway, then shouldn't our package managers
| support C?
|
| But C has probably the most awkward build system culture of
| any language, complicating the job of packaging quite a bit.
| A lot of language specific package management systems get
| lift from offloading the ugly C support problems to other
| layers. See python and "manylinux" for instance.
| cillian64 wrote:
| If you particularly want to replicate the common C situation
| where your dependencies are in-tree or you just don't have
| external dependencies and write everything yourself then you
| are totally welcome to do that, even with Cargo. Nobody is
| forcing you to use external dependencies. If your problem is
| other people writing software with external dependencies, then
| I guess you have different priorities to them.
| nyanpasu64 wrote:
| My problem is dependencies taking on dependencies taking on
| hundreds of dependencies, making it more difficult to find a
| self-contained dependency which solves a problem, compared to
| C++ (outside of Boost or ffmpeg and such).
| cillian64 wrote:
| That's fair, and I hope we see crates being more
| considerate over what deps they pull in (or people making
| alternatives with minimal dependencies).
| HidyBush wrote:
| I already loathe the fact that I need Perl to compile the kernel
| just because people are too lazy to rewrite a few parsing scripts
| in C. The moment Rust is introduced in Linux without being a
| complete replacement is the day I leave for greener pastures
| [deleted]
| bitcharmer wrote:
| I agree on the Perl requirement bit but see Rust as a complete
| replacement for C or at least great alternative in systems
| engineering domain in the long term.
| HidyBush wrote:
| Please read again my comment. If tomorrow all the Linux C
| codebased switched to Rust I would have almost zero problems
| about it. My problem stems from having multiple toolchains to
| build the foundations of an operating system.
| bitcharmer wrote:
| The expectation for the switch to happen overnight in its
| entirety is very unrealistic.
| sophacles wrote:
| Why haven't you rewritten those parsing scripts you loathe so
| much?
|
| Since you haven't, it seems a bit ironic that your are
| complaining about those who didn't.
| caslon wrote:
| It's incredible, isn't it?
|
| Just as _web developers_ , of all people, are finally, after over
| a decade, coming to the conclusion that arbitrarily-packaged
| dependencies are a bad idea _for so many obvious reasons,_ Rust
| developers are trying to replicate NPM in the kernel.
|
| I was told that Rust in the kernel would be a good thing, and
| that not much would change. If these are the types of people who
| are writing Rust for the kernel... can we go back? I really want
| to go back.
| 12baad4db82 wrote:
| You could argue that there are some crates in the Rust
| ecosystem that suffer from a deep dependency tree, I don't
| think you can argue that Rust developers are trying to do this
| for the kernel.
|
| The article states:
|
| > The Rust-for-Linux developers understand this situation and
| are not envisioning adding the ability to pull in modules with
| a tool like Cargo
| nullc wrote:
| > You could argue that there are some crates in the Rust
| ecosystem that suffer from a deep dependency tree
|
| Some? Dependency graphs of well over a hundred packages are
| ubiquitous. I've long since stopped being surprised when I
| compile a rust package that makes no network connections and
| see it (indirectly) pulling in multiple HTTPS libraries.
| Grimburger wrote:
| > well over a hundred packages
|
| Tad understated. Popular rust http frameworks with basic
| functionality are hitting 300+ deps straight off the bat
| easily.
|
| For the security conscious, move to vendoring or alias
| cargo to always run in offline mode.
| 12baad4db82 wrote:
| Right, great example. I think 'understand the situation'
| refers to this. I imagine the Rust for Linux developers
| would be going through any dependencies they pull in with a
| fine comb, in this case a dependency that pulls in HTTPS
| libraries for presumably no reason should be rejected. If
| that particular dependency makes it into the kernel, well,
| then you can start complaining about it. But right now
| feels a bit premature.
| account-5 wrote:
| Not any sort of developer myself but if you're writing an OS
| kernel, and for what it's worth, my opinion is you should know
| the library's your code relies upon is safe. I have to agree
| with the part of the article taking about bloat and security
| problems.
| Starlevel001 wrote:
| > Just as web developers, of all people, are finally, after
| over a decade, coming to the conclusion that arbitrarily-
| packaged dependencies are a bad idea for so many obvious
| reasons, Rust developers are trying to replicate NPM in the
| kernel.
|
| Web developers and Rust developers are the same group
| proto_lambda wrote:
| I'm sure there are web developers that are also Rust
| developers and vice versa, but the vast majority of
| developers I know that belong to either of those groups don't
| belong to the other.
| pjmlp wrote:
| Given it was born at Mozilla, is getting traction at
| WebAssembly and WebGPU efforts, WGSL is loosely based on
| its syntax, and some Cloud Native projects are migrating
| from Go to Rust, I am quite sure there is some overlapping.
|
| Not that I agree with Rust's adoption at that level, as I
| think languages with automatic memory management make more
| sense, but that's me.
| kibwen wrote:
| What people are roundaboutly observing is that the vast
| majority of developers these days either begin their
| career or spend at least a part of their career as web
| developers. That Rust is such a successful onramp to
| systems development even for people who haven't been
| classically trained in it is one of the strengths of the
| language.
| pjmlp wrote:
| While it is good that such people are embracing Rust for
| such purposes, they would have had a similar high level
| experience with something like Modula-2 or Object Pascal.
|
| If anything, it is the pseudo-macro Assembler approach to
| C's design that made it a scary experience for some
| developers.
| sanderjd wrote:
| Browser developers are not web developers...
| pjmlp wrote:
| Silly me thinking browsers are related to the Web.
| sanderjd wrote:
| "related to" is doing a lot of work there. Browsers are
| "related to" the web, but surely you don't think Mozilla,
| if hiring for someone to work on the CSS layout engine or
| something for Firefox in c++ or rust, would advertise for
| a "web developer". It's just not the same job in any way.
| estebank wrote:
| Reminds me of https://pbs.twimg.com/media/DebEU7DXUAAJQp-
| ?format=jpg&name=...
| 2wrist wrote:
| Thats a really well written article, nicely balanced. I also
| agree, it can be great opportunity to learn.
| simion314 wrote:
| Still makes no sense to me to spend energy into pushing Rust into
| Linux instead of creating some new better kernels, I mean having
| some of this big companies paying a few experts to do such a
| kernel and keep compatibility with user applications. If Rust is
| much better then C and adding on top the fact you start fresh
| with no baggage you should get a better Linux and everyone would
| use it since is safer and faster especially on servers.
|
| I don't know of such a Rust kernel being worked on that is not
| some hobby side project but something more like Servo was, where
| you have paid experienced in domain developers.
| proto_lambda wrote:
| "starting fresh with no baggage" also means starting fresh with
| no drivers. Having drivers for a wide range of hardware is a
| major factor in operating system adoption.
| psychoslave wrote:
| Didn't FreeBSD incorporated an interface that allow to use
| Linux drivers?
|
| Not that I think it would make the point of having a bright
| new kernel easily compete with such a huge and firmly
| grounded project as Linux though.
|
| All the more, there already are multiple projects using Rust
| to build OSes out there, if I'm not mistaken.
| simion314 wrote:
| I was not suggesting this kernel to be used on laptops, it
| would be a server kernel, if it would actually be faster and
| safer then Google and Amazon will use it on their servers and
| they will make sure there are drivers, Google could use it in
| some mobile devices too if they care for safety. AMD has an
| open source driver so if desired that could be ported to Rust
| to also make sure the GPU is safe.
| xxpor wrote:
| Things work the opposite way. In order to get companies to
| adopt things you need to get people working their
| interested in it. Which means they need to be able to run
| it on their PCs. There's a reason why x86 ate the world.
|
| The exception to this is when you can develop something
| _and_ have a good case to use it yourself, like Google and
| Fuchsia. But I doubt Fuchsia will gain wide organic
| adoption because of the same factors.
| simion314 wrote:
| >Things work the opposite way. In order to get companies
| to adopt things you need to get people working their
| interested in it.
|
| This would mean that the only OSes are first created by
| hobbyists, then fanboys use it then a company adopts it.
| But there are commercial OSes and commercial software
| that are created by a company to solve a problem and not
| by random dudes and companies then adopts it.
|
| IMO is sad that we the tech industry are stuck with old
| OSes and old architecture while you have this giants that
| use open source software to make billions and they could
| find researching and creating of a few new revolutionary
| OSes like it happened in the past. And I don't mean
| desktop OS that will defeat Windows.
| xxpor wrote:
| Name a single successful (non-niche, stuff like INTEGRITY
| doesn't count) OS kernel created post-Windows NT 3.1
| that's commercial. Even Mac OS X still runs Mach as a
| kernel. The model is dead, because of the internet. OSes
| have MASSIVE network effects.
| Bancakes wrote:
| We simply don't have better kernels. POSIX is the epitome of
| infrastructure. That's it. We did it. It's like asking for
| something more than Turing completeness - we have everything,
| just optimise the existing code. To that end, introducing Rust
| is a pessimization.
| elihu wrote:
| The basic POSIX interface works, but it's hardly optimal. It
| has a lot of weird built-in assumptions that we're just so
| used to working around that it's hard to imagine it being any
| other way.
|
| As an example: why can a process only have one current
| working directory? Wouldn't it be nice to be able to have a
| process maintain pointers to two or more locations in the
| filesystem at once? Wouldn't it make software more modular if
| a library could "chdir" into some directory without worrying
| about breaking the application that depends on it? The
| filesystem APIs could be extended with a "CWD" handle
| argument that can be passed around sort of like a file
| descriptor instead of having one implicit CWD for the whole
| process.
|
| The same could be done with UIDs. Why not have processes that
| can use multiple UIDs? Again, you could have UID parameters
| to API functions that require authorization.
|
| POSIX is pretty good by the standards of 80's computers, but
| in a lot of ways it's showing its age. We can do better. But
| it's kind of depressing that OS interface design is treated
| as a solved problem, and so those interfaces stagnate.
| humanrebar wrote:
| It would be good if APIs like open and creat existed that
| took something safer than null-terminated strings.
|
| Even if kernel devs are careful to avoid the usual pitfalls,
| it's just a bad API that encourages use of null-terminated
| strings elsewhere. If you're using other, better string types
| elsewhere, you often need to make copies to then make system
| calls (the article hints at that with the addition of a
| CString type).
| Bancakes wrote:
| What's wrong with null termination? No, seriously. It's
| much cleaner than wasting an argument, hence CPU register,
| for a second pointer. Or heavens forbid, two size_t length
| and capacity counters.
| orf wrote:
| How do I pass a sub-slice of a source string to a
| function that expects a null terminated string without
| copying or modifying the source string?
| Bancakes wrote:
| I don't understand the question. Sub-slices are not sub-
| strings. They are partially constructed objects. But I do
| get the problem.
| orf wrote:
| The question is the problem: how do you do zero-copy
| sharing of data contained within a string. You can't
| unless you couple it with the length. In rust parlance a
| slice is a well defined type.
|
| There are also other reasons why having the length
| embedded in a string (or a string slice) is a good thing.
| You might want your "str_contains" function to do
| something different with different sized inputs: doing
| some vectorised lookup might only be worth it at a
| certain point, or if the length of the needle is greater
| than the haystack itself then there isn't much point
| doing anything.
|
| NULL terminated strings are a _huge_ mistake that brings
| in security issues, needless copying and inefficient code
| that might contain several redundant strlen calls at
| different levels.
| humanrebar wrote:
| Other than the usual arguments [1], I just said that it
| causes contamination of other APIs. That's a big problem
| because string is a "vocabulary type". It is passed
| around across various libraries within a given process
| constantly. There should be a lot of convergence on
| vocabulary types to avoid copying string data around
| incessantly.
|
| Null terminated strings are a poor choice to converge on
| for high level application code in particular. It's just
| an inefficient (expensive substrings, redundant length
| calculation, unclear ownership) and unsafe default for
| too many use cases.
|
| I don't mind certain performance sensitive applications
| using C strings to avoid extra registers, though even
| that is a premature optimization in certain situations.
|
| [1] See Limitations:
| https://en.m.wikipedia.org/wiki/Null-terminated_string
| pjmlp wrote:
| Except Cloud as OS doesn't care about POSIX at all, only type
| 1 hypervisors.
| scns wrote:
| Sorry for the snark in advance. Do you have an idea how many
| man years of work are in Linux? The estimated cost of
| developing it was 1.4 billion dollars in 2008.
|
| https://www.linuxfoundation.org/press-release/linux-foundati...
| dunefox wrote:
| That sounds pretty cheap actually.
| pferde wrote:
| Not that I do not have respect for Linux, but that doesn't
| sound like much in an age when we're reading about modern
| aristocracy buying up useless microblogging platforms for
| forty-plus billions. :)
| jvanderbot wrote:
| So... a patronized kernel from a wealthy kernel enthusiast?
| bombcar wrote:
| I think the point is that if the Linux kernel "costs" say
| 4 billion to make, that it's perfectly within the
| capabilities of a company _if_ they would see the value
| in doing so - which has been greatly reduced because
| Linux works so well for so many people and companies.
|
| But if Walmart needed a kernel for some reason, and it
| would provide value to them, they could afford it, it's
| about a quarter's worth of profit. But, again, Linux
| would likely do anything they needed, and if it was
| missing something they could add it.
| acomar wrote:
| the monetary cost is misleading. you can't take 4 billion
| dollars and use it to hire developers for a year and then
| expect something equivalent to the linux kernel after a
| year. money doesn't actually translate on its own into
| valuable things - people have to actually do work to make
| that possible. and not every task is infinitely divisible
| such that it can be worked on in parallel by many hands.
|
| the actual cost is in person-years of work, not dollars,
| and even then you can only speed up the time it takes so
| much by throwing money at the problem.
| pferde wrote:
| Why not, if it's released under a good open license?
| scns wrote:
| Elon Musk is not in the OS business.
| simion314 wrote:
| Sure , but that all those years of work are not of creating
| new stuff, there is a lot of work on just modifying existing
| code and not breaking stuff.
|
| You do not need to get all Linux features at start, I would
| focus on the server stuff, support the popular server
| hardware and popular server work so filesystems and
| networking. I would hope some competent developers would
| write a new kernel from scratch then getting 10% of some
| Linux rustiffied. When you edit shit you need to make sure
| not break shit, so you are limited, you move slow and you
| have to implement backwards compatibility.
| Amazon,Google,Facebook could work on this if they care for
| security but probably using VMs is cheaper for them.
| ilovecaching wrote:
| I don't think you understand the scope of what you're talking
| about. Linux is _tens of millions_ of lines of code, most of
| which is drivers that have to be written by vendors.
|
| Vendors aren't going to write drivers for your hobby kernel. No
| one is using your hobby kernel. Bootstrapping a new kernel
| without billions of dollars to invest in development time is
| almost impossible, and anyone who is investing billions of
| dollars is likely going to have dubious proprietary reasons for
| doing so.
|
| A successful kernel is Rust is probably the worst thing that
| could happen to the open source community.
| simion314 wrote:
| >I don't think you understand the scope of what you're
| talking about. Linux is tens of millions of lines of code,
| most of which is drivers that have to be written by vendors.
|
| But if you are Google and you believe that this Rust kernel
| is super safe and fast and has clean code, and
| parallelism,async and candy ... how many drivers do google
| Data centers use ?
|
| After you prove the kernel is real good by using it in data
| centers and devices that need security you can slowly expand,
| for most devices you could create soem compatibility layer.
| Who knows if there are soem competent developeres hired to
| work on it they might use some better architecture, like keep
| the drivers outside the kernel.
| qPM9l3XJrF wrote:
| Could a new kernel implement a compatibility layer of some
| kind for Linux drivers? Or just expose the same API?
| rcxdude wrote:
| Not very easily: Linux quite deliberately does not have any
| stable internal driver interface, so any such compatibility
| layer would have a very fast moving target to keep up with.
| throwaway15908 wrote:
| Tangetial question: Seen all the recent supply chain attacks, is
| there a way to defend FOSS projects with something like project
| defined capabilities for every third party dependency? So eg. you
| could include a math lib and mask everything regarding filesystem
| or network, etc? Especially for rust i would like to see a
| holisitic solution to this growing thread, facing the
| accelerating trend of central repositories.
| cillian64 wrote:
| This is more plausible for a language with a runtime like
| Python. But Rust is fundamentally designed to be a systems
| language where you have full access to everything (and, if you
| use unsafe, raw access to memory and arbitrary code execution).
| It's hard to imagine how you'd add a sandboxing layer to the
| language, it seems more like something the OS would have to do
| for you.
| throwaway15908 wrote:
| On a broad scope, you could solve this at compile time. The
| source simply does not compile when filesystem or networking
| crates/builtins are not defined.
|
| If you want to have more fine grained white listing, like
| only grant access to a certain directory, this could get
| really messy quick, trying to solve this at compile time.
| oconnor663 wrote:
| You'd probably have to start by banning unsafe code in
| general, but then whitelisting/allowlisting specific
| versions of specific crates that are allowed to use it, so
| that at least the most popular dependencies don't break.
| miloignis wrote:
| On the impl side, WebAssembly was built from the beginning to
| support that use case, and many languages are adding backend
| support for it. On the language side, I really like type
| systems with Algebraic Effects (koka-lang.org has some of the
| cool research into it).
| PoignardAzur wrote:
| There's currently talks about sandboxing proc macros, install
| scripts and build scripts.
|
| It would be nowhere near enough to make it safe to download
| untrusted dependencies, but it would at least stop the most
| trivial forms of attack.
|
| Sandboxed dependencies with user-specified capabilities are a
| very, very long way away, if they ever happen.
| goodpoint wrote:
| You use packages from a well known Linux distribution.
| ratorx wrote:
| That doesn't solve the problem, just moves the chain of
| trust.
| goodpoint wrote:
| Not at all. It creates a chain of trust instead of relying
| on random unknown github accounts.
|
| When was the last time a major distribution found a
| backdoor in a popular package?
| estebank wrote:
| > When was the last time a major distribution found a
| backdoor in a popular package?
|
| Packagers not finding a backdoor doesn't mean that there
| isn't one. How many packagers actively audit the code
| they support for a given distro? It is not uncommon for
| distros that support esoteric platforms will claim a
| given package works for that platform because it
| compiles, but it reliably segfaults on execution. Who's
| responsible for that? Packagers have even _introduced_
| [1] vulnerabilities by "fixing" code they didn't fully
| understand at the time.
|
| Packagers have a difficult, thankless task, and we're
| doing them no favors by being confused at what their job
| is. They ensure that the package builds, integrates with
| the rest of the distribution as much as possible and
| updates/patches swiftly when issues are found upstream.
|
| [1]: https://lwn.net/Articles/282038/
| mkj wrote:
| My rough understanding is that Caja for JavaScript kind of did
| that. Libraries didn't have access to the whole scope, could
| only access what they were given via capabilities.
| https://en.wikipedia.org/wiki/Caja_project
| ATsch wrote:
| The Pony language has this concept, but I personally found it
| mostly a pain to use.
| bawolff wrote:
| I feel like that is very hard to make work securely when the
| boundries are so ill defined. At the end of the day, untrusted
| code is untrusted. What happens when you use your math library
| to do some math on a security critical value?
___________________________________________________________________
(page generated 2022-04-15 23:02 UTC)