[HN Gopher] FreeBSD-rustdate, a reimplementation of FreeBSD-update
___________________________________________________________________
FreeBSD-rustdate, a reimplementation of FreeBSD-update
Author : ggm
Score : 96 points
Date : 2024-08-27 03:46 UTC (19 hours ago)
(HTM) web link (rustdate.over-yonder.net)
(TXT) w3m dump (rustdate.over-yonder.net)
| toast0 wrote:
| Thank you, thank you. I haven't reviewed this, and I may or may
| not use it, but wow was freebsd-update from 13.0 to 14.0 slow,
| especially since I run my freebsd systems off spinning drives,
| and most of them have src.
|
| I did some looking at freebsd-update, and it's quite the shell
| script, which makes it daunting to consider changing. All the
| forking and whatever doesn't seem great; using data structures
| should be a lot simpler!
|
| But my systems running freebsd-update spend an awful lot of time
| on I/O, and not so much time on cpu, as I recall. I think the
| general flow for a replaced file is gunzip to the update dir,
| then install to the destination. This means all the files are
| written twice, and I thought there was a third write that I can't
| remember at the moment. Install doesn't have an option to
| decompress, but it'd be super handy if it did --- then you could
| write the file only once to the destination directory and move it
| into place.
|
| I'm never a fan of quite as many files in a directory as freebsd-
| update gets up to either. Given that everything is already named
| by hashes, it's not too hard to have 16 or 256 directories named
| 0-f or 00-ff that get a slice of the files. Large directories
| have gotten better over time, but they're still slow.
|
| There's some ability to exploit the embarassing parallelism
| available too, but that's hard in a complex shell script, so some
| other environment makes sense. I didn't look, but hopefully
| that's a knob in rustdate ... Sometimes you'd want it and other
| times one thing at a time is better, even when it's slow.
|
| One feature I know update can do that I want to look at more but
| haven't had time is the zfs boot environment stuff --- seems like
| with the right arguments you could do the update on an inactive
| environment and reboot to switch, instead of the default way
| which does the install on the live environment but you could
| reboot to switch back. I'd be much more patient if the live
| environment continued working and I could reboot at my
| convenience once the process finished. Especially if I might be
| able to run pkg upgrade in the boot environment before the
| reboot. If rustdate supports that too, that'd be neat.
| deaddodo wrote:
| > I did some looking at freebsd-update, and it's quite the
| shell script, which makes it daunting to consider changing. All
| the forking and whatever doesn't seem great; using data
| structures should be a lot simpler!
|
| It doesn't look too crazy:
|
| https://github.com/freebsd/freebsd-src/blob/main/usr.sbin/fr...
|
| It's fairly well broken down, and pretty well organized. I
| think the most difficult part is that so much of it relies on
| shell nice-isms that are a pain to re-implement in something
| like C (although, I guess you could just call fetch and the
| like directly via 'exec' or 'system'; rather than using their
| libraries to manage things directly).
| throw0101d wrote:
| > _It 's fairly well broken down, and pretty well organized._
|
| Not surprising given that it's written by Colin Percival, who
| was the Security Officer for FreeBSD for a while, and who
| runs/develops Tarsnap:
|
| * https://www.tarsnap.com
|
| * https://en.wikipedia.org/wiki/Colin_Percival
|
| * https://news.ycombinator.com/user?id=cperciva
| kstrauser wrote:
| I do have to say that's about the cleanest shell script of
| that size I've ever seen. cperciva, if you're reading this,
| _that 's pretty_.
| cryptonector wrote:
| Hmmm, lots of `for C in $@;` that should be `for C in "$@";`
| instead.
| ptx wrote:
| Hmm. Yup. For example, this is broken: #
| freebsd-update -f "/tmp/path with spaces/freebsd-
| update.conf" fetch
| usage: freebsd-update [options] command ... ...
| cryptonector wrote:
| And it's not even setting IFS to the empty string.
| mustache_kimono wrote:
| Would you mind mirroring to a repo, with a license, so others
| might view?
| stackghost wrote:
| The source tarball, on the downloads page, contains a license.
| ThePowerOfFuet wrote:
| Shouldn't have to download and unpack a tarball to see how
| it's licensed.
| stackghost wrote:
| This seems like a real "beggars being choosers" situation.
| Pet_Ant wrote:
| I think this may be a person who is concerned a out being
| tainted by a license. Life if the code was source-
| available only and they saw it and then were accused to
| copying some functionality into another open source
| project.
|
| A bit paranoid, sure, but not impossible.
| quectophoton wrote:
| It's only a .xz one though, why the worry? /s
|
| Now seriously, I'm not particularly interested on this, but
| if it helps here are additional points of reference to at
| least be able to verify _some_ integrity in case you want
| to try it.
|
| SHA256: 90d7b2d632a3887bf26c10c42584ba899
| a3071725dae7dcddddcbab54768b54b freebsd-
| rustdate-0.6.1.tar.xz
|
| Archive link: https://web.archive.org/web/20240827090825/ht
| tps://rustdate....
|
| Doesn't prove the absence of any malicious intent, but at
| least it should help prove that nothing is changing between
| requests (i.e. that the file I got is the same as the file
| you got).
| cqqxo4zV46cp wrote:
| Thankfully you can ask for your money back.
| jeroenhd wrote:
| The license:
|
| ---
|
| Copyright 2024 Matthew D. Fuller <fullermd@over-yonder.net>
|
| Redistribution and use in source and binary forms, with or
| without modification, are permitted provided that the following
| conditions are met:
|
| 1. Redistributions of source code must retain the above
| copyright notice, this list of conditions and the following
| disclaimer.
|
| 2. Redistributions in binary form must reproduce the above
| copyright notice, this list of conditions and the following
| disclaimer in the documentation and/or other materials provided
| with the distribution.
|
| THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
| CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
| INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
| MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
| DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR
| CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
| SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
| NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
| LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
| HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
| CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
| OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
| EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
| ---
| erk__ wrote:
| It is on Launchpad here: https://launchpad.net/freebsd-rustdate
| ode wrote:
| See https://rustdate.over-yonder.net/speed.html for the numbers.
| jiripospisil wrote:
| Impressive. The current state doesn't really bother me given
| the frequency of system updates but I do sometimes wonder what
| on earth it's even doing that's taking so long.
| alberth wrote:
| > This is freebsd-rustdate, a reimplementation of freebsd-update.
|
| > In usage, it's expected to be similar, but _not_ identical to
| freebsd-update.
|
| Doesn't reimplementing it to _not_ be identical, defeat the point
| of reimplementing it?
|
| (since it can't be used as a drop-in replacement)
| youwot0 wrote:
| The ability to rewrite incompatible subsets of basic Unix
| utilities is where Rust truly shines.
| jchw wrote:
| At the very least, the uutils project seems to be making good
| progress on becoming a "true" drop-in replacement for the GNU
| coreutils that will pass the GNU test suites and treat
| compatibility issues as bugs.
|
| Honestly though, as time goes on, I'm not sure if rewriting
| UNIX utilities to be memory-safe actually gets you that much.
| I mean, having multiple implementations of something as
| important as the coreutils is good: nobody complains about
| the co-existence of e.g. BSD and GNU utilities, and IMO
| there's really no reason to feel any different about RiiR
| rewrites. That said, I can't really figure out what attack
| model would really make memory safety a particular priority.
| In most cases you would be isolating things that require
| these sorts of utilities using namespaces/jails/etc, and I
| wonder if memory bugs in the coreutils is actually a serious
| attack vector.
|
| Of course in this case (FreeBSD-rustdate) it's literally for
| performance reasons rather than safety, so everything sort of
| makes sense (not necessarily using Rust for safety per-se,
| but for robust threading. That does make sense.)
| iforgotpassword wrote:
| I think libmagic (the file util) might be a good target for
| this. I think it had security issues in the past and
| considering how it contains a gazillion random parser for
| weird file formats, surely there is a good chance there are
| some more. But nobody would ever run file on a random file
| you just downloaded, would you? ;)
| jchw wrote:
| Yeah, something like that would probably be quite good.
| That said, I think even if you do have a nice memory-safe
| implementation of libmagic/file, it's _probably_ a good
| idea to still use seccomp /namespacing/etc. to jail it
| when using it in security critical contexts. Those
| features don't really incur much cost so it's a free
| extra layer of security, and you still get the robustness
| bonus of guaranteed memory safety.
| adamnemecek wrote:
| It's not just memory safety, it's also hackability. Idk how
| I would go about adding a feature to ls (both GNU and BSD).
| I have a concrete idea of how I would go about adding a
| feature to the Rust implementation.
| jchw wrote:
| That doesn't sound very compelling: C is still a much
| more popular language than Rust and a lot simpler too. I
| realize the latter is somewhat subjective, but it's
| blatant enough that I'm willing to assert it confidently.
|
| If you want to know how to add a feature to ls in GNU
| coreutils, you don't have to look very far. The whole
| program is in src/ls.c. It's pretty large due to the
| amount of options, but it's pretty simple code all things
| considered. If you look at a program with less switches,
| like mkdir, it's even easier.
|
| https://github.com/coreutils/coreutils/blob/master/src/mk
| dir...
|
| So what about FreeBSD? Well, not only does it have an
| official GitHub mirror, but it even accepts "simple" pull
| requests, apparently. And the implementation of ls is a
| bit more approachable, since BSD utilities typically
| don't have as much functionality.
|
| https://github.com/freebsd/freebsd-
| src/blob/main/bin/ls/ls.c
|
| I don't think Rust is going to make things significantly
| more accessible. It does indeed draw people in and maybe
| make them less fearful versus C, but in practice a lot of
| the reason why contributing to these large projects is
| scary is nothing to do with the language and more to do
| with the arduous requirements of any project that is as
| complicated and widely used as they are.
|
| For example, does Rust actually make kernel development
| more accessible? Maybe not:
|
| > Despite more novice developers being attracted by Rust
| to the kernel community, we have found their commits are
| mainly for constructing Rust-relevant toolchains as well
| as Rust crates alone; they do not, however, take part in
| kernel code development. By contrast, 5 out of 6
| investigated drivers (as seen in Table 5) are mainly
| contributed by authors from the Linux community. This
| implies a disconnection between the young and the
| seasoned developers, and that the bar of kernel
| programming is not lowered by Rust language.
|
| https://www.usenix.org/conference/atc24/presentation/li-
| hong...
|
| So if you want to add a feature to ls, maybe you
| shouldn't wait for operating systems to switch to Rust
| alternatives.
|
| But when you think about it, it is not surprising that
| adding a feature to ls is not easy. I mean, you really
| wouldn't want new features being added to utilities like
| ls without a ton of care and thought put in, and that is
| true no matter what language the utility is written in.
| The difficulty of contributing to some projects has very,
| very little to do with the actual process of writing
| code.
| adamnemecek wrote:
| > Despite more novice developers being attracted by Rust
| to the kernel community, we have found their commits are
| mainly for constructing Rust-relevant toolchains as well
| as Rust crates alone;
|
| Is this surprising? The tooling is what needs fixing
| right now. Rust has been in the kernel for what a year?
| What are you expecting?
| jchw wrote:
| Oh sure, but for anyone hoping that a glut of new Linux
| contributors are just around the bend once the Rust
| tooling is in place might be suffering from unwarranted
| optimism. Judging by mailing list discussions, it's
| looking like Rust in the Linux kernel is just as
| complicated as C in the Linux kernel, and it won't really
| be a revolution so much as an incremental step. I believe
| that many kernel developers will probably adopt Rust
| because it offers them better facilities, but I don't
| think most Rust developers will suddenly take up kernel
| programming.
| tialaramex wrote:
| "Simpler" is part of the problem. The "simplicity" is
| delivered by keeping more in the programmer's head and
| not writing it down in the C source code. But if you're
| _not_ the original programmer (or you have since
| forgotten) that 's much worse.
|
| Trivial example would be the benefits of a richer type
| system: You have a TCP socket, a file descriptor, and a
| non-negative counter
|
| In C that's an int, another int, an unsigned int
|
| In Rust that's TcpStream, OwnedFd, usize
|
| The C is simpler, there are only two types and they're
| both just integers. But of course in reality the more
| complicated Rust types model what's actually going on,
| these are _not_ just integers, we shouldn 't do
| arithmetic on an OwnedFd or a TcpStream, and we can't use
| a usize when we needed a TcpStream, they aren't the same
| kind of thing. In C that knowledge lives in your head as
| the programmer instead.
|
| And I disagree in practice too, as a very experienced
| (decades) C programmer and a relatively novice (less than
| 5 years) Rust programmer I am much more confident
| contributing to a Rust project.
| jchw wrote:
| To be honest, Rust having opaque handles is a pretty
| unimpressive demonstration of its power. If that's the
| main thing that code had to benefit from Rust, it would
| absolutely not be worth a massive new toolchain with
| increased memory requirements and longer compile times.
|
| Obviously, programming language design hasn't stood still
| in the intervening years since C was created, but C had a
| lot of momentum and we don't just do massive rewrites for
| small incremental gains. IMO the only real reason why
| Rust has been compelling and stands out from most other
| options is the borrow checker. That's the thing that is
| truly compelling and that people have been enduring great
| pains for. However, it's asymmetrical; not all code is
| going to have very much to gain from the borrow checker,
| at which point _any_ modern systems programming language
| and some older ones are similarly compelling. (Note that
| I 'm not suggesting there isn't still benefits to having
| the guarantees of the borrow checker, especially if you
| really need memory safety guarantees... but many people
| don't necessarily, and some that do need even stronger
| guarantees like formal proofs.)
|
| As far as opaque handles go, you could easily accomplish
| that with a subset of C++, or even with C (with an
| asterisk, but one that would not hinder a project like
| coreutils from doing so.) When talking about stuff like
| this we're not even really talking about C, but more
| coding standards that were prevalent with old UNIX and
| UNIX-like code. And sure, improving that is net gain, but
| it has little to do with the unique merits of Rust.
|
| Outside of handles, the integer type situation in C is
| worse than Rust or really any modern programming
| language, but at least most modern code will cling fairly
| strongly to <stdint.h> types instead.
|
| P.S.: I speak with relatively similar experience on both
| fronts, but I can't say the same about being particularly
| more confident contributing to Rust projects. It's not
| that it's particularly low, but 1. I don't find too many
| opportunities and 2. I have never found contributing to
| C/C++ projects to be that terrible, and I still do so
| occasionally.
| Brian_K_White wrote:
| I think it's worthwhile just to get better correctness,
| regardless of security.
|
| There is some project to write or re-write a cad engine in
| rust which I think is valuable not for any security reasons
| but just to get a less buggy cad engine, including easier
| to keep less buggy as work goes on for years.
|
| I wouldn't have thought freebsd-update was very high on the
| list of things in desperate need of an overhaul, but I'm
| willing to grant if the people working on it think it's
| worth their time, then it probably is.
| johnisgood wrote:
| > I think it's worthwhile just to get better correctness,
| regardless of security.
|
| Yeah, well, the Rust version of coreutils had lots of
| funny logic errors involving security.
| nmz wrote:
| Given that sh can print everything as it runs, it would be nice
| to know what is it that takes so long, instead of doing a full
| rewrite.
|
| I remember once replacing a filter written in pure bash by an awk
| one, the slowdown was around 2 minutes v awk which was a second.
| kelsey98765431 wrote:
| edit the .sh, add `set +x` then pipe it to ts
|
| ```
|
| $ cat > example.sh ; bash example.sh |ts
|
| set -x
|
| echo a
|
| sleep 3
|
| echo b
|
| sleep 1
|
| echo c
|
| #end
|
| + echo a
|
| + sleep 3
|
| Aug 27 09:48:02 a
|
| + echo b
|
| + sleep 1
|
| Aug 27 09:48:05 b
|
| + echo c
|
| Aug 27 09:48:06 c
|
| ```
| kstrauser wrote:
| You can do all that without modifying the script or using a
| separate tool: PS4='+ $(date --iso-8601)\011
| ' bash -x example.sh
| alberth wrote:
| Suggestion: call this something else - because 'rustdate' makes
| me think 'rust date', which makes me believe this is date/time
| related.
___________________________________________________________________
(page generated 2024-08-27 23:01 UTC)