[HN Gopher] Rust: Does the published crate match the upstream so...
___________________________________________________________________
Rust: Does the published crate match the upstream source?
Author : lukastyrychtr
Score : 81 points
Date : 2021-10-11 12:39 UTC (10 hours ago)
(HTM) web link (codeandbitters.com)
(TXT) w3m dump (codeandbitters.com)
| verdverm wrote:
| Makes me appreciate what Go eventually did for their module
| system with respect to the issues raised here for Rust.
| lambda wrote:
| And what did Go do?
| Laremere wrote:
| When you use `go get` to retrieve a version of a dependency,
| it adds a line to a `go.sum` while with a hash of the code at
| the version specified. You can distribute your code without a
| copy of the dependency. When someone else runs go get on your
| module, it will attempt to retrieve the same version of the
| source, checking the hash. If the hashes are different, an
| error is thrown.
| TheDong wrote:
| This is in the context of rust.
|
| > it adds a line to a `go.sum` while with a hash of the
| code at the version specified
|
| Cargo.lock also contains a checksum
|
| > You can distribute your code without a copy of the
| dependency
|
| Also true in rust, and the default way of using rust/cargo.
|
| > If the hashes are different, an error is thrown.
|
| Also true in rust.
|
| For an example of what this looks like: https://github.com/
| servo/webrender/blob/54b725be37f13b166946...
|
| You haven't described anything different between go and
| rust in your comment since every feature you've pointed out
| applies equally to both.
| fnord123 wrote:
| That's what a Cargo.lock has. But it's recommended that you
| don't ship Cargo.lock that with libraries. I think the
| reason is because you can't have multiple versions of the
| same package. But I could be wrong. (If I'm wrong then you
| end up with horrific bloat with multiple trees of the same
| dep).
|
| What does go do when you have different deps depending on
| different versions of the same common dep?
| estebank wrote:
| Cargo.lock is _ignored_ for libraries, it is only checked
| for binaries.
|
| https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-
| hav...
|
| > If a library ends up being used transitively by several
| dependencies, it's likely that just a single copy of the
| library is desired (based on semver compatibility). If
| Cargo used all of the dependencies' Cargo.lock files,
| then multiple copies of the library could be used, and
| perhaps even a version conflict.
|
| > In other words, libraries specify SemVer requirements
| for their dependencies but cannot see the full picture.
| Only end products like binaries have a full picture to
| decide what versions of dependencies should be used.
| verdverm wrote:
| Minimum Version Selection
| (https://research.swtch.com/vgo-mvs)
|
| tl;dr, Go does a BFS of the dep tree, selecting the
| maximum version found. There are no ranges, so you can
| only use a version which has been listed.
|
| "minimum" comes for a minimal implementation which can
| create reproducible results without a lockfile
| verdverm wrote:
| Versions are only importable from tags (semver style)*
|
| There is no NPM or PyPi like host, sources originate from
| code hosts
|
| Modules are prefixed with a domain name (avoid dep confusion)
|
| Hashes are calculated on the module, it will fail if it is
| not correct
|
| GoProxy has a global transparency log (like a blockchain, but
| not) so that everyone is contributing and sharing hashes
|
| *there are special versions (v0.0.0 and branch names). They
| have special meaning, and get fixed with the access date and
| commit hash
|
| For the curious: https://golang.org/ref/mod and https://go.go
| oglesource.com/proposal/+/master/design/25530-s...
| Groxx wrote:
| So:
|
| > _Versions are only importable from tags (semver style)_
|
| Tags are mutable in git, which sometimes causes problems.
| Not that I think they had much choice here - git and tags
| are the de-facto community standard, they're just using it
| like everyone else.
|
| > _There is no NPM or PyPi like host, sources originate
| from code hosts_
|
| > _Modules are prefixed with a domain name (avoid dep
| confusion)_
|
| Agreed, I like this quite a bit... but it comes with the
| significant risk of domain name or hosting changes. Github
| probably won't add the metadata to redirect your old module
| name to your new non-github name, for instance. I've
| watched several modules fight through this sequence, and
| the end result is always "can't do anything, contact your
| users yourself / hope they read the readme".
|
| You win some, you lose some. In large part I think this
| part of Go modules is better than others'.
|
| > _Hashes are calculated on the module, it will fail if it
| is not correct_
|
| Nearly all lock-file-based dependency systems do this,
| Cargo included.
|
| > _GoProxy has a global transparency log (like a
| blockchain, but not) so that everyone is contributing and
| sharing hashes_
|
| Publishing a log now doesn't make it any safer tomorrow,
| it's purely about convincing you to trust them. This is
| _precisely_ the same attack target as an NPM or PyPi. The
| same party hosts the tar 'd source files + documentation +
| checksums: if it ever gets changed there, it's changed for
| everyone. Lock and go.sum files will only protect you from
| changes to your previously-known dependencies.
|
| They've done a pretty good job making goproxy/gosum easy to
| run yourself, and you can skip past the goproxy and
| download directly from the module's owner (which is a
| domain name, which works great here), which mitigates _this
| specific_ risk at least. But no, they 're absolutely
| running a pypi, they just don't let repo owners make
| changes directly (only by committing changes, e.g. module
| revocation).
| verdverm wrote:
| RE: git, if the package has been fetched through Go Proxy
| once before the tag change, the new tag will never pass
| validation
|
| > if it ever gets changed there, it's changed for
| everyone
|
| sum.golang.org has a hash chain, so you have to change
| everything that comes after a malicious hash change. So
| it does seem safer the more time that passes.
|
| Go uses zip files, as a fyi.
|
| I've had more trouble with custom domain based Go mods
| than the major code hosts. I've come to avoid those on
| custom domains as much as possible.
| TheDong wrote:
| A go module is just a git repo (or svn/hg/etc repo, but
| almost always git).
|
| Versions are git tags or commits. Dependencies are specified
| in a go.mod file, specified here:
| https://golang.org/ref/mod#go-mod-file
|
| From the following go import path: import
| "mywebsite.com/foo/bar/baz/v2/fizz"
|
| We can determine that one of the following things is true:
|
| 1. mywebsite.com has a 'meta name="go-import"' html tag in it
| when accessed by the go tool that points to a git repo/svn
| repo/etc, and the contained package has the folder structure
| "foo/bar/baz/v2/fizz"
|
| 2. mywebsite.com/foo has a 'meta name="go-import"' html
| tag...
|
| (several obvious cases omitted)
|
| 3. mywebsite.com/foo/bar/baz is a git repo that has no
| version tag, but has a "v2" folder in it
|
| 4. mywebsite.com/foo/bar/baz is a git repo that has a version
| tag with v2.x.y
|
| 5. the go.mod has a replace or other directive for this
| import path
|
| The 4th case is the relevant one here. github.com/foo/bar/v2
| means that the bar repo has a v2 tag usually.
|
| So, what version of that repo is used if there's v2.0.0 and
| v2.1.0? Well, the version is the smallest version that you or
| any module you depend on uses, so you simply have to
| recursively parse all go.mod files to understand the version
| used. Much simpler than Cargo.lock.
|
| How do you enable or disable features in a sub-go-module?
| Let's say I have a module that has a "+ui" or "-ui" build tag
| because you can optionally build an openGL ui or such.
|
| In rust, this would be a 'features = [ "ui" ]' in your
| Cargo.toml. In go, you would change your own compilation step
| to be "go build -tags ui"... And if you have two dependencies
| with those flags, and you wish to make them opposite values,
| that is simply impossible in go even though it's trivial in
| rust.
|
| There are other differences, but I feel like I've already
| made my point. My main point is that go modules make import
| paths way more opaque. It used to be that every path element
| was either part of the import path or a folder on the
| filesystem. Now, it's either part of the import path, or a
| git tag, or a folder, and none of those things are obvious.
|
| My secondary point is that go has a much simpler set of
| requirements than rust, mostly due to missing important
| features rust has.
| dane-pgp wrote:
| > Some very similar ideas were discussed in 2018 here.
|
| The link points to a page[0] of discussions from "February 18,
| 2021". I don't know if that's a date parsing bug, but the
| question of whether packages match their upstream source actually
| was considered in 2018 for the NPM ecosystem[1] which led to the
| creation of a tool called TBV (Trust But Verify). Unfortunately
| the idea didn't seem to get much traction beyond that.
|
| [0] https://internals.rust-lang.org/t/making-crates-io-verify-
| co...
|
| [1] https://hackernoon.com/what-if-we-could-verify-npm-
| packages-...
| yakubin wrote:
| I routinely read code of crates I use (which is placed under _~
| /.cargo_ if you do not vendor) and of the Rust standard library
| (which is placed under _~ /.rustup_). It's usually the fastest
| way to figure out how something works, since Googling/Stack
| Overflow will usually result in a lot of related, but not really
| applicable answers, and some incorrect ones.
|
| As for supply-chain issues, I just vendor my dependencies.
| Problem solved. Just run _cargo vendor_ and copy stdout to
| _.cargo /config.toml_ in your project. Done. Now not only you're
| not vulnerable to someone pushing malicious code to crates.io,
| but also have one place where all the dependencies of your
| project have their source code available to read for you. If you
| archive your project and then want to get back to it after years
| on another computer which did not have the dependencies
| installed, you don't need to hunt on the internet for
| dependencies that don't exist anymore[1]. If you want to work on
| a project without an Internet connection, all you need is the
| repo, you don't need to worry if a given computer has all the
| dependencies installed in the right versions. If crates.io ever
| goes down, or they take down a crate, you're safe as well.
| Although it's not the default in Rust, it's very well supported
| by cargo, if you want to develop in this model.
|
| [1]: Yes, those are my bad experiences with Python dependencies.
| ch33zer wrote:
| >Now not only you're not vulnerable to someone pushing
| malicious code to crates.io
|
| Obviously you're only protected against new compromises. If the
| code is already compromised you're screwed (but not worse than
| using normal downloaded dependencies). This does make it
| possible that you miss security updates and other important
| features.
| yakubin wrote:
| No. Vendoring doesn't change the picture in that way in Rust
| at all. Let's go one by one:
|
| _> Obviously you 're only protected against new
| compromises._
|
| Same as in the case of default Rust workflow. Cargo.lock
| ensures that versions of your dependencies are the same
| across different machines/devs.
|
| _> If the code is already compromised you 're screwed (but
| not worse than using normal downloaded dependencies)._
|
| Yes, I generally read the code of my dependencies, so that's
| fine. Checking all the digests and crypto ever won't protect
| me from having to check what I use.
|
| _> This does make it possible that you miss security updates
| and other important features._
|
| Again. Same as in normal Rust, thanks to Cargo.lock. But I
| think that's great, because the flip side of it is: I'm also
| going to miss all the new bugs. I don't want updates to be
| something that's happening behind my back. I want to check
| them and do them at the time of my choosing.
|
| What vendoring brings to the table is the simplest solution
| to "is the code on crates.io the same as in the upstream
| repo", because with vendoring I don't care about the upstream
| repo at all. I only care about the copy of code that I have
| in my own repo. Going to GitHub and then checking if digests
| match would be a very roundabout way to do that, and then
| over the years people discover flaws in the hash functions
| etc. etc. I don't bother, I choose not to have to wonder
| about it at all. I don't care about what's on GitHub, I only
| care about what's in my own repo. And then again, there are
| issues of disappearing dependencies, which digests do not
| solve, but vendoring solves in the simplest way possible.
| Digests (already incorporated by Cargo by default) only solve
| someone pushing malicious code; vendoring solves the problem
| wholesale (assuming that you actually read the code).
| KajMagnus wrote:
| > _assuming that you actually read the code_
|
| How large percent of the development time, would you say
| you spend on reading the code in the dependencies?
|
| If you would be ok with posting a link to any online repo
| of yours,
|
| then, it'd be interesting to have a look at how many lines
| of code there are, in the vendored dependencies, and
| compare with num lines in the actual program. Maybe I'd run
| `cloc`
| SV_BubbleTime wrote:
| Good article, I feel like it's rare anymore that I see easily
| actionable advice at the end of an article. Easy to complain, not
| so easy to add solutions.
| pornel wrote:
| BTW, docs.rs has browsable copies of actual sources of published
| crates ("Source" link under crates's name in the top nav), e.g.
| https://docs.rs/crate/libc/0.2.103/source/
|
| And if you actually review crates' sources, record that via
| https://lib.rs/cargo-crev so that others can benefit, too.
___________________________________________________________________
(page generated 2021-10-11 23:01 UTC)