[HN Gopher] Riff, automatically provide external dependencies fo...
___________________________________________________________________
Riff, automatically provide external dependencies for Rust projects
Author : grhmc
Score : 82 points
Date : 2022-09-06 17:05 UTC (5 hours ago)
(HTM) web link (determinate.systems)
(TXT) w3m dump (determinate.systems)
| [deleted]
| yewenjie wrote:
| Managing Python projects with Nix has been unbelievably painful
| for me - so hoping there is Python support soon. In general super
| excited that this tool is coming from Nix OGs.
| gregwebs wrote:
| Have you considered integrating this functionality with this nix
| project builder? https://bob.build/
|
| Or this dependency installer that uses nix?
| https://www.jetpack.io/blog/devbox/
|
| It is amazing to see all this user-friendly tooling come out now
| that is leveraging nix.
| biggestlou wrote:
| We're currently working on adding Riff to Nixpkgs, and as soon
| as that happens you'll be able to use it as a dependency in
| Bob.
|
| As for Devbox, that's certainly an interesting tool but not
| really something that can be integrated with Riff due to the
| language difference (Go vs. Rust), but it is fair to say that
| the projects have related goals.
| Surfactant7 wrote:
| > Riff is a tool that we built to help developers write software
| without having to wrangle dependencies or complicated
| configuration. It's built to enable you to clone a project and
| get to work in seconds. ...
|
| The title specifically mentions Rust projects. The main
| competition there is Cargo, which does exactly what the quote
| above claims. It's not at all clear what Riff offers over Cargo.
| edave64 wrote:
| It is fairly clear that it's not in competition with cargo,
| since it is literally based on cargo. It installs dependencies
| that are not managed by cargo, apparently mainly C
| dependencies.
| bombela wrote:
| external means non-rust dependencies here.
| cmyr wrote:
| I mean, it is expressed pretty clearly in the article; it uses
| Nix to manage external dependencies, and manages setting link
| paths and other build bookkeeping.
| Surfactant7 wrote:
| Thanks. I scanned right past the most important part for the
| article to get get right, mainly because it was buried in a
| paragraph, and wrapped in parenthesis no less:
|
| > ... But unfortunately they're not so great at installing
| dependencies written in other languages (we call these
| external dependencies). ...
|
| I'm sure it's a fine project. However, the article itself is
| not great at explaining the value proposition.
| grhmc wrote:
| I think there is a lot of fertile ground in the area we're
| researching here. I'd be into talking more about where we can go,
| and how Nix is behind it.
| jf wrote:
| Is there a newsletter I can subscribe to so that I get get
| notified of new posts to your blog?
| gpm wrote:
| > While Riff does its best to infer external dependencies from
| your project's crate dependencies
|
| How does it do this? Just a list of known packages and their
| dependencies, or?
|
| I'm using https://github.com/LaurentMazare/tch-rs/ (rust bindings
| for pytorch's c++ api). Depending on the hardware I'm building
| for I either need to download a cpu build of pytorch, or a cuda
| build of pytorch. How would this work with riff?
|
| Is there any hope of a good cross compiling story? If so, that
| would be awesome (currently I just don't even try if there are
| significant C dependencies)!
|
| How thoroughly are dependencies cached? Backed to the pytorch
| example it's a fairly big (and thus slow) download, I don't want
| that happening too frequently.
|
| ---
|
| In general this looks like something that could be really useful.
| Using projects with significant C dependencies currently sort of
| sucks, and this looks like it might actually fix the problem :)
| hoverbear wrote:
| > How does it do this? Just a list of known packages and their
| dependencies, or?
|
| It checks a known registry or reads package metadata.
|
| > I'm using https://github.com/LaurentMazare/tch-rs/ (rust
| bindings for pytorch's c++ api). Depending on the hardware I'm
| building for I either need to download a cpu build of pytorch,
| or a cuda build of pytorch. How would this work with riff?
|
| Riff doesn't track which GPU you use at this time, so it can't
| track that which build of `pytorch` to use, sorry. It might be
| something we make it aware of if we have enough evidence it
| would be helpful.
|
| > Is there any hope of a good cross compiling story? If so,
| that would be awesome (currently I just don't even try if there
| are significant C dependencies)!
|
| Regarding cross compiling: We've spoken about it, and I
| certainly believe it would be desirable. I imagine it would
| come in a later version. We've taken some steps to ensure we
| don't rule out that feature later.
|
| > How thoroughly are dependencies cached? Backed to the pytorch
| example it's a fairly big (and thus slow) download, I don't
| want that happening too frequently.
|
| Dependencies are cached by Nix, so they should not become
| invalidated very often. If your tempdir erases we may lose the
| generated lock file (We've been discussing where to place these
| perhaps more permanently) which may force another download if
| there was an update.
| gpm wrote:
| > Riff doesn't track which GPU you use at this time, so it
| can't track that which build of `pytorch` to use, sorry. It
| might be something we make it aware of if we have enough
| evidence it would be helpful.
|
| Would it be possible to do something like make it depend on a
| cargo feature flag, or something similar?
|
| It's probably better that this isn't automatic, since the
| correct choice of device isn't always obvious just based on
| what is available.
| larusso wrote:
| I have a strange feeling when thinking about the solution
| especially on macOS. The nix installation needs its store located
| at /nix which got harder to initialize since macOS Catalina or
| Mojave added the root protection. In general this is no problem
| when installing nix on its own. But when a wrapper tool does this
| and breaks the setup etc that feels hrmm problematic.
|
| I see potential setup issues coming up with weird store error
| messages no normal human being being able to understand.
|
| I really like nix and nixos. But I had already a nix setup which
| refused to build a rust project because of some weird linkage
| errors with libc or something. And I couldn't make heads or tails
| out of it.
|
| But I will check out this tool to get a better feeling. A general
| solution to this issue would be awesome. I would prefer JNI build
| support as well ;)
| grhmc wrote:
| What do you mean about when a wrapper tool "does this"? Riff
| doesn't install its own Nix.
| larusso wrote:
| I only checked the post so far and my understanding was that
| it uses nix. It is ironic that a tool who sets out to solve
| the issue of installing external dependencies, needs a
| preinstalled nix setup.
|
| My understanding was that it uses nix under the hood and the
| user has no real interaction with this part. But come to
| think of it it kinda makes sense.
| xiphias2 wrote:
| It looks awesome, but couldn't it be a cargo plugin instead? (Or
| nix plugin).
|
| Something like cargo --nix build ...
|
| The problem with the name riff is that it's a new tool name to
| learn again, and nix already has name recognition. I feel that
| there are just too many tool names to learn in the open source
| ecosystem.
| grhmc wrote:
| It could be a Cargo plugin, and we looked at doing that. There
| may also be some neat tricks around build.rs integration.
|
| However, we decided to wait on those. After all: it only
| supports rust _today_ , but we're eyeing a future where we
| support Go + Javascript too. We want Riff to work relatively
| consistently across these ecosystems.
| biggestlou wrote:
| If you have any questions or comments about Riff, we at
| Determinate Systems will be actively monitoring this space for
| the next few hours, so fire away.
| brundolf wrote:
| This looks really exciting, but I'm not clear on how this
| works: So what has happened here? Riff has:
| - read your project's Cargo.toml, - determined which
| external dependencies are necessary based on your
| crate dependency graph,
|
| How does Riff go from crate -> external package dependencies?
| Are those listed in the Cargo.toml somewhere? Does it just make
| a best-effort attempt to scrape the project and figure out
| what's referenced? Or does Riff maintain its own internal
| registry of which dependencies are needed by which popular
| crates?
| grhmc wrote:
| Riff has an internal registry mapping crates to external
| dependencies. However, the registry is intended to bootstrap
| Riff's utility. If projects define their own Riff metadata in
| their own Cargo.toml, that data is preferred. The README
| shows how to do that:
| https://github.com/determinateSystems/riff#how-to-declare-
| pa...
| brundolf wrote:
| Neat! Sounds reasonable
| xenago wrote:
| This looks like a really good idea for solving that pain point
___________________________________________________________________
(page generated 2022-09-06 23:01 UTC)