[HN Gopher] Single-file scripts that download their dependencies
___________________________________________________________________
Single-file scripts that download their dependencies
Author : networked
Score : 56 points
Date : 2023-01-15 12:13 UTC (10 hours ago)
(HTM) web link (dbohdan.com)
(TXT) w3m dump (dbohdan.com)
| mkl95 wrote:
| One of the reasons I like Go.
| gpvos wrote:
| Style points for using leftpad as the example.
| DangitBobby wrote:
| The python example is missing the most important part, which is
| how to make your script work with external dependencies. Here is
| a working example.py with external dependencies:
| #! /usr/bin/env nix-shell #! nix-shell -i python3 -p
| python3 -p python3Packages.requests -p python3Packages.fire
| import requests import fire class CLI:
| def get(self, url): return
| requests.get(url).status_code fire.Fire(CLI)
|
| Run with chmod +x example.py ./example.py get https://example.com
| nathan_phoenix wrote:
| The nix part seems quite useful. It basically allows one to run a
| script in any language, with it being self contained and without
| polluting the global path with all kinds or runtimes. With the
| only user installed dependency being nix.
|
| Although I'm wondering about how fast it starts on the 2nd run
| after the dependencies have been download and how it deals with
| language specify dependencies, e.g. npm/pip packages. Read
| somewhere thst it can be a pain deal with those in nix.
| rkeene2 wrote:
| AppFS [0] solves a similar problem but with much lower learning
| curve and overhead for using it, since it removes the install
| step altogether.
|
| [0] https://appfs.net/
| DangitBobby wrote:
| From the linked docs, it's not obvious to me how I could use
| this tool in the way TFA uses nix-shell to resolve script
| dependencies.
| rkeene2 wrote:
| You just use the dependencies, for example:
| $ cat /tmp/hello.py #!
| /opt/appfs/rkeene.org/python3/platform/latest/bin/python3
| print("Hello, world!".rjust(20, "-")) $
| /tmp/hello.py -------Hello, world!
| DangitBobby wrote:
| The really nice thing about the nix-shell example is I
| can specify external python dependencies within the
| script itself, and even the python version itself, all
| within one file, and nix-shell just takes care of it.
| rkeene2 wrote:
| If you change `latest` to the version [0], then you'll
| get that exact version.
|
| Additionally any external dependencies can be specified
| by just accessing them the same way you would in `nix-
| shell`, just using the path instead of installing. For
| example, if you wanted to use the "awscli" (the only
| Python package I already created within AppFS) you would
| do: $ cat /tmp/aws.py #!
| /opt/appfs/rkeene.org/python3/platform/latest/bin/python3
| import sys
| sys.path.append('/opt/appfs/rkeene.org/aws-
| cli/platform/latest/lib/python3.6/site-packages')
| import awscli print(awscli) $
| /tmp/aws.py <module 'awscli' from
| '/opt/appfs/rkeene.org/aws-
| cli/platform/latest/lib/python3.6/site-
| packages/awscli/__init__.py'>
|
| [0] https://browser.appfs.net/rkeene.org/python3/linux-x8
| 6_64/3....
| one-punch wrote:
| 1. Not sure how AppFS could have lower overhead than Nix in
| terms of removing the install step.
|
| 2. AppFS appears to be written by a user named rkeene, and I
| am replying to a comment by rkeene2. I think a disclosure
| would be helpful when comparing AppFS with other tools.
| rkeene2 wrote:
| Nix has an install step to use a package, AppFS does not --
| not having a step is easier than having one. If you want to
| use a package, you just use it.
|
| Additionally, I'm the author of AppFS.net just redirects
| you to AppFS.rkeene.org -- I'm not sure what kind of
| disclosure is needed when discussing it in this context
| though.
| jyooru wrote:
| Nix is really cool, and gets even cooler with NixOS.
|
| To answer your questions: On the second run all packages are
| saved in the local store, so it's super fast. Language specific
| dependencies are treated like all other Nix packages, so Python
| dependencies can depend on Node, etc.
| ALLTaken wrote:
| It actually creates a directory with hashes of the packages,
| so if you run another install that requires a package within
| that was previously downloaded, it just skips the download
| too.
| aseipp wrote:
| Python is pretty easy. You just add something like '-i python3
| -p python3Packages.click' or whatever to the shebang and it
| normally works and you can keep adding '-p ...' flags (assuming
| the package in question is upstream.) I don't know enough about
| the state of node to comment.
| mananaysiempre wrote:
| Evaluation can be slow, which is why flakes-based tools usually
| cache the evaluation results. (There aren't going to be
| duplicate downloads in any case.)
|
| There's no shebang support in nix3 (the new, flakes-native CLI
| with a single "nix" command). The closest you can get is
| probably "nix run"--can't be distilled down to a single file,
| but can fetch and cache even online Git repos (use "nix
| registry pin" to freeze the commit hash).
| one-punch wrote:
| > There's no shebang support in nix3.
|
| This is a pain point, but there appears to be discussions and
| a PR for Shebang support with flakes:
| https://github.com/NixOS/nix/pull/5189
| ewuhic wrote:
| What would be an example filetree structure for the nix run
| to match #!-interpreter via flake? Cannot all the definitions
| fit into a single "flake.nix"? Also, would it be possible to
| use some other name than "flake.nix"?
| mananaysiempre wrote:
| As far as I know, flake.nix is hardcoded, so you're going
| to use a separate directory for every script anyway. And
| then there's flake.lock, which you have to go out of your
| way to avoid creating, and you'll be spammed with obnoxious
| warnings if you still do.
|
| So would you want to fit everything into just flake.nix? I
| don't see why. Could you? Yeah, sure: http://ix.io/4ll5 is
| an example for a single shell script.
| PaulHoule wrote:
| Seems like you could write a Python script that branches two
| ways, either makes a venv or runs.
| andreareina wrote:
| PyPI packages don't always have the same name as what you
| import. PIL comes to mind, as well as others I don't remember
| right now.
| felipelemos wrote:
| pyyaml is a good example as well
| rubenfiszel wrote:
| Indeed: https://github.com/windmill-
| labs/windmill/blob/3f318b4ac4170... :)
| PaulHoule wrote:
| Is that really a problem? Can't you just stuff a list of deps
| in your script?
|
| This doesn't seem too different from the old 'setup.py'
| approach of Python packaging except that the 'one script' has
| both a 'setup.py' in it and the real script.
| andreareina wrote:
| Sure. I skimmed the article and only really looked at the
| code for the deno one so I missed that they pretty much all
| had some special way to declare the deps that needed to be
| fetched.
| rubenfiszel wrote:
| It's a topic close to my heart since that is one of the key thing
| that we solve in the OSS project windmill [1], letting users
| define simple single-file scripts in python/deno/go and handle
| the dependencies for them.
|
| For deno, as the article points out, it's easy, deno has this
| feature baked in and we just have to deno run the scripts,
| similar for go where the imports are pretty much a fully
| qualified pointer to their source and version.
|
| For python, we have to do a bit of machinery: we parse the AST,
| look at the imports, use the heuristics that most import names
| correspond to their pypi package (we maintain also a list of
| exception-mappings), pip-compile [2] all the imports, and then
| get a requirement file that we attach to the script, pip-install
| it before running the script (then we do a LOT of magic to cache
| the dependencies so one doesn't actually have to install them
| 99.9% of the time).
|
| Note: if you're interested in a collection of single-file scripts
| to many APIs, that you can just copy-paste and call the main
| function of, that's what windmill's hub is for [3]
|
| [1]: https://github.com/windmill-labs/windmill [2]:
| https://github.com/jazzband/pip-tools [3]:
| https://hub.windmill.dev/
| agoose77 wrote:
| Python actually has support for this with `pip-run`. There's a
| PR to get similar behaviour into `pipx`, which is effectively
| the `npx` of Python world. I think the single-script-with-
| dependencies workflow has been somewhat ignored until now; it
| would be nice to see that become easier.
| rubenfiszel wrote:
| If I'm not mistaken, looking at the docs of:
| https://pypi.org/project/pip-run/ it doesn't parse the AST to
| infer the dependencies, you still have to declare them
| (either in script with # Requirements comment (which we also
| use as an escape hatch
| https://docs.windmill.dev/docs/reference#dependency-
| manageme...)) or externally declared.
|
| So it's close but not exactly the same.
| dottedmag wrote:
| Static binaries.
|
| Cosmopolitan libc for poly-OS static binaries.
| AgentME wrote:
| It's nice to see Deno recognized here. The ability of scripts
| written for Deno to import their dependencies by full URL instead
| of being dependent on a separate package.json + package manager
| install step being done first helps a surprising amount with
| certain tasks. It becomes possible to post a complete working
| block of code that uses arbitrary libraries into a single
| immediately executable code block online for example. Compare to
| Node or Python where any code block casually posted online to be
| really complete needs to be paired with an explanation to first
| npm install some specific packages (with specific version
| numbers, if the code snippet is meant to be useful in the
| future). There's a difference between posting code that's usable
| on an end user's machine with a few manual often-implicit steps
| done separately vs one that's immediately executable with exactly
| the intended environment as-is.
| [deleted]
| real_joschi wrote:
| The author forgot JBang (Java, Groovy, Kotlin).
|
| https://www.jbang.dev/
| ciconia wrote:
| Ruby has this too: require 'bundler/inline'
| gemfile do gem 'httparty' end p
| HTTParty.get("https://acme.com/")
| hardwaresofton wrote:
| You should "drop him a line" (if you haven't already)
| networked wrote:
| Thanks for the suggestion. I replied about Ruby here:
| https://news.ycombinator.com/item?id=34389323.
| Borkdude wrote:
| Babashka (https://babashka.org/) example: (ns
| malli (:require [babashka.deps :as deps]))
| ;; This adds malli (https://github.com/metosin/malli) to
| babashka's classpath: (deps/add-deps '{:deps
| {metosin/malli {:mvn/version "0.10.0"}}}) ;; Now
| we're able to load malli: (require '[malli.core :as
| malli]) (prn (malli/validate [:+ :int] [1 2 3])) ;;
| true
| zackees wrote:
| [dead]
| mhitza wrote:
| In the Haskell ecosystem this is also achievable if you have the
| stack tool installed.
|
| https://www.fpcomplete.com/haskell/tutorial/stack-script/
| vinceguidry wrote:
| Nobody ever mentions Ruby in these lists. :sad-panda: Check out
| inline Bundler:
|
| https://bundler.io/guides/bundler_in_a_single_file_ruby_scri...
|
| Combine this with a shebang and you can get a heck of a lot of
| power into one file.
| networked wrote:
| Does this install dependencies on a per-script basis? Can
| different scripts' dependencies conflict with each other? I
| think I saw this when I initially put together the list, but I
| didn't include Ruby because I wasn't sure how it worked.
|
| I'll add Ruby if conflicts are normally impossible. (If
| conflicts are possible, the method is equivalent to shelling
| out to the language's package manager at the top of the script
| before the imports, which you can do in most scripting
| languages.)
| thedanbob wrote:
| Conflicts aren't an issue. Multiple versions of a library can
| coexist and bundler will install/activate the requested
| version on a per script or project basis.
| vinceguidry wrote:
| I believe they get installed to a particular location that
| can be of course set in bundler's config. I believe this is
| also per script, so they one script won't interfere with
| another. I've used it before, and never had a problem
| networked wrote:
| Thanks. I'll do a little research to confirm it and add
| Ruby if this is the case.
| ddlsmurf wrote:
| I think they missed the best one: https://github.com/teaxyz/cli
|
| It doesn't even need tea itself to run, it'll make a bash script
| that'll get it too if absent.
| imiric wrote:
| Tea seems interesting, but the antagonistic tone in the README
| is very off-putting. Just tell me what it does without ranting
| about every other tool I currently use, and presenting yourself
| as the next best thing. Chances are you actually aren't.
| [deleted]
| samsquire wrote:
| It seems the complicated part of using a computer is not adding
| numbers together but organisation and preparing an environment
| for computation.
|
| To run a computation I need code and data to be in the right
| place. If you analysed the number of add instructions of an
| average compiled program, I feel we would be surprised how many
| instructions are actual computations and not copying or moving.
| I'm excluding address calculations and referring to "add" or
| "subtract".
|
| Preparing an environment for computation is copying data to the
| machine and preparing dependencies such as docker, package
| managers, compilation and other work. It's also moving data
| between registers in preparation for a calculation. It's similar
| to a factory processing raw ingredients that need transportation.
|
| How do you represent all this movement and logistics efficiently
| and untediously? I feel computer science could probably expend
| some effort on representing logistics efficiently.
| Serenacula wrote:
| I think the entire field of computer programming might be
| described as various methods of organising and representing
| logical structures efficiently.
| samsquire wrote:
| You're right.
|
| Within a codebase there's a arbitrary number mental models
| that that particular code base uses to solve problems.
| Perhaps it uses a calling convention, or a REST API or a C
| FFI API. Or Spring. The problem is ultimately how do I get
| data from here to there so that I understand it for doing an
| organised computation on it.
|
| When you go outside that codebase, you switch mental models
| of how to solve and think of problems. It's similar to
| switching between writing in Python and then writing commands
| in bash shell.
|
| It's similar to switching between HTML, CSS and Javascript as
| a frontend developer. Or embedded C programming or the code
| of a compiler.
|
| I feel the logistical problem is space or position based,
| which can also be computed with numbers or vectors as with
| language models.
|
| How many of the problems that people solve with programs are
| the same problem, said slightly differently? BSD sockets
| standardised communication but we have so many solutions to
| positional+placement logistics: systemd, sysv,
| Tomcat/application servers, kubernetes, package managers,
| docker, wasm runtimes, virtual machines, firecracker, OSGI,
| LXC, processes, threads, databases and so on. Said another
| way: what is where.
| 082349872349872 wrote:
| https://en.wikipedia.org/wiki/KeyKOS was an OS that attempted
| to solve the problem of bringing code and data together when
| both parties don't wish to leak any information to the other.
| (example: the code is a third-party key management app, and the
| data are an org's private keys)
|
| Unfortunately, having been in development from 1977-1988, it
| was an example of Solving a Problem Too Early.
| ThinkBeat wrote:
| I am not a fan of having a script pull down all manner of things
| in order to run.
|
| The smart is of course to review the script and see what its
| dependencies are.
|
| This is not always straight forward, esp. when it gets recursive.
|
| In that NPM is the worst offender I have tried.
|
| It can easily expand from 10 libraries to hundreds of
| dependencies (and dependencies of the dependencies)
| dtgriscom wrote:
| Only hundreds? You must be pretty careful to choose libraries
| with few dependencies. (I should be more careful...)
| TJSomething wrote:
| You can do something similar in F#. This creates a cross-platform
| desktop GUI. #!/usr/bin/env -S dotnet fsi
| #r "nuget: Avalonia, 11.0.0-preview4" #r "nuget:
| Avalonia.Desktop, 11.0.0-preview4" #r "nuget:
| Avalonia.Themes.Fluent, 11.0.0-preview4" #r "nuget:
| JaggerJo.Avalonia.FuncUI, 0.6.0-preview4" open
| Avalonia open Avalonia.Controls open
| Avalonia.Controls.ApplicationLifetimes open
| Avalonia.FuncUI open Avalonia.FuncUI.DSL open
| Avalonia.FuncUI.Hosts open Avalonia.Layout open
| Avalonia.Themes.Fluent open System module
| Main = let view = Component(fun ctx
| -> DockPanel.create [
| DockPanel.children [
| TextBlock.create [
| TextBlock.fontSize 48.0
| TextBlock.verticalAlignment VerticalAlignment.Center
| TextBlock.horizontalAlignment HorizontalAlignment.Center
| TextBlock.text "Hello world!" ] ] ]) type
| MainWindow() = inherit HostWindow()
| do base.Title <- "Hello World!"
| base.Height <- 400.0 base.Width <- 400.0
| base.Content <- Main.view type App() =
| inherit Application() override this.Initialize()
| = this.Styles.Add(FluentTheme(baseUri = null,
| Mode = FluentThemeMode.Dark)) override
| this.OnFrameworkInitializationCompleted() = match
| this.ApplicationLifetime with | :?
| IClassicDesktopStyleApplicationLifetime as desktopLifetime ->
| let mainWindow = MainWindow()
| desktopLifetime.MainWindow <- mainWindow | _ ->
| () AppBuilder .Configure<App>()
| .UsePlatformDetect() .UseSkia() .StartWit
| hClassicDesktopLifetime(Environment.GetCommandLineArgs())
| tinyspacewizard wrote:
| Suave is really cool for this too.
|
| Here is a hello world server that fetches all of its own
| dependencies - just 3 lines! #r "nuget:
| Suave, 2.6.2" open Suave
| startWebServer defaultConfig (Successful.OK "Hello World!")
| 1wd wrote:
| pipx might get this ability:
| https://github.com/pypa/pipx/pull/916
| Alifatisk wrote:
| Interesting to see D in the list!
___________________________________________________________________
(page generated 2023-01-15 23:01 UTC)