[HN Gopher] Ask HN: What to use instead of Bash / Sh for scripting?
___________________________________________________________________
Ask HN: What to use instead of Bash / Sh for scripting?
I'm at the point where I feel a certain fatigue writing Bash
scripts, but I am just not sure of what the alternative is for
medium sized (say, ~150-500 LOC) scripts. The common refrain of
"use Python" hasn't really worked fantastically: I don't know what
version of Python I'm going to have on the system, installing
dependencies is not fun, shelling out when needed is not pleasant,
and the size of program always seemingly doubles. I'm willing to
accept something that's not on the system as long as it's one
smallish binary that's available in multiple architectures. Right
now, I've settled on (ab)using jq, using it whenever tasks get too
complex, but I'm wondering if anyone else has found a better way
that should also hopefully not be completely a black box to my
colleagues?
Author : lordgroff
Score : 26 points
Date : 2021-04-10 13:05 UTC (9 hours ago)
| badhabit wrote:
| #!/usr/bin/newlisp
| 2OEH8eoCRo0 wrote:
| Perl is nice and ubiquitous.
| hedora wrote:
| Agreed. Also, Perl's sweet spot is 1-500 lines of code. (Beyond
| that, not having types starts to hurt.)
|
| Also, for scripting stuff, my (readable, extensible) perl is
| about 10% as long as other people's python, so 500 lines of
| perl ends up being 5000 lines of python. That creates a huge
| difference in maintainability and correctness.
|
| With code golf, my perl can be 10x shorter than that, but then
| it is neither readable nor extensible, and doesn't scale to 500
| (or even 50) lines before becoming more trouble than it's
| worth.
| mistermithras wrote:
| Have you tried other shells like csh or tcsh?
| michaelcampbell wrote:
| The oldie Tom Christiansen anti (t)csh post is worth a read for
| those considering alternatives.
|
| https://shlomif.github.io/open-source/anti/csh/ has a link to
| it, and other things.
| elviejo wrote:
| I've been wondering the same myself... I want: - reliable
| scripts::in my case this means strong typing. Because for example
| currently to see the exit status of a program one need to check
| the exit number 0 or non zero plus the string for the command. -
| crosplatform:: this eliminates interpreted languages like ruby
| and python - easy to interact with the environment :: is
| something that like bash can actually call programs installed on
| the computer.
|
| As the song says: "But I still... haven't found... what I'm
| looking for."
| brundolf wrote:
| I recommended Node in another comment, but if you want typing
| I'd recommend Deno. Similar reasons, but add the fact that you
| get type checks without a build step.
| hedora wrote:
| Go might be a decent choice. It's basically python, but with
| reasonable type and error handling.
|
| It still has other python-style problems, like epic dependency
| graphs, with all the compatibility and bitrot nightmares those
| imply.
| nailer wrote:
| With typescript installed:
|
| usr bin env ts-node
|
| On phone but you get what I mean
| zzo38computer wrote:
| Depends on the program. Sometimes I use shell scripts, and
| sometimes I use Node.js, and (more rarely) PostScript.
| brundolf wrote:
| I really like NodeJS for more complex system-scripting. In
| contrast with Python:
|
| 1) There aren't two disjoint versions to juggle
|
| 2) Dependency installation is clean and contained
|
| 3) Dependencies can easily be installed locally or globally,
| depending on your preference for isolation vs reuse
|
| 4) Lots of packages are available for doing cross-platform (read:
| Windows compatible) operations
|
| 5) JS makes it really easy to do efficient async tasks, for
| example high-IO scripting involving reading and writing lots of
| files and making network requests (without any dependencies, I
| might add)
|
| At my last company I converted all our shell scripts to Node
| scripts when the pandemic hit, so they'd work predictably on
| Windows machines, and it was great. These days I'd do it that way
| to begin with.
| lordgroff wrote:
| JavaScript as a shell replacement, what a time to be alive!
|
| I am familiar with and use quite a bit of js on the front end,
| but how is node for shelling out? This is something I find
| Python to be quite terrible at.
| sbacic wrote:
| Let me put it this way; I recently wrote a script to parse
| the output of last and format it.
|
| The Node version was some 90 LoC but if I included the
| dependencies it came to about 5 MB. Then I rewrote the same
| script in Python3 and it was 60 LoC and no dependencies that
| needed to be installed separately.
|
| I will admit that the JS code looked _a lot_ cleaner than the
| the Python code though.
| brundolf wrote:
| I wouldn't use it as an interactive prompt personally - it's
| a bit too verbose - though I did know someone in college who
| did that :)
|
| Calling shell commands is reasonably straightforward:
| const { exec} = require('child_process') const child =
| exec('ls', ['-lh', '/usr']) console.log('error',
| child.error) console.log('stdout ', child.stdout)
| console.log('stderr ', child.stderr)
|
| However, you can also get cool packages like this to use
| instead: https://github.com/shelljs/shelljs
|
| I will admit though that in _most_ of the cases where I 've
| used it I didn't really do any shelling-out, so I can't speak
| much to how that experience scales in practice. Mostly I was
| either 1) doing self-contained logic with files, web
| requests, etc, or 2) delegating out to JS build tools (which
| even when they have a CLI, almost always expose a nice JS API
| too). Ymmv, etc.
| schwartzworld wrote:
| Node is great for shell scripts. Async programming is a
| breeze in JavaScript. I wrote a quick wrapper function around
| child_process.exec to promisify it, and this allows me to use
| features like async / await and Promise.all in my scripts.
| chatmasta wrote:
| I tend to write a lot of bash, partially for masochistic
| reasons, but I would probably choose Node over Python in 2021
| for basic dev tooling scripts. Setting up a node environment
| with nvm can be done in a few lines of code, and it's pretty
| reliable across platforms. Python is just too difficult to
| support consistently.
|
| Really you should pick whatever you currently have in your
| codebase. We happen to have both Python and Node, with the
| languages generally being developed separately by different
| devs. So we've got tools in both, but a Node developer is
| unlikely to ever need/use the Python tools (although they can
| all run in Docker anyway). What annoys me a bit is that pre-
| commit requires a Python env to bootstrap itself, but it can
| install nvm without any fuss.
| lastofus wrote:
| Check out Babashka if you enjoy lisps/Clojure.
|
| https://github.com/babashka/babashka
| lordgroff wrote:
| Oh my god. I do, and will.
| [deleted]
| [deleted]
| auslegung wrote:
| I've seen people use Go because it's a simpler language. It's
| what I would probably use if I didn't want to use bash.
|
| What do you think about editing the title to include "for
| scripting"? I thought you were asking for recommendations like
| zsh or fish when I first clicked
| magicpointer wrote:
| Also go has the advantage of producing a single static binary.
| Easy to build on your machine and run on another host.
| berkes wrote:
| Which is why I'm writing most of my tooling in Rust _.
|
| Deployment or installation is just one scp away.
|
| _ not rewriting, just when It needs large refactoring, or is
| currently a mess, or brand new.
| brundolf wrote:
| Assuming you're deploying to the same OS/architecture that
| you're working from, otherwise you'd have to rebuild (which
| Rust makes easy to do, but still)
| Raed667 wrote:
| How easy is it to cross-compile with Rust?
| brundolf wrote:
| Trivial. All of the tooling, dependency management,
| linking, etc. is neatly bundled together under cargo.
| Just run "cargo build" for the current system, or specify
| one or more other architectures as arguments to get
| multiple binaries. Everything's statically-linked so you
| should be able to scp those wherever.
|
| Still, when we're talking about casual "scripts" even
| this extra step seems a little bit burdensome
| lordgroff wrote:
| Good point, changed.
| cpach wrote:
| You might want to experiment with Powershell. It runs on Linux.
|
| You might not adopt it, but it can give perspective on the pros
| and cons that comes with Bash.
| openfuture wrote:
| With nix you can make anything into a script, there's some people
| using haskell for example.
|
| I tend to use common lisp but I'm learning guile scheme and will
| probably move to that instead.
| hollerith wrote:
| I don't understand (even though Nixos was my only OS for a
| couple of months).
|
| How would Nix help me make Haskell into a script?
| jiehong wrote:
| I think OP means that it would allow you to pull it from
| anywhere you have nix installed...
| 002445 wrote:
| I use Fish as my shell instead of bash precisely because I really
| like the scripting language for my personal scripts.
|
| It's not the best fit for everyone but it has served me well.
| ilyash wrote:
| Next Generation Shell was born exactly out of the pain of using
| bash and Python. I'm the author and I felt the same way.
|
| > shelling out when needed is not pleasant,
|
| Handled properly because the language was built ground up for Ops
| tasks. There is even a syntax for "run a command and parse the
| output". Rationale and examples - https://ngs-
| lang.org/doc/latest/man/ngswhy.1.html
|
| Currently NGS supports Linux and MacOS.
|
| > smallish binary
|
| Unfortunately there is no static build yet so can't check this
| checkbox
|
| Next Generation Shell - https://github.com/ngs-lang/ngs
|
| Alternative Shells list by @chubot -
| https://github.com/oilshell/oil/wiki/Alternative-Shells
| zbuf wrote:
| I don't think the breaking point is the number of lines in the
| script; your pain point is probably one or two specific concepts
| or features? (or lack of)
|
| Shell is likely to remain the best way to connect together other
| pieces of software.
|
| The tipping point, in my experience, is folks trying to use
| arrays or other buffering of data. At that point they're writing
| in shell, but wishing for a procedural language.
|
| Whereas, shell works excellently (and incredibly efficiently) if
| you can compose your task strictly into streams of data (a more
| 'functional' feel)
|
| So the real answer I think is specific to the types of script you
| are writing!
|
| But also it could be worth revisiting the style in which you're
| using shell; especially if you can call upon your own helper
| programs (eg. in Python) where you require buffering or non-
| streamed access to data.
|
| Whilst there's a lot of documentation of sh/bash functionality, I
| don't think there's anything like the same momentum behind 'best
| practices' as other languages like Python or even C.
| TheGrassyKnoll wrote:
| I tried using Python with import sh for awhile, but...meh.
| https://amoffat.github.io/sh/sections/faq.html
| runjake wrote:
| Just use shell or Python3 and stop nitpicking IMHO. There aren't
| really any other choices.
|
| Most OSes come with relatively modern versions of either. Pretty
| much everyone is or should be familiar with either. It's not like
| it's some Lispy thing.
| exdsq wrote:
| I'd say Go might be your best bet - it's easy to generate
| executables, you get a simple type system, and it's pretty easy
| for anyone to pick up and modify a program in. However another
| alternative is to use the language of the project so it's
| uniform. Most languages are pretty usable for scripting.
| westurner wrote:
| A configuration management system may have you write e.g. YAML
| with Jinja2 so that you don't reinvent the idempotent wheel.
|
| It's really easy to write dangerous shell scripts ("${@}" vs ${@}
| for example) and also easy to write dangerous Python scripts
| (cmd="{}; {}").
|
| Sarge is one way to use subprocess in Python.
| https://sarge.readthedocs.io/en/latest/
|
| If you're doing installation and configuration, the most team-
| maintainable thing is to _avoid custom code_ and work with a
| configuration management system _test runner_.
|
| When you "A shell script will be fine, all I have to do is [...]"
| and then you realize that you need a portable _POSIX_ shell
| script and to be merged it must have actual _automated tests_ of
| things that are supposed to run as root - now in a fresh vm
| /container for testing - and manual verification of `set +xev`
| output isn't an automated assertion.
| ThePhysicist wrote:
| Perl (5). It's an incredibly capable scripting language, many
| things essential for Shell programming are much better integrated
| than in Python (e.g. regular expressions). It gets a lot of hate
| these days but it's still a damn fine language and perfectly
| suited to shell programming (and larger projects).
| lordgroff wrote:
| One of the things about Perl is that I can't quite believe
| myself how far it has fallen.
|
| I was never a Perl monk by any means but I knew enough to get
| by. And why did I not even think about it? I haven't coded Perl
| in years, but if my memory serves me correctly, bundling extra
| dependencies in Perl was a heck of a lot less painful than
| Python too.
| max_hammer wrote:
| If you are ready to invest some time then try `perl`. It is
| awesome.
| curun1r wrote:
| This won't work for many people who don't know the language or
| its ecosystem, but I've started doing my ad-hoc scripting in Rust
| using rust-script [0]. I've got a template with lots of
| boilerplate for everything I might need and I just copy it and
| delete the parts I don't want. Having argument parsing, dotenv
| init, toml configuration and such ready to go with just a few
| tweaks, simple user prompting with dialoguer [1] and easy
| delegation to other tools with cmd_lib [2] means I can often
| write the tool I'm envisioning in just a couple of minutes.
|
| [0] https://rust-script.org/
|
| [1] https://crates.io/crates/dialoguer
|
| [2] https://crates.io/crates/cmd_lib
| JohnL4 wrote:
| Can you be more specific than "fatigue"? That sounds like ennui,
| in which case maybe consider a job switch instead of a language
| switch (not being sarcastic or intentionally mean). Bash plus all
| the Unix tools (awk, grep, sed, tsort, comm, the ones I don't
| know, etc.) have pretty long legs, in terms of functionality and
| the syntax is pretty understandable. They provide most of what
| Perl offers, apart from OO (-ish) and line noise.
|
| If you want strong typing (are you scripting in the large?),
| then, almost by definition, no scripting language provides that.
| If you need performance, switch to a compiled language (or a jit-
| compiled interpreted language) that most of your co-workers
| understand.
|
| If Python brings too many dependencies, I guess you're trying to
| do something that is way out of the realm of possibility for
| bash, so you're not looking for an "alternative" to bash, you're
| looking for the best language for whatever project you're
| undertaking. And you may not be able to get away from the
| dependencies if you're trying something specific/sophisticated.
| majkinetor wrote:
| > If you want strong typing (are you scripting in the large?),
| then, almost by definition, no scripting language provides
| that.
|
| Powershell is basically strongly typed.
|
| > Can you be more specific than "fatigue"? That sounds like
| ennui
|
| I love my job, but bash makes me wanna hang myself. I can say
| the same about stored procedures. Its like going back to cobol
| days. Some people have strong tastes I guess.
| akagusu wrote:
| The question you should ask yourself is not what use instead of
| bash but what is the problem you are trying to solve and what is
| the best way to solve it.
|
| It's hard for us to help you with you giving us some context.
|
| 150-500 loc of bash are not that much, specially when you combine
| bash with standard Unix utilities.
|
| These scripts you wrote, for what do you use them? Are they
| executed by humans or machines? On which platform? Linux, Mac,
| Windows or all?
|
| You mentioned jq. Are you parsing json from HTTP requests or
| files? And why are you parsing json with bash?
|
| Without a better context, A lot of people will say to use Python
| and Python is a good choice with you are running you scripts on
| Linux because you probably already have Python 3 installed and
| whatever you are writing with 500 loc of bash can be accomplished
| with zero dependencies with Python standard library.
|
| If relying on the Python runtime is really a problem for you,
| your next best option is Go. Go is easy to learn and easy to use
| for building small cli apps with only the standard library, but
| even if you need some dependency, you will ship a single binary,
| so no problems with runtime compatibility or dependencies.
| majkinetor wrote:
| Just use PowerShell. You can do so much in it without any
| dependency (jq is native in it and many other things).
|
| It is cross-platform so you can use your scripts anywhere.
|
| Don't use python or go or ruby or whatever as those are not
| scripting languages.
|
| IMO Bash is horror and should die already.
| lordgroff wrote:
| I use Powershell in Windows and am a big fan. It is a
| biiiiiiiiiiiiiiiiiiiig download on Linux, making it pretty
| impractical managing many different Linux machines.
| majkinetor wrote:
| > It is a biiiiiiiiiiiiiiiiiiiig download
|
| How big really? Can't be more then 200MB? Should be less then
| minute to download and install? If you have docker, can you
| go that route?
|
| Also, if you account the fact that you have variants of many
| other popular tools included OTB is that really a downside ?
|
| That is certainly different concern, you want fast and
| practical script programming and why would you care if it has
| first time latency of couple of minutes if you are set good
| forever after that ?
| scaladev wrote:
| I just checked, if you already have .NET installed, the
| powershell package is about 20 MBs, which is... tolerable.
| However, I see that the latest version still starts very
| slowly compared to *nix shells (and also Python).
|
| The shell was running from tmpfs (so basically a RAM disk).
| I have a decent processor and an NVMe SSD, and this is what
| I see (it was tested 10 times in a row to pull everything
| into the page cache properly, and the numbers are pretty
| stable): $ time ./pwsh <<<"" >/dev/null
| real 0m1.050s user 0m1.464s sys 0m0.067s
|
| More than a second to execute an empty 'script', and even
| longer to get to the interactive prompt (which is harder to
| time properly so I'm not publishing any numbers).
|
| It's even slower on an older Windows laptop which I have
| lying around (starts in about 3-4 seconds). Absolutely
| terrible for an interactive shell/script execution engine.
|
| Just for comparison's sake, the same laptop can run an
| empty bash script in 2-3 milliseconds. This is very
| important when running build jobs, for example.
| majkinetor wrote:
| How about pwsh -NoLogo -NoProfile ? >
| Measure-Command { pwsh -NoProfile -NoLogo exit }
| Days : 0 Hours : 0
| Minutes : 0 Seconds : 0
| Milliseconds : 254
|
| BTW, Windows OS is often subject of Windows Defender and
| other AVs which are not present on Linux machines and
| their slowdown of the system is phenomenal.
| bsg75 wrote:
| Python is not a scripting language?
| majkinetor wrote:
| Not designed for shell scripting.
| pskinner wrote:
| Powershell is a horror you mean, it's got the verbosity of
| objective c with the consistency that comes with all Microsoft
| designed products.
|
| Just use a proper scripting language.
| majkinetor wrote:
| No, I said what I mean. Bash and its disasters like [ as
| executable and mega error prone and awkward language designed
| 50 years ago IS horror.
|
| Verbosity of powershell is optional, I don't use it.
|
| > consistency that comes with all Microsoft designed
| products.
|
| Oh, lets not troll here.
| jolmg wrote:
| > its disasters like [ as executable
|
| Why is that a disaster? What problem has it caused you that
| wouldn't have happened if it wasn't an executable?
|
| [[ is a non-executable version of [. Does that solve your
| problem?
| majkinetor wrote:
| I don't have a problem - I don't use bash unless its few
| lines or absolute must.
|
| Its disaster because launching process to achieve simple
| math expression is just ridiculous, besides being slow
| and besides having to terminate it with ;. Its wrong on
| so many levels I don't even know where to start, they
| must have been smoking some nasty things back in time.
___________________________________________________________________
(page generated 2021-04-10 23:02 UTC)