[HN Gopher] Write Posix Shell
___________________________________________________________________
Write Posix Shell
Author : j3s
Score : 92 points
Date : 2023-03-11 15:08 UTC (7 hours ago)
(HTM) web link (j3s.sh)
(TXT) w3m dump (j3s.sh)
| jmmv wrote:
| I cringe when I hear people (often senior) saying things like:
| "Yikes, a 500-line shell script. We must rewrite it in Python for
| maintainability!!!!1!" -- without even looking at how it has been
| written.
|
| Yes. Shell scripts can be terrible, and most are. But if one
| approaches them thinking that "the shell is a real language and
| must be written as such", with solid principles in mind, it is
| perfectly possible to write clean, maintainable, _and_ testable
| shell scripts.
|
| Oh, and that 500-line shell script probably ends up being a
| 5000-line Python monster anyway.
|
| (Edited to add a couple of links to related stuff I wrote in the
| distant past: 1. https://jmmv.dev/series.html#Shell%20readability
| and 2. https://github.com/jmmv/shtk)
| simias wrote:
| 100% disagree. Standard /bin/sh has terrible error handling
| facility, which on its own is enough to disqualify it as a
| "proper" programming language in my book. Even C is not that
| bad, and C is pretty terrible.
|
| I'm proficient at POSIX shell scripting and use it all the
| time, but every time I discover that a tool I need to maintain
| or, god forbid, improve upon is written in pure shell I
| immediately get a migraine.
|
| The shell is not a real language and must be written as such.
| alganet wrote:
| There's no such thing as "standard /bin/sh". You're talking
| about busybox ash, or dash, or something else that is shipped
| as /bin/sh in your distro.
|
| Take a look at winetricks, or clitest, or crossroot-ng, or
| shellspec. They are modular, easy to read, well written
| pieces of software that are written in shell.
|
| In the 2000s, people used to say that it was impossible to
| write large stuff in JavaScript, and it was not a real
| language. They were wrong, because they only saw browser
| differences and the issues with it, not the great things
| about it. The bourne shell is in a similar state.
| simias wrote:
| I meant POSIX shell or more specifically a loosely defined
| subset of all these shell dialects that is reasonably
| portable in practice.
|
| I'm not saying that it's impossible to do, I'm saying that
| it's very difficult, requires every contributor to be
| extremely well versed into shell coding and the particular
| convention of the codebase, and small issues can easily
| snowball out of control. It's doable, but it's almost never
| worth it IMO. One notable exception is when you need max
| portability with effectively zero dependencies.
|
| >In the 2000s, people used to say that it was impossible to
| write large stuff in JavaScript, and it was not a real
| language.
|
| They were partly right, which is why modern web development
| bears very little resemblance with early 2000's JS hacking.
| Remember that even JQuery only appeared in 2006. And those
| were the dark days of IE6...
|
| But IMO that's not exactly the same thing: the issue with
| Javascript was less with its syntax and more with the very
| poor Javascript API and standard lib (or lack thereof),
| shell scripting is the other way around in a sense: you
| have access to all the power of the OS and all the programs
| in it, but it's the actual programming language that's
| lacking.
|
| Actually a decent language that would transpile to POSIX sh
| would be interesting, now that I think about it.
| alganet wrote:
| The POSIX standard for sh is broken, it was born broken.
|
| For example, all moderns shells support some form of
| `local` for declaring local variables. There are some
| small subtleties but consistent behavior can be achieved
| across interpreters. It's not on POSIX though. Shellspec
| complaints about it (but it shouldn't).
|
| It sounds small, but it's a huge thing to have local
| variables instead of a huge global scope.
|
| There are many other features that are widely available
| but with slightly different interfaces. Kind of what
| happened with ActiveX/XmlHttpRequest in JavaScript.
|
| What is missing is precisely documentation on the subset
| of stuff that _can_ be portable with coding standards,
| polyfills and adapters.
|
| I'm not talking about core utilities here. No `cat` or
| `grep` or nothing like it. Basially, `PATH=''`, only
| builtins are allowed. Even with this constraint, the
| shell is usable and powerful.
|
| Unfortunatelly, most people are well versed in
| bash+coreutils, not raw portable shell. It requires
| learning a few things that are not straightforward at
| first (variable substitution, how the shell forks
| processes, making evals without making a mess, nested
| quoting, etc).
|
| > Actually a decent language that would transpile to
| POSIX sh would be interesting, now that I think about it.
|
| That would be nice. I believe it can be written in shell
| itself, self-hosted and portable. It's hard work though
| (on the same level of writing babel from scratch).
| als0 wrote:
| > Oh, and that 500-line shell script probably ends up being a
| 5000-line Python monster anyway.
|
| This is an important distinction, and probably why I still make
| shell scripts.
|
| I've found utilities like `shellcheck` invaluable for making
| robust scripts and maintaining them.
| sigzero wrote:
| I love shellcheck. You can run it locally or via the web if
| you don't want to install it.
|
| https://www.shellcheck.net/
| jonhohle wrote:
| There are even great unit testing frameworks for testing across
| a variety of shells and versions. This can be integrated pretty
| easily into existing build tools and run the same way the other
| code it may (or may not) support is run.
| HL33tibCe7 wrote:
| > it is perfectly possible to write clean, maintainable, and
| testable shell scripts.
|
| Possible? Maybe. Easy? No. Especially the "testable" part.
|
| The vast majority of engineers do not know how to write clean
| shell, let alone testable shell. There's a reason for this:
| it's extremely difficult and non-obvious to do, and generally
| isn't worth investing the time in learning (you can just use a
| different language instead: perhaps one with support for
| _advanced features_ like... arrays...)
|
| So sure, you can write your 500 line shell script and it may
| well end up being beautiful. But what happens in 5 years when
| someone who isn't as enlightened as you has to maintain it?
|
| (This all without mentioning the worst aspect of shell
| scripting: security. Good luck handling secrets without leaking
| them in one way or another).
| jmmv wrote:
| I see your point, but what happens in your alternate world of
| having to maintain the 5000-line Python version 5 years down
| the line? You face the same issues you describe. The people
| that wrote the original beautiful code aren't there anymore,
| and whoever comes touching the code "messes it up". Most
| Python "scripts" I have had to deal with that had been around
| for a while were pretty terrible as well.
| HL33tibCe7 wrote:
| I don't think you do see my point: my point is that writing
| clean, testable, maintainable Python is both easier and a
| more widespread skill than writing equivalently clean,
| testable and maintainable shell. Both of these factors will
| mean that the Python remains maintainable for longer.
| _dain_ wrote:
| First, I doubt that the SLOC conversion ratio between sh
| and Python is really as high as 10x.
|
| Second, yes, I do think the 5000 line Python script would
| be more maintainable than than the 500 line sh script. I'd
| choose real types and data structures any day of the week.
| fbdab103 wrote:
| Bad code is bad code, regardless of language. At least with
| Python, I more or less know all of the idioms and can at a
| glance understand what the flow of a program is trying to
| accomplish. Shell has a lot of esoteric voodoo syntax to
| replicate better string/path/array handling that is just a
| given in other languages.
| waynesonfire wrote:
| > Possible? Maybe. Easy? No. Especially the "testable" part.
|
| a testable shell script? Never seen one.
|
| Thinking about scripts I've read in the past, I remember
| seeing Jason Donenfeld's bash script for wireguard-wg and
| thinking how productive and readable it was,
|
| https://github.com/WireGuard/wireguard-
| tools/blob/master/src...
| eviks wrote:
| > But if one approaches them thinking that "the shell is a real
| language and must be written as such", with solid principles in
| mind
|
| Then one would simply choose a proper language that had more of
| those solid principles laid at its foundation
| enriquto wrote:
| Unix shell _is_ a proper language and has very solid
| principles beautifully laid at its foundation by Douglas
| McIlroy.
| chasil wrote:
| I understood that Stephen Borne's love of ALGOL was the
| chief structural influence of the POSIX shell (so much so
| that he used the C preprocessor to turn his source code
| into fake ALGOL).
|
| "Stephen Bourne's coding style was influenced by his
| experience with the ALGOL 68C compiler that he had been
| working on at Cambridge University... Moreover, - although
| the v7 shell is written in C - Bourne took advantage of
| some macros to give the C source code an ALGOL 68 flavor.
| These macros (along with the finger command distributed in
| Unix version 4.2BSD) inspired the International Obfuscated
| C Code Contest (IOCCC)."
|
| https://en.m.wikipedia.org/wiki/Bourne_shell
| wizhi wrote:
| Do you have some writing on this? I'd love to read it.
| sureglymop wrote:
| I cringe everytime my manager tells me not to write shell
| scripts but javascript instead. There is ShellCheck, Bats and
| there are extensions for e.g. Visual Studio Code or Jetbrains
| IDEA that make writing shell and bash scripts a pleasure. I
| find it generally much more productive to write a shell script
| because I already know all the gnu/unix userland tools. And i
| know this shell script will be portable and still running
| decades from now unless some version specific node script with
| dependencies.
| j3s wrote:
| shell is loose and offers a lot of power per line, so it's
| pretty easy to make a spaghetti nightmare. but it's very
| possible to write nice shell.
|
| i dunno about testing it, i feel like that's getting pretty
| close to the barrier of shell's utility
| oneshtein wrote:
| Bash is turing-complete, so it's possible to write automated
| test cases in bash. Example:
| https://github.com/vlisivka/bash-modules/blob/master/bash-
| mo...
| jbverschoor wrote:
| A program in any language can be maintainable when written
| properly and maintained by someone who is capable, diligent,
| and has a good test setup.
| d0mine wrote:
| > Oh, and that 500-line shell script probably ends up being a
| 5000-line Python monster anyway.
|
| Agreed on avoiding rewrites if possible but 1/10 ratio is
| laughable.
| throw0101c wrote:
| > _Oh, and that 500-line shell script probably ends up being a
| 5000-line Python monster anyway._
|
| The _dehydrated_ ACME client is 2400 lines of bash /zsh:
|
| * https://github.com/dehydrated-io/dehydrated
|
| And its external dependencies are OpenSSL and cURL. The
| _acme.sh_ shell ACME client is 8000 lines of shell:
|
| * https://github.com/acmesh-official/acme.sh
|
| The official Let's Encrypt client is written in Python, and the
| core 'executable' is much longer, and in addition it pulls in a
| boatload of dependencies:
|
| * https://packages.debian.org/bullseye/python3-certbot
| ori_b wrote:
| I'm not sure why -- the one shipped with 9front is less than
| a thousand lines of C, and supports both DNS and HTTP
| challenges:
|
| https://git.9front.org/plan9front/plan9front/HEAD/sys/src/cm.
| ..
|
| http://man.9front.org/8/acmed
|
| The ACME protocol isn't that complicated.
| m1keil wrote:
| The question is if the script outgrew its original intended
| lifespan.
|
| In my opinion, shell scripts should be used either as glue or
| due to portability reasons. Rewriting a perfectly good and
| working script just because makes no sense.
|
| So if the script started as some quick and easy way to deploy
| to prod but slowly becomes a massive cli "one tool to rule them
| all" kind of thing.. maybe it's time to change course.
| pjmlp wrote:
| Do you a good anecdote regarding shell scripts?
|
| Long time ago some dude created a couple of korn shell scripts
| on Aix to automate file uploads and ETL related tasks.
|
| With time that work moved from Aix into Red-Hat Linux, still
| korn shell, with the adaptations from Aix into GNU/Linux
| command line parameters and such.
|
| I got tasked to port them into Java, because the team that got
| the job to maintain the servers lacked UNIX experience, and
| they thought selling the porting project was a good idea.
|
| Feedback from customer, slower than those scripts and not as
| easy to change, what a surprise.
| kps wrote:
| I use Korn shell (ksh93) for quick scripts. Besides arrays
| and such (it's approximately a superset of bash, since bash
| copied some but not all of ksh), the killer feature is a
| declarative extended `getopts` that makes it trivial to
| provide even the quickest and dirtiest script with short/long
| option processing with full help, so that the purpose and
| usage of the script remain comprehensible five days or five
| years later. https://gist.github.com/kpschoedel/91fdcfa934111
| f3846efb5034...
| jamal-kumar wrote:
| From the article: consider this sentence:
| "arrays do not exist in posix shell" ...
| "NOOO!" you shout, because you like data structures a LOT
| "now THAT'S a good reason to use bash! bash supports arrays!"
| absolutely not! i believe that you should use shell
| if your problem is: - small and scoped (~200 lines of
| shell or less) - unlikely to increase in size and scope
| as time goes on - not very complex if you
| find yourself desiring arrays, good error handling,
| static typing, structs, etc, then your problem should be
| solved using a different language, not shell.
|
| I tend to agree with this heuristic.
|
| It's also why you see people bothering to use perl 5 at all in
| this day and age. It just happens to be preinstalled on a
| decent majority of *nix systems.
| djbusby wrote:
| Perl and Python are part of LSB
|
| https://refspecs.linuxfoundation.org/lsb.shtml
| mananaysiempre wrote:
| So is the RPM format. I like LSB particularly as a
| reference for obscure ABI details, but I don't think
| anybody actually builds either systems or applications to
| it.
| hedora wrote:
| Yeah, but perl 5 hasn't broken compatibility (as far as
| I've noticed) since 2000 or so. Python breaks compatibility
| every week, in practice. Does LSB include some sort of
| magical frozen in time version of python 2 or something?
| temac wrote:
| LSB support has been mostly dropped by some distros (e.g.
| Debian dropped support in 2015)
| mananaysiempre wrote:
| A young traveller came to visit Emperor Sh, and found him
| sitting in his sparsely furnished temple.
| "Emperor Sh," he said, "I am told you are the greatest
| scholar of shell that the world has known."
| Emperor Sh made no reply. The traveller continued.
| "I have come to ask your advice. I am thinking of developing
| a character-based graphing tool. It will interactively
| change the plot based on key presses. Which shell
| commands should I use?" "Don't do it in shell,"
| said Emperor Sh, curtly. The young traveller was
| confused. He tried again. "Well... I am also working on
| a database audit script. It needs to verify that certain
| characters do not appear in any fields in several
| tables. What should--" "Don't do it in shell,"
| interrupted Emperor Sh. The traveller began to
| despair. "I have journeyed one thousand miles, tried
| thirteen distributions of my operating system, and waded
| through hundreds of badly-written manual pages," he
| cried, "and now I have finally come to Emperor Sh, the
| greatest shell programmer in the world, and I am told to use
| no shell at all! Perhaps I should do what my Python
| user friends told me to do, and just pipe together some
| scripts in venv environments!" "Good idea," said
| Emperor Sh. "Do it in shell." Enlightenment
| crushed down on the young traveller and he bowed to the
| Emperor, sobbing with reverent joy.
|
| https://sanctum.geek.nz/etc/emperor-sh-and-the-traveller.txt
| jbverschoor wrote:
| The reason I don't like shell scripts are spaces and the weird
| constructs sometimes.
|
| I never got into Perl, but I understand why it was popular.
|
| Sure it's possible, but it's therefore also brittle, when
| someone touches it and is not a complete expert.
|
| Shell is nice not because of the shell language, but the ease
| of executing and chaining other programs.
| pushedx wrote:
| The shell provides an interface to the filesystem and
| concurrent pipelines like no other readily-available language.
|
| Many tasks are more concisely expressed and readable in shell
| than in any other language.
| lapinot wrote:
| The fact that posix shell is somehow state of the art of
| concurrent pipeline and filesystem interface is really more a
| critique of state of the art than a praise of shell.
|
| I'm doing programming language research and it's just
| infuriating how _retrograde_ shell is. Its design is a fossil
| of history, and boy did our understanding of what makes a
| good language evolve. As a proof of this look at the jq tool.
| It is so powerful and clean. The implementation is simple and
| reasonably fast. It has been designed by a researcher from
| functional languages (an ocaml guy). _This_ is a way to build
| a language around pipelining. At a lesser level, look at how
| moving from vimscript to lua in neovim has brought so many
| high-quality new plugins, look at how many people now tinker
| with low-level stuff and write high-perf libraries now that C
| /C++ is not the only choice. Surface language and having very
| little very orthogonal concepts is important.
|
| Some core implementation patterns for PL are very old, but
| there have been lots of simplifications and rationalizations
| in surface language. Error handling, values hygiene, scoping,
| simple datastructures. Posix shell might have been a prowess
| in the 70s, now it is mostly a hack and this is a fact. Let's
| be a bit less superficial when honoring the unix mantras and
| dream about something better (making a better shell for its
| core ideas(!), that is, handling processes and fds).
| lozenge wrote:
| Powershell is the new shell.
|
| As it passes objects the filtering, sorting, joining is
| generic across files or any other data type.
| chasil wrote:
| And it is so wonderously mistaken in assigning a boolean
| type to $?.
|
| I really cannot fathom how this could be, as Microsoft
| bundles curl, which has nearly 100 exit codes on various
| types of failure.
|
| Whoever made this language decision needs to explain
| themselves.
| _dain_ wrote:
| The oil shell is an attempt to move forward:
|
| https://www.oilshell.org/
| yjftsjthsd-h wrote:
| Yep; shell isn't the best tool for every job, but the things
| it's good at, it's _really_ good at.
| ra1231963 wrote:
| This is terrible advice. Senior people say don't do this
| because they've seen it done and it's an absolute nightmare
| (I've seen it happen multiple times).
|
| It's untyped, untestable, arcane, and unmaintabale. Even the
| link says don't do this. Just don't.
|
| Congrats on having the chops (and neckbeard) to pull this off.
| If you want to do it in a personal project, more power to you.
| But it doesn't scale at organizations.
| hedora wrote:
| Python is untyped and untestable. The only way I've seen
| people use it is by importing whatever libraries were popular
| in whatever year they wrote it, so it's also arcane.
|
| Also, to understand someone else's python, you have to read
| at least 10x more lines of line noise, so it somehow manages
| to be less maintainable than shell.
|
| Concrete example: Every python code base I've encountered has
| some function named _exec_ that poorly reimplements shell job
| control and argument passing. As the python code bases grow,
| they end up with more than one of these, all of which have
| different bugs.
|
| I usually discover this about 6 hours after the script has
| been sent to me, since someone who has heard the words
| "supply chain vulnerability" locked down whatever machine I
| need to run the script on.
|
| The last ten times this has happened (at multiple companies,
| with multiple teams) it ended up being faster to figure out
| what shell commands the python script wanted to invoke, port
| a few things to _jq_ and then reimplement the rest in a mix
| of shell and perl.
|
| The result is always 1-10% the lines of code of the python,
| and actually runs without downloading an entire python
| distribution, dependency manager, and hundreds of broken
| packages at runtime.
| _dain_ wrote:
| >Python is untyped and untestable.
|
| Completely untrue. You're conflating static typing with
| strong typing. And pytest is a great test framework.
|
| And you're recommending _shell_ for better typing and
| testability? The language where everything is a string,
| there are no proper arrays, and functions don 't have local
| variables? You have to be trolling.
| oweiler wrote:
| On top of that, fn in Bash can only return their status
| code. Yes, command substitution exists but is full of
| footguns.
| jmmv wrote:
| Anyone writing code in a language they don't know will write
| nightmarish code. No matter if it is shell, Python, Go, or
| what have you. I have seen it happen multiple times as well,
| with all of these languages. And /that/'s the problem.
|
| So, it depends on the context. "Organization" is a very broad
| term. If the team you are in is familiar with the shell, and
| the shell is the right tool for the job, there isn't that
| much of a problem.
|
| I'm not arguing for writing large shell scripts when there
| are better alternatives. But, sometimes, it's the right
| choice and it can be written properly. And even if it
| isn't... 500 lines of poorly written shell aren't that many
| in any case, and they aren't that bad compared to 5000 lines
| of poorly written Python for example.
| kaba0 wrote:
| I fail to see why you think it would take more lines to do
| in Python.
|
| And the problem with bash is similar to C: many people
| think they can write correct C code, but as it has been
| shown plenty of times, that doesn't really work out in the
| end. Languages have different tradeoffs and bash really has
| no redeeming qualities, imo.
| hedora wrote:
| I'll bite. Show me the equivalent of this in python,
| including error handling, parallelism, disk footprint,
| and real-time status output: #!/bin/bash
| -eux -o pipefail TMPDIR=$(mktemp -d
| tempdirXXXXX) trap "rm -rf $TMPDIR" SIGINT SIGTERM
| ERR EXIT ( cd $TMPDIR foo |
| grep x > foo.out & bar > bar.out &
| baz $TMP | tee baz.out & ) wait
| tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh
| server.com bash -c "cat - > out.tag.zstd"
| EuAndreh wrote:
| You can forego the final "bash -c" by leveraging dd(1) to
| avoid the redirection problem: ... |
| ssh server.com dd of=out.tag.zstd
| EuAndreh wrote:
| dd(1) reads from STDIN by default unless given an "if="
| parameter, and writes to STDOUT by default umless given
| an "of=" parameter.
| hedora wrote:
| Thanks! Will definitely use that moving forward.
| _dain_ wrote:
| $ shellcheck myscript Line 1:
| #!/bin/bash -eux -o pipefail ^-- SC2096 (error):
| On most OS, shebangs can only specify a single parameter.
| Line 4: trap "rm -rf $TMPDIR" SIGINT SIGTERM ERR
| EXIT ^-- SC2064 (warning): Use
| single quotes, otherwise this expands now rather than
| when signalled. Line 6:
| cd $TMPDIR ^-- SC2086 (info): Double
| quote to prevent globbing and word splitting.
| Did you mean: (apply this, apply all SC2086)
| cd "$TMPDIR" Line 9: baz
| $TMP | tee baz.out & ^-- SC2086 (info):
| Double quote to prevent globbing and word splitting.
| Did you mean: (apply this, apply all SC2086)
| baz "$TMP" | tee baz.out & Line 13:
| tar -cf - -C $TEMPDIR | zstd -6 -z - | pv | ssh
| server.com bash -c "cat - > out.tag.zstd"
| ^-- SC2086 (info): Double quote to prevent globbing and
| word splitting. ^-- SC2153 (info):
| Possible misspelling: TEMPDIR may not be assigned. Did
| you mean TMPDIR? Did you mean: (apply this,
| apply all SC2086) tar -cf - -C "$TEMPDIR" | zstd
| -6 -z - | pv | ssh server.com bash -c "cat - >
| out.tag.zstd"
| mananaysiempre wrote:
| > I fail to see why you think it would take more lines to
| do in Python.
|
| Because Bourne shell is not a Python-class language like
| Ruby or perhaps even Scheme or AWK are. Things that are
| easy in shell can be hard in Python and vice versa:
| awkward wrangling of subprocess.Popen in Python can be a
| single line of shell; simple tree munging with lxml in
| Python can easily be a hundred or more lines of wasteful
| xmlstarlet invocations in shell. Parallel processing,
| even without the heavy artillery of make or GNU parallel,
| is easy in shell but hard in Python. Structured data is
| hard in shell but easy in Python.
|
| (Similarly, I wouldn't use ML or Haskell to write a web
| scraper, and I wouldn't use Python to write a compiler.)
| patrakov wrote:
| Regarding parallel processing, I have to disagree. It is
| bad in shell, because it makes best practices such as
| "set -e" impractical. The usual failure mode is a script
| that does not stop correctly on SIGTERM, and its children
| have to be hunted down and killed separately.
| mananaysiempre wrote:
| That's fair.
|
| Perhaps a better way to put it would be that sloppy
| concurrency is easy. This doesn't sound all that good,
| but it's easy enough that I find myself parallelizing
| one-off things in shell that would be such a hassle to do
| nonsequentially in Python (compared to the magnitude of
| the problem) that the option wouldn't even surface in my
| brain. (If I rewrite the shell script in Python later,
| the rewritten version is often sequential.)
|
| The shell error handling story for "run stream processing
| steps in lockstep" parallelism is much better than for
| "spawn a bunch of identical children" parallelism,
| though, and that's often enough.
| mattpallissard wrote:
| > Anyone writing code in a language they don't know will
| write nightmarish code. No matter if it is shell, Python,
| Go, or what have you. I have seen it happen multiple times
| as well, with all of these languages. And /that/'s the
| problem.
|
| Came here to say this.
| meindnoch wrote:
| Thanks, but no thanks. I'll stick with bash. It's ubiquitious
| enough for my purposes, and I can't live without =~ and <().
| ape4 wrote:
| I wonder of shellcheck could suggest when its time to rewrite in
| another language.
| kaba0 wrote:
| if (lineCount >= 3) { return true; }
| mastax wrote:
| For legacy reasons, my filesystem is full of files and
| directories with spaces. Are they impossible to deal with in
| shell? No. Quoting, print0, it's not even hard. But it's like
| taking a Sunday stroll through an unburied minefield. I'd rather
| walk in the park.
| throw0101c wrote:
| But also please consider using Shellcheck if you do:
|
| * https://www.shellcheck.net
| jamal-kumar wrote:
| It's such an essential tool. Always such a beast to install it
| though, definitely keep it off any servers.
|
| It's probably saved myself from weird hard to see mistakes so
| many times I wouldn't be able to count them because if I didn't
| have that tool I might not even see it until runtime, which is
| a terrible place to discover an error.
| fbdab103 wrote:
| >Always such a beast to install it though, definitely keep it
| off any servers.
|
| I do not follow. Isn't it just a static executable? I have
| only ever apt-get installed, but I am unsure where the
| complication rests, unless you are compiling from source.
| blueflow wrote:
| Shellcheck is useful if you are learning shell, but it gets
| really annoying over time with its amount of false positives.
| Its still a static checker that does not understand context.
| I've seen too many scripts where half of all comments lines
| were shellcheck disables.
| return_to_monke wrote:
| I'd rather have false lints than miss lints
| codeflo wrote:
| One important con that isn't mentioned: It's so unnecessarily
| confusing how to write a shell script that can properly deal with
| whitespace, dollar signs or other special characters in
| filenames.
|
| Of course it's possible and learnable. But even if you know the
| tricks, it's tedious, and human error prone. I had a team mate
| (many years ago) who used to test the local admins at any new
| workplace by putting files names named "$HOME" and similar in his
| home directory. He claimed it wasn't that rare for this to crash
| some backup script.
|
| Yet this isn't inherently hard. In fact it's trivial in literally
| any "proper" programming language, no matter which one you end up
| picking. I get why shell is "scripting of last resort", I
| regularly write shell scripts myself. I don't get why people
| would choose it over anything else where choices are available.
| fbdab103 wrote:
| >putting files names "$HOME" and similar in his home directory
|
| If you do this, the fallout is on you. Does he also make files
| called `~`?
| Kwpolska wrote:
| The backup script's job is to back up all files, not crash
| fatally because someone used a dollar sign, a tilde, or a
| space, or `rm -rf` in a file name. The sysadmin's job is to
| write robust tools, not expect users to understand arcane
| UNIX stuff.
| fbdab103 wrote:
| You are right of course. Just my knee jerk reaction that if
| someone is deliberately creating such files, they know
| exactly how much peril they are creating for themselves.
| Owing to so many years on the terminal, I still hesitate to
| name files with a space - a practice which the uninitiated
| find odd.
| joshlk wrote:
| Shell has its place as a glue language between commands. I tried
| writing a good CLI in bash this week and yikes it was hard. The
| lack of a package manager and good libraries really hurt. If you
| could "pip" install bash libraries like an argparse equivalent it
| would help a lot
| abathur wrote:
| https://t-ravis.com/post/shell/no_look_no_leap_shell_with_ni...
|
| (but of course, that's it's own portability/bootstrapping
| problem if you need genuine internet-rando-level portability)
| abathur wrote:
| The post I linked focuses on standalone executable
| dependencies, but the same concept applies to libraries.
|
| If you want a quick example, you can look at how:
|
| - shellswain depends on a little trap-namespacing library
| called comity: https://github.com/search?q=repo%3Aabathur%2Fs
| hellswain+comi...
|
| - comity in turn depends on a bash events library called
| bashup.events: https://github.com/search?q=repo%3Aabathur%2Fc
| omity+events+p...
|
| Early last year I also wrote a little more generally about
| the idea of being able to finally build this shell code-
| sharing ecosystem in https://t-ravis.com/post/shell/neighborl
| y_shell_with_bashup....
| kristopolous wrote:
| I'm a big fan of not posix but instead modern bash and to all the
| complainers about dash and ash, I say "tough cookies".
|
| Bash has arithmetic, dictionaries, regex matching, nice string
| manipulation, co-processes, TCP pipes built in, it's pretty nice.
|
| However, bash has very low tolerance for incompetency. It's a
| brutal friend. It's cooking dinner with a pocket knife. You gotta
| actually read the docs and be sincere in your studies.
|
| I was around when supporting Solaris, HPUX, AIX, and Irix was a
| real concern. Those times sucked and those platforms are so minor
| I stopped caring about them except for enthusiast projects many
| years ago. (And I still actually have an hpux machine)
|
| Sometimes I'll even use zsh. It can even do floating point.
|
| Here's some example of a modern tool I have written for a subject
| I call "music discovery"
|
| https://github.com/kristopolous/music-explorer/tree/master/t...
|
| You'll see many languages in there but you won't see posix sh.
| It's slow, it's clunky, it's a pain in the ass.
|
| If you don't like my practice then I guess don't use it. I've
| been using/developing these particular tools nearly every day for
| over 3 years and it works well for me.
|
| I'm not going to say bash is awesome but it's pretty great for
| programming.
|
| I use zsh as my interactive shell though, it's miles ahead of
| bash in that department. Watching people who _really_ know what
| they 're doing in zsh is a total mindfuck.
| chasil wrote:
| Dash is 4x faster and 10x smaller than Bash. Bash's own manual
| page concludes that it's "to big and too slow."
|
| I have applications where I need those properties, and I am not
| shy in selecting the correct tool.
|
| Bash's POSIX mode also does some wacky things when a script
| transitions in/out that encourage me to avoid it.
|
| https://unix.stackexchange.com/questions/148035/is-dash-or-s...
| cleanchit wrote:
| > It can even do floating point.
|
| Is this a bait post.
| User23 wrote:
| > I was around when supporting Solaris, HPUX, AIX, and Irix was
| a real concern.
|
| Back in the 90s it was surprising to not find gnu userspace
| installed on the systems I used.
| kristopolous wrote:
| About 1/3 of the debug sessions involved looking at the
| screen peeved going
|
| "yes, obviously I meant gsed and ggrep you dumbass."
|
| You go through the man page of the vendor supplied version
| like "What do you mean you don't support feature x? This
| can't be real. I swear I'm being recorded right now. A bunch
| of people are sitting around a large conference table in
| mountain view right now laughing at my pain"
|
| Eventually I just read them _as parody_. Things like this can
| be really funny if you allow them to be.
| nailer wrote:
| Remember how Solaris would show disk usage in ... IIRC
| multiples of 512K? And the awful package manager?
|
| And all they'd do was talk about kernel debugging like that
| was a daily activity people needed help with.
| ever1337 wrote:
| I think the real point is that most of the time when someone
| writes a bashism, it's unnecessary. You don't really lose
| anything 99% of the time by writing posix shell. If you need
| bash, write bash, and just change the shebang.
| bccdee wrote:
| I think real arrays are the exception to this rule. They're
| the only non-posix shell feature I use consistently.
| Otherwise, whenever you want to gather a list of strings or
| filenames and then iterate through them, you need to worry
| about delimiters and splitting and parsing and IFS and that's
| simply not worth the dubious benefit of not including a bash
| shebang.
|
| If you really need to run the script on alpine, just install
| bash. If you're _really_ space-constrained, then sure, go
| back and refactor the arrays into newline-delimited strings
| or whatever, but I 've never had to do that and I doubt I
| ever will. Getting to use arrays means my scripts can avoid
| dealing with the horrible quirks of shell tokenization, and
| that's 100% worth it to me.
| lelandfe wrote:
| > people who really know what they're doing in zsh
|
| Any examples of cool stuff you can do in zsh?
| gavinhoward wrote:
| I haven't seen anyone mention one important reason for writing
| POSIX shell over other shells. The article implies it, but never
| says in directly: portability.
|
| I wrote a 2100 line monster of a POSIX shell script. [1]
|
| The purpose was entirely for portability. That script allows me
| to use a straight POSIX Makefile for building my bc. The
| combination of the two means that my bc builds without
| modification on any semi-POSIX system.
|
| That said, it was hard. I basically had to constrain myself to
| POSIX utilities and still implement a template language. I got it
| done, and it was worth it, but if you don't have to, don't do it.
|
| [1]:
| https://git.gavinhoward.com/gavin/bc/src/branch/master/confi...
| nailer wrote:
| I used to be one of the write .sh people. Now I just ensure my
| shells scripts end in .bash and don't insist people limit
| themselves to an effectively unmaintained standard.
| mogoh wrote:
| My reason for staying away from posix-shell as much as possible
| is that it has so many ways to shoot itself in the foot. This is
| also only indirectly touched in the article. I'd rather write 10x
| as long Python scripts than to shoot myself in the foot with
| posix-shell again.
| jraph wrote:
| Many people advise against using shell scripts for non trivial
| stuff. Is there a language as convenient as shell for calling and
| piping external programs but with proper typing and error
| handling?
|
| People advise using Python but I don't see myself using
| subprocess and Popen for gluing programs together and end up
| writing shell scripts anyway, I'm pretty comfortable with posix
| shell.
| [deleted]
| grumblehound wrote:
| I tried https://plumbum.readthedocs.io/en/latest/ recently and
| it was quite good. You might want to write a wrapper function
| depending on what error or newline splitting behaviour you
| want.
| nickjj wrote:
| > I'm pretty comfortable with posix shell.
|
| It's ok to trust yourself.
|
| Write 1 file shell scripts for convenient zero dependency
| command line tools that will work on most systems. If things
| start to get rough due to shell limitations, then Python
| scripts without dependencies is another 1 file solution that's
| quite portable. You can even mix and match (mostly use shell
| and call Python from it for the complicated bits as a last
| resort).
|
| I wrote about this exact topic recently around trying to deal
| with string splitting in Bash:
| https://nickjanetakis.com/blog/splitting-strings-by-a-delimi...
|
| The takeaway is that for 99% of cases using IFS or Bash's
| string replace in a variable is great but if you have strict
| parsing requirements then maybe going with Python is worth it.
| The post links to a massive SO answer that demonstrates how
| string manipulation in shell scripting is hard.
| 8n4vidtmkvmk wrote:
| zx is ok. it's just a node package that makes working with
| subprocesses easier. but js does good async.
| BiteCode_dev wrote:
| If you code in Python, your probably should use the language as
| much as possible and avoid calling shell commands.
|
| E.G:
|
| - manipulate the file system with pathlib
|
| - do hashes with hashlib
|
| - zip with zipfile
|
| - set error code with sys.exit
|
| - use os.environ for env vars
|
| - print to stderr with print(..., file=...)
|
| - sometimes you'll need to install lib. Like, if you want to
| manipulate a git repo, instead of calling the git command, use
| gitpython (https://gitpython.readthedocs.io/en/stable/)
|
| But if you don't feel like installing a too many libs, or just
| really want to call commands because you know them well, then
| the "sh" lib is going to make things smoother:
|
| https://pypi.org/project/sh/
|
| Also, enjoy the fact Python comes with argparse to parse script
| arguments (or if you feel like installing stuff, use typer). It
| sucks to do it in bash .
|
| If what you need is more build oriented, like something to
| replace "make", then I would instead recommend "doit":
|
| https://pydoit.org/
|
| It's the only task runner that I haven't run away from yet.
|
| Remember to always to everything in a venv. But you can have a
| giant venv for all the scripts, and just she-bang the venv
| python executable so that it's transparent. Things don't have
| to be difficult.
|
| Finally, sometimes it's just easier to do things in ipython
| directly. Indeed, with a good PYTHONSTARTUP script, you don't
| have to import everything, you get superb completion+help, and
| you can call any bash command by just doing "!cmd". In fact,
| the result can be store in a python variable. Also you can set
| autocall so that you don't have to write parenthesis in simple
| function calls.
| remram wrote:
| What's wrong with Python and the subprocess module? It is
| exactly what I advise.
| travisjungroth wrote:
| If you have a single command it's fine. Not _great_ because
| of the list syntax, but fine. But I've found when I'm doing
| shell commands and using Python as glue, the Python takes up
| most of the script while doing barely any of the work.
|
| Don't get me wrong, I like Python more and can barely write
| shell (ChatGPT/Copilot has made it easy). But it is not as
| convenient for calling and piping commands as shell.
| loevborg wrote:
| Shell scripting's best kept secret is babashka
| (https://babashka.org/).
|
| - It's a flavor of Lisp (Clojure, specifically), a language
| whose flexibility makes it ideally suited for gluing together
| programs and working with data
|
| - Compare to alternative scripting shells, babashka is very
| pragmatic and 100% production ready today, as it's built on top
| of Java, GraalVM and Clojure
|
| - Even though it's Java under the hood, it's FAST (`time bb -e
| '(+ 1 1)' -> 0.014 ms)
|
| - Support for filesystem operations (e.g. globbing) is as good
| or better than Bash
|
| - Working with subprocesses is better than in bash, because
| there are fewer gotchas
|
| - Piping is easier in bash, but most of the time you _don't_
| really want to use pipes. `cat /etc/fstab | grep /usr` should
| be a set of function calls, not subprocesses
|
| Give it a try, you won't regret it
| User23 wrote:
| Perl has the attributes you describe.
| bqmjjx0kac wrote:
| Once I started running Shellcheck on my scripts, I felt like
| they were full of footguns waiting to go off.
| _dain_ wrote:
| I'm not convinced.
|
| _> when should i use shell?_
|
| _> in short; when the problem you're solving is small, well
| defined, and unlikely to change, consider shell._
|
| If you have to caveat it this much, it's a pretty weak statement.
| _Any_ language works when the problem is small, well defined, and
| unlikely to change!
|
| Strict POSIX compatibility is something of a chimera. You don't
| just need to read the POSIX shell spec, because you're not just
| programming in POSIX shell, you have to use the coreutils as well
| to get anything done. Do you remember which flags on grep are
| POSIX-only, and which are extensions provided by the GNU version?
| And then the same for sed, ls, and the rest of the UNIX zoo?
| There's an order of magnitude more shit you have to remember and
| worry about.
|
| Needing strict POSIX compatibility is an increasingly niche use-
| case. I find it baffling that, on the one hand, we have these
| modern software isolation and distribution systems like
| containers and Nix and so on, technologies that are meant to let
| us run anything anywhere reliably, and then we forget we have
| these marvels and wear the POSIX hair-shirt anyway. Why can't you
| just include a better language in the container? A container is
| supposed to contain things! Stop hitting yourself in the face
| repeatedly!
|
| Finally if you somehow really do need to live in the POSIX cave,
| you don't need to use shell. POSIX also specifies a much saner
| language: awk.
| blueflow wrote:
| > Do you remember which flags on grep are POSIX-only, and which
| are extensions provided by the GNU version?
|
| Use the posix man pages as reference. `man 1p grep` and you
| only see the portable options. Its only one step away from
| using the regular manpages.
| abathur wrote:
| til :)
| Karellen wrote:
| Worth noting that `man-pages-posix` is not under any kind of
| Free/Open Source/CC-style license, so may not be available by
| default for your OS.
|
| e.g. in Debian you have to enable the `non-free` section to
| make them apt-installable.
| fpoling wrote:
| My rule of thumb is that one needs to switch to other language
| when one needs maps or complex array operations. Arrays for
| proper processing of arguments or storing a sequence of results
| are doable in a POSIX shell with couple of utilities.
| RcouF1uZ4gsC wrote:
| Rust may actually be a solution in this space. It is better typed
| and more testable than Python. It has fewer issues with
| environmental dependencies due. It is easy to install and build.
|
| There is ongoing work to rewrite the coreutils in Rust, and if
| they were exposed with an ergonomic library interface, would
| provide a lot of the value of shell with much better safety.
| maleldil wrote:
| Rust is _a lot_ slower to develop than sh/bash/Python, though.
| If your script is in reality a semi-complex CLI tool, then Rust
| is great, but if it's a 1k line shell script, Python/Ruby/JS
| (especially with Deno) are better options.
| k3vinw wrote:
| For sure! I used to think that python was the next logical step
| for a growing project that started out life as a shell script,
| but now I think we have even better choices like rust.
___________________________________________________________________
(page generated 2023-03-11 23:01 UTC)