[HN Gopher] Practical Shell Patterns I Use
___________________________________________________________________
Practical Shell Patterns I Use
Author : zwischenzug
Score : 61 points
Date : 2022-01-09 08:44 UTC (1 days ago)
(HTM) web link (zwischenzugs.com)
(TXT) w3m dump (zwischenzugs.com)
| t43562 wrote:
| The one that I find repeatedly useful:
|
| find . -name <some pattern> | { while read F; do <do something to
| file F>; done; }
|
| This lets you do things of almost any level of complexity to
| files or data listed as input and it doesn't require temporary
| files so you can process a lot of things.
|
| e.g. you might use sed -i to alter the files in some way or you
| might find media files of some kind into ffmpeg to convert them.
| chasil wrote:
| If you really had a lot of ffmpeg work to do, then GNU xargs
| can let you run the jobs in parallel, one for each of your
| available CPUs. You can even adjust the number of processes up
| or down by sending the xargs parent the appropriate signals.
|
| https://www.linuxjournal.com/content/parallel-shells-xargs-u...
| bxparks wrote:
| Interesting stuff. One small correction that I noticed:
| $ sudo locate cfg | grep \.cfg$
|
| should contain quotes, like this: $ sudo locate
| cfg | grep '\.cfg$'
|
| because the shell swallows the backslash.
|
| But locate(1) supports globs, so I think this could be simplified
| to just: $ sudo locate '*.cfg'
|
| But on my Ubuntu, /etc/updatedb.conf has PRUNE_BIND_MOUNTS="yes",
| so my /home, on a separate partition, does not get updated. So I
| often resort to: $ find -name '*.cfg'
|
| somewhere in my $HOME directory.
| revscat wrote:
| Or using globbing in zsh: $ echo **/*.cfg
|
| You can even order my most recently modified:
| $ echo **/*.cfg(om)
|
| `setopt EXTENDED_GLOB` needs to be on.
| nuxi wrote:
| You can create a separate locate database for /home if you
| want. Set it up for periodic updates (via cron or other means)
| and then use "locate -d <path_to_db>" for searching.
| honkycat wrote:
| Going to echo: Bash is bad. Bash is a feature-impoverished
| language and belongs in the dustbin. I don't understand why we
| script with one hand tied behind our back.
|
| With Python I can install "click" and get great argument parsing
| and multi-command support with a simple install. Bash's argument
| parsing is much more verbose and arcane.
|
| Sure, it has nice ways to combine tools and parse strings and
| such. However: You could also implement those nice abstractions
| in a higher quality language. So a good thing about bash does not
| cancel out all the bad.
|
| I would LOVE a toolkit that is:
|
| - A single binary
|
| - That parses a single file like make ( but NOT a makefile )
|
| - That uses a well known, powerful language.
|
| - That lets me declare tasks, validations ( required arguments,
| etc ), and workflows as a first-class citizen
|
| - With out of the box `--help` support
|
| - That lets me import other makefile-like files to improve code
| re-use ( both remote and local )
| t43562 wrote:
| I think the features that it has make it the language that it
| is.
|
| If you try to remove the need for combining other programs by
| making everything builtin you wouldn't design it the way it is.
| gmuslera wrote:
| I try to use "smaller" commands instead of more versatile but
| complex alternatives. I'm not sure if I get better cpu/io/etc
| usage by using cut or tr instead of sed or awk (or perl and so
| on) but it is a pattern I followed by decades, at least for
| simple enough commands. And I may have something less to remember
| if I do i.e. rev | cut | rev instead of awk to get the last
| field.
|
| And having something less to remember, and lowering the
| complexity for the reader (that not always be me, or if I am, may
| not be in the same mind state as when I wrote that) is usually
| good. But it also is having predictable/consistent patterns.
| yakshaving_jgt wrote:
| Being able to write a file directly in the shell with heredocs is
| so cool, but I know I'll never use it because it's far quicker to
| just open vim and type with all the commands and verbs and
| motions that I'm used to.
| oandrew wrote:
| One feature I really wish for in shells is something similar to
| Perl's unquoted strings / single line concise HEREDOC syntax /
| raw literal syntax. e.g. $ echo q(no ' escaping `
| required " here) no ' escaping ` required " here
|
| This would make typing sql / json much easier. To my knowledge
| none of the shells implement this. Does anyone know why?
| chasil wrote:
| The POSIX shell was set in stone in the early '90s. The
| standards board actually removed features from the Korn shell
| in order for Xenix-286 or comparable systems to be able to run
| it in a 64k text segment, with clean and maintainable C (ksh88
| is very ugly C).
|
| The standards for the POSIX shell are controlled by the Austin
| group/OSF, and they are not receptive to changes,
| unfortunately.
| oandrew wrote:
| bash 4 supports `|&` as an alternative to `2>&1` which looks
| better in pipelines e.g. $ docker logs container
| |& grep word
| LanternLight83 wrote:
| I was looking for this just the other day, but thought it would
| have been the other way around (&|, consistent with &>), and
| moved on with 2>&1 |
| kstenerud wrote:
| After almost 30 years and tens of thousands of LOC, possibly even
| into 6 figures, I've now thrown in the towel and just use Python
| for all my scripting needs. I've yet to have a single shell
| script that hasn't turned into a maintenance nightmare,
| regardless of "bash best practices" (such as they are).
|
| Bash is a terrible language, and needs to die.
| agumonkey wrote:
| maybe micropython to save some memory and cycles, python for
| the larger scripts if need be
|
| i'm surprised how bash survives.. I think it's part of a few
| traditions that are tied to *nix, just like sysvinit, or raw
| string pipes as IPC .. they'll soon fall in sequence
| SkyMarshal wrote:
| I've occasionally wondered why Python isn't the default
| scripting language on *nix by now. It's almost always installed
| by default, is more capable than bash, is dynamic/interpreted,
| has a repl, and seemingly everything else you would want in a
| Bash replacement. Is it just Bash's momentum, or something else
| that keeps bash hanging around?
| Yen wrote:
| I've also wanted a better shell scripting experience, and
| have bashed [no pun intended] my head against this several
| times. I think some of the major pain points that resist
| adoption of a language like Python or Ruby for in-line shell
| scripting or simple automation is:
|
| * These languages prefer to operate on structured data. If it
| parses from json, you have a much easier time. But, most
| commands you'd invoke from the shell emit unstructured text
| by default. You can deal with this, but it's a pain point,
| and it means any serious scripting starts off with a "here's
| how you parse the output of <foo>", and several rounds of
| debugging.
|
| * The shell's first and foremost usage is a user-facing
| interactive interface to the computer. Most of what you do is
| invoking other programs and seeing their output, and doing
| this is very easy. While python & ruby have REPLs, these are
| mostly focused on trying out language features or testing
| code, not invoking other programs, nor navigating a file
| tree. A lot of shell scripts start as 1-liners that grow and
| grow.
|
| * Invoking other programs: In sh, invoking other programs is
| the primary activity, and it's relatively straightforward -
| you type the name of that program, press enter, and hope that
| it's on your path. In Python or Ruby, it requires importing
| the proper library, choosing the correct command, wrapping it
| and arguments, and ensuring everything escaped correctly.
| [Ruby _does_ have the backticks operator, which does actually
| make a lot of 1-off scripts easy, but this is not a panacea]
|
| * In sh, a lot of the 'utility' programs like cut, sed, awk,
| grep, head, tail, etc., are standing in for the capabilities
| of the language itself. In pure Python or Ruby, you'd do
| these kinds of things with language built-ins. But, that's a
| learning curve, and perhaps a bit more error-prone than "|
| head".
|
| * On top of all that, yes, momentum. If tomorrow you showed
| me a shell replacement for _nix that was_ unambiguously*
| improved in every way, had excellent documentation, community
| support, and was actually pre-installed on every machine, it
| would still take a decade or more before it was really a
| default.
|
| -----
|
| I want it to happen, so I'd never discourage anyone from
| taking a swing. IMO, some of the top-level considerations
| that are necessary for making a successful sh alternative
| are:
|
| * minimize the additional # of characters required to invoke
| a program with arguments, compared to bash.
|
| * Decide which suite of typical utilities should actually be
| built-ins (i.e., things like cd, ls, cp, grep, curl), and
| make those standard library, built-in, without additional
| import or namespacing.
|
| * Focus on an append-style workflow. Functional programing
| styles can kind of help here. Wrapping things in loops or
| blocks is a point of friction.
|
| * An additional highly-desired feature which just isn't in sh
| by default, to overcome momentum. I have no idea what this
| would be. More reliability and better workflow are _nice_,
| but sh is sticky.
| mdaniel wrote:
| Because shell is really good at gluing together commands and
| has reasonably straightforward error management mechanisms
|
| Contrast that with:
| subprocess.run("some_thing with-output")
| subprocess.run("the_next_thing")
|
| which requires that the caller take steps to grab, check, and
| then re-emit any errors (or even just the "normal" output)
| produced by either of those, not to mention the horrors of
| output buffering for long-running processes and the ever-
| present encoding woes since subprocess interfaces as _bytes_
| not _str_
|
| I guess it is the same situation with all programming --
| diligent programmers can make it work, less disciplined users
| inflict pain upon all downstream users of poorly coded
| scripts
| enriquto wrote:
| Common shell tasks (piping, process substitution, create a
| string with the output of a program, creating and redirecting
| output and input to files) are extremely cumbersome in
| python.
| nixpulvis wrote:
| Please show me how to do `ls | grep -i old` in Python?
|
| For a short while I was following
| https://github.com/matz/streem (from the original author of
| Ruby) with interest in possibly being adapted to superseed
| Bash. I have no idea how realistic that actually is.
|
| I also use fish a lot, but it has a lot of it's own problems.
|
| The thing about all these "shells" that really seperates them
| from Python (et all) is that they lookup commands from PATH
| and execute them directly. Anything else is simply not a
| shell language no matter how great it is at scripting.
| justsomehnguy wrote:
| ls | ? name -eq old ls | ? Name -match '^old$'
|
| Any other /property/ of the file (datetime fileds, name,
| extension, path) can be matched woth a string
| representation or an object methods.
|
| But yes, this isn't Python.
| jcranmer wrote:
| > Please show me how to do `ls | grep -i old` in Python?
| for filename in os.listdir(os.curdir): if 'old' in
| filename.lower(): print(filename)
|
| One of the advantages of using a language like Python is
| you don't _need_ to set up a lengthy pipeline of program
| executions just to reformat program output. In my personal
| experience using Python for basic automation scripting, I
| just haven 't really come across any need for me to pipe
| output from one command into another.
|
| The problem I have with a lot of shells is that their
| variable syntax is almost completely broken if you want to
| do anything slightly harder than trivial. A recent example
| I had was trying to write a script that reads a shell and
| executes the last line with some extra arguments. So that
| should be `PROG=$(tail -n1 commands.sh); creduce-script
| "$PROG"`, nothing hard about that... except the last line
| was `clang "-DFOO= " bar.c`.
| nixpulvis wrote:
| I'm not arguing that something like Python is nice for
| when I'm writing scripts out and saving/committing them.
|
| A shell is a REPL designed to use quickly and easily for
| one-off jobs. Shell scripts are also sometimes just
| simpler to string together than referencing a whole new
| language.
| tkot wrote:
| How about Groovy?
|
| ("ls".execute() | "grep -i
| old".execute()).waitForProcessOutput(System.out,
| System.out)
|
| It can also be written as
|
| "ls".execute().or("grep -i
| old".execute()).waitForProcessOutput(System.out,
| System.out)
|
| It's not exactly the same because it uses a thread and a
| buffer instead of an OS pipe but I guess it's close enough.
| nixpulvis wrote:
| Why should I need to explicitly state that I want to wait
| for my process output? I'm in a shell, I put a `&`
| afterwards if I want things in the background. The model
| is synchronous by default.
| josephcsible wrote:
| I wonder why Groovy decided to map "or" to the shell's
| "|". Wouldn't it have made more sense to map "or" to the
| shell's "||" and "pipe" to the shell's "|"?
| tkot wrote:
| I think this is just how operator overloading works in
| Groovy: https://groovy-lang.org/operators.html#Operator-
| Overloading
|
| "Or" is the name of the "|" operator. From the looks of
| it it's impossible to overload "||" in groovy.
|
| By the way the "|" operator and the "or" method both call
| "pipeTo" directly so a third way to write it would be:
|
| "ls".execute().pipeTo("grep -i
| old".execute()).waitForProcessOutput(System.out,
| System.out)
| deckard1 wrote:
| historically, this is exactly the role Perl has played. It's
| also why people keep repeating the nonsense that Perl looks
| like line noise. Because the only Perl code people are
| familiar with are throwaway scripts written by (usually) some
| system admin person who is absolutely _not_ a developer.
| [deleted]
| seanhunter wrote:
| As an aside, "zwischenzug" is a great name. A "zwischenzug" is an
| "inbetween move" in chess that might happen in the middle of a
| sequence, and can often lead to unexpected results if one side
| hasn't been precise in their move order.
|
| https://www.chessjournal.com/zwischenzug/ for example has some
| more info if you're curious.
| pizza234 wrote:
| > ps -ef | grep VBoxHeadless | awk '{print $2}' | xargs kill -9
|
| There's pkill for that :) pkill -KILL
| VBoxHeadless
|
| (the above assumes that the binary is called `VBoxHeadless`)
| cryptonector wrote:
| Generally you might want to use the `-x` argument to `pkill`.
| inetknght wrote:
| kill -9 $(ps aux | grep .exe | sed -r 's/ +/ /g' | cut -d ' '
| -f 2)
|
| I fucking hate wine. I wouldn't use it at all if it weren't for
| steam. And this is the only way to kill a wine process.
| deckard1 wrote:
| let me introduce you to TASK_COMM_LEN $ pgrep
| systemd-journald $
|
| and: $ pgrep systemd-journal 431
| $
|
| Guess how long it took me to figure this out. This affects
| killall, pkill, and pgrep. You can use the "-f" flag with
| pkill/pgrep.
| kragen wrote:
| Or killall.
| gbrown_ wrote:
| I know there are many ways the same thing can be done in the
| shell but there are so many problems here. Please take this as
| feedback and not harsh criticism (I know there's comment section
| on the blog but I'd rather not give my e-mail to yet another
| website). > cat /etc/passwd | sed
| 's/\([^:]*\):.*:\(.*\)/user: \1 shell: \2/'
|
| Besides the needless cat this is clearer in awk, a tool you
| mention just prior. awk '{FS=":"} {print "user:
| " $1, "shell: " $NF}' /etc/passwd > $ ls | xargs -t
| ls -l
|
| Beware files with whitespace in their names. >
| find . | grep azurerm | grep tf$ | xargs -n1 dirname | sed
| 's/^.\///' > > ... > > for each
| of those files, removes the leading dot and forward slash,
| leaving the final bare filename (eg somefile.tf)
|
| No it doesn't it returns the names of parent directories where
| files ending in "tf" are found and where the path also includes
| the string "azurerm". > $ ls | grep Aug | xargs
| -IXXX mv XXX aug_folder
|
| Ew. mv *Aug* aug_folder/
|
| Though now the issue of whitespace in path names is mentioned.
| Another minior point here is that with GNU xargs at least when
| using -I -L 1 is implied, so the original example is equivalent
| to a loop. > $ ps -ef | grep VBoxHeadless | awk
| '{print $2}' | xargs kill -9 pkill VBoxHeadless
|
| You acknowledge to avoid SIGKILL if possible so I don't know why
| you put it in the example. > $ sudo updatedb
| > $ sudo locate cfg | grep \.cfg$ locate '*.cfg'
|
| No idea why sudo is used when running locate, also it takes a
| glob so the grep here can be avoided. > $ grep
| ^.https doc.md | sed 's/^.(h[^])]).*/open \1/' | sh
|
| Now this is just nutty. > This example looks
| for https links at the start of lines in the doc.md file
|
| No it doesn't, it matches the start of the line, followed by a
| character, then "https".
|
| The sed then matches start of the line, followed by a character,
| starts a capture group of the letter "h" followed by a single
| character that is not "]" or ")", closes the capture group and
| continues to match anything.
|
| The example given will always result in "open ht" for any hit.
|
| Then there's the piping to sh. You mention like to drop this to
| review the list. I'd suggest focusing on extracting the list of
| links then pipe to xargs open. If you get a pipeline wrong you
| could unintentionally blast something into a shell executing
| anything which might be very bad. > Give Me All
| The Output With 2>&1 &> stdout-and-stderr
|
| But each to their own. > env | grep -w PATH |
| tr ':' '\n' echo $PATH | tr ':' '\n'
|
| or tr ':' '\n' <<< $PATH
|
| Edit: I know the formatting of this comment is terrible, I think
| quoting the sed example caused the rest of the comment to be
| marked up as italics so I indented it along with the other quotes
| as code.
| vrnvu wrote:
| Thanks for taking the time to explain all of these cases.
|
| While I was reading I noticed some of them being weird but your
| insight was really helpful.
___________________________________________________________________
(page generated 2022-01-10 23:01 UTC)