[HN Gopher] Bash Patterns I Use Weekly
___________________________________________________________________
Bash Patterns I Use Weekly
Author : gcmeplz
Score : 149 points
Date : 2021-11-23 15:16 UTC (7 hours ago)
(HTM) web link (will-keleher.com)
(TXT) w3m dump (will-keleher.com)
| stewartbutler wrote:
| I've always had trouble getting `for` loops to work predictably,
| so my common loop pattern is this: grep -l -r
| pattern /path/to/files | while read x; do echo $x; done
|
| or the like.
|
| This uses bash read to split the input line into words, then each
| word can be accessed in the loop with variable `$x`. Pipe
| friendly and doesn't use a subshell so no unexpected scoping
| issues. It also doesn't require futzing around with arrays or the
| like.
|
| One place I _do_ use bash for loops is when iterating over args,
| e.g. if you create a bash function: function
| my_func() { for arg; do echo $arg
| done }
|
| This'll take a list of arguments and echo each on a separate
| line. Useful if you need a function that does some operation
| against a list of files, for example.
|
| Also, bash expansions
| (https://www.gnu.org/software/bash/manual/html_node/Shell-Par...)
| can save you a ton of time for various common operations on
| variables.
| jolmg wrote:
| > I've always had trouble getting `for` loops to work
| predictably, so my common loop pattern is this:
|
| The problem with the pipe-while-read pattern is that you can't
| modify variables in the loop, since it runs in a subshell.
| edgyquant wrote:
| It's a trade off. I had to use a piped loop earlier this year
| to extract fallout new Vegas mods and textures since they all
| had spaces in their file names. For this it was perfect to
| pipe a list of the names to a loop, but for 99% of things I
| just use a for loop
| jolmg wrote:
| Yup. Nearly everything has tradeoffs.
|
| BTW, the problem I mentioned earlier can be avoided by
| using `< <()`: $ x=1 $ seq 5 | while
| read n; do (( x++ )); done $ echo $x 1 $
| while read n; do (( x++ )); done < <(seq 5) $ echo $x
| 6
|
| Almost makes me wonder what the benefit of preferring a
| pipe here is. I guess it's just about not having to specify
| what part of the pipeline is in the same shell.
| chubot wrote:
| BTW you can make it work in bash by setting shopt -s
| lastpipe. It runs the last part of the pipeline in the main
| shell, so the mutation variables of will persist.
|
| Both OSH and zsh behave that way by default, which was
| tangential to a point in the latest release notes
| https://news.ycombinator.com/item?id=29292187
|
| Another trick I've seen in POSIX shell is to add a subshell
| after the pipeline until the last time you want to read the
| variable. Like cat foo.txt | ( while read
| line; do f=$line done echo
| "we're still in the subshell f=$f" )
| edgyquant wrote:
| For me I always used for loops and only recently (after a
| decade of using Linux daily) have learned about the power of
| piped-loops. It's strange to me you are more comfortable with
| those than for loops, but I think it does make sense as you're
| letting a program generate the list to iterate over. A pain
| point in for loops is getting that right, e.g. there isn't a
| good way to iterate over files with spaces in them using a for
| loop (this is why I learned about piped loops recently.)
| zwayhowder wrote:
| You can also have your script change the bash file seperator.
|
| https://bash.cyberciti.biz/guide/$IFS
|
| Something I wish I'd learned 23 years ago instead of 3 years
| ago :(
| jolmg wrote:
| > A pain point in for loops is getting that right, e.g. there
| isn't a good way to iterate over files with spaces in them
| using a for loop
|
| If those files came as arguments, you can use a for-loop as
| long as they're kept in an array: for f in
| "${files[@]}";
|
| That handles even newlines in the filenames, while I'm not
| sure if you can handle that with a while-read-loop. IFS=$'\0'
| doesn't seem to cut it.
|
| for-loops seem preferable for working with filenames. If a
| command is generating the list, then something like `xargs
| -0` is preferable.
| marcodiego wrote:
| We need a simpler regex format. One that allows easy searching
| and replacing in source code. Of course, some IDE's already to
| that pretty well, but I'd like to be able to do it from the
| command line with a stand alone tool I can easily use in scripts.
|
| The simplest thing I know that is able to do that is coccinelle,
| but even coccinelle is not handy enough.
| f0e4c2f7 wrote:
| It's not perfect but I like sed for this.
| marginalia_nu wrote:
| I think what we really need is _one_ regex format. You have
| POSIX, PCRE, and also various degrees of needing to double-
| escape the slashes to get past whatever language you 're using
| the regex in. Always adds a large element of guesswork even
| when you are familiar with regular expressions.
| barbazoo wrote:
| > git bisect is the "real" way to do this, but it's not something
| I've ever needed
|
| uh, yeah, you did need it, that's why you came up with "2. Track
| down a commit when a command started failing". Seriously though,
| git bisect is really useful to track down that bug in O(log n)
| rather than O(n).
| masklinn wrote:
| Also provides super useful commands like skipping commits
| because some of your colleagues are assholes and commit non-
| working code.
| deckard1 wrote:
| For the given command, if the assumption is the command failed
| recently, it's likely faster than bisect. You can start it and
| go grab a coffee. It's automatic.
|
| I wish my usage of bisect were that trivial, though. Usually I
| need to find a bug in a giant web app. Which means finding a
| good commit, doing the npm install/start dance, etc. for each
| round.
| guruparan18 wrote:
| I am confused how this works. I would assume `SECONDS` would just
| be a shell variable and it was first assigned `0` and then it
| should stay same, why did it keep counting the seconds?
| > SECONDS bash: SECONDS: command not found >
| SECONDS=0; sleep 5; echo $SECONDS; 5 > echo "Your
| command completed after $SECONDS seconds"; Your command
| completed after 41 seconds > echo "Your command completed
| after $SECONDS seconds"; Your command completed after 51
| seconds > echo "Your command completed after $SECONDS
| seconds"; Your command completed after 53 seconds
| gcmeplz wrote:
| `SECONDS` is like `PWD`: the shell keeps track of updating it
| somewhere, and it'll tell you how long your shell has been
| running.
|
| https://www.oreilly.com/library/view/shell-scripting-expert/...
| guruparan18 wrote:
| Thank you. That make sense now.
| ghostly_s wrote:
| Huh, never knew about $SECONDS.
| unixhero wrote:
| A lot of genious moves here I have never seen or thought of.
| Brilliant. Grabbing pids was mindblowingly effective.
| R0flcopt3r wrote:
| You can use `wait` to wait for jobs to finish.
| some_command & some_other_command & wait
| masklinn wrote:
| One issue with that is it won't reflect the failure of those
| commands.
|
| In bash you can fix that by looping around, checking for status
| 127, and using `-n` (which waits for the first job of the set
| to complete), but not all shells have `-n`.
| michaelhoffman wrote:
| I have something like this in my bashrc: preexec
| () { # shellcheck disable=2034
| _CMD_START="$(date +%s)" } trap 'preexec; trap
| - DEBUG' DEBUG PROMPT_COMMAND="_CMD_STOP=\$(date +%s)
| let _CMD_ELAPSED=_CMD_STOP-_CMD_START if [
| \$_CMD_ELAPSED -gt 5 ]; then _TIME_STR=\"
| (\${_CMD_ELAPSED}s)\" else _TIME_STR=''
| fi; " PS1="\n\u@\h \w\$_TIME_STR\n\\$ "
| PROMPT_COMMAND+="trap 'preexec; trap - DEBUG' DEBUG"
|
| Whenever a command takes more than 5 s it tells me exactly how
| long at the next prompt.
|
| I didn't know about `$SECONDS` so I'm going to change it to use
| that.
| masklinn wrote:
| FWIW that is supported by many zsh themes like pure, p9k, p10k,
| ... (and often enabled by default).
|
| Also: REPORTTIME If nonnegative,
| commands whose combined user and system execution times
| (measured in seconds) are greater than this value have timing
| statistics printed for them.
|
| which is slightly different but generally useful, and built-in.
| oweiler wrote:
| Bash 5.0 also has $EPOCHREALTIME
| caymanjim wrote:
| Installing GNU stuff with the 'g' prefix (gsed instead of sed)
| means having to remember to include the 'g' when you're on a Mac
| and leave it off when you're on Linux, or use aliases, or some
| other confusing and inconvenient thing, and then if you're
| writing a script meant for multi-platform use, it still won't
| work. I find it's a much better idea to install the entire GNU
| suite without the 'g' prefix and use PATH to control which is
| used. I use MacPorts to do this (/opt/local/libexec/gnubin), and
| even Homebrew finally supports this, although it does it in a
| stupid way that requires adding a PATH element for each
| individual GNU utility (e.g. /usr/local/opt/gnu-
| sed/libexec/gnubin).
| jph wrote:
| > git bisect is the "real" way to do this, but it's not something
| I've ever needed
|
| git bisect is great and worth trying; it does what you're doing
| in your bash loop, plus faster and with more capabilities such as
| logging, visualizing, skipping, etc.
|
| The syntax is: $ git bisect run <command> [arguments]
|
| https://git-scm.com/docs/git-bisect
| OskarS wrote:
| Yes, git bisect is the way to go: in addition to the stuff you
| mentioned, his method only dives into one parent branch of
| merge commits. git bisect handles that correctly. A gem of a
| tool, git bisect.
| ljm wrote:
| Bisect also does a binary search so if you're looking for one
| bad commit amongst many others, you'll find it much more
| quickly than linearly testing commits, one at a time, until
| you find a working one.
| bloopernova wrote:
| This thread seems like a good place to ask this:
|
| When you're running a script, what is the expected behaviour if
| you just run it with no arguments? I think it shouldn't make any
| changes to your system, and it should print out a help message
| with common options. Is there anything else you expect a script
| to do?
|
| Do you prefer a script that has a set of default assumptions
| about how it's going to work? If you need to modify that, you
| pass in parameters.
|
| Do you expect that a script will lay out the changes it's about
| to make, then ask for confirmation? Or should it just get out of
| your way and do what it was written to do?
|
| I'm asking all these fairly basic questions because I'm trying to
| put together a list of things everyone expects from a script. Not
| exactly patterns per se, more conventions or standard behaviours.
| gmuslera wrote:
| What is your intended audience? It is you? A batch job or
| called by another program? Or a person that may or not be able
| to read bash and will call it manually?
|
| A good and descriptive name comes first, then the action and
| the people that may have to run it are next.
| scbrg wrote:
| A script is just another command, the only difference in this
| case is that you wrote it and not someone else. If its purpose
| is to make changes, and it's obvious what changes it should
| make without any arguments, I'd say it can do so without
| further ado. _poweroff_ doesn 't ask me what I want to do - I
| already told it by executing it - and that's a pretty drastic
| change.
|
| Commands that halt halfway through and expect user confirmation
| should definitely have an option to skip that behavior. I want
| to be able to use anything in a script of my own.
| gjulianm wrote:
| > When you're running a script, what is the expected behaviour
| if you just run it with no arguments? I think it shouldn't make
| any changes to your system, and it should print out a help
| message with common options. Is there anything else you expect
| a script to do?
|
| Most scripts I use, custom-made or not, should be clear enough
| in their name for what they do. If in doubt, always call with
| --help/-h. But for example, it doesn't make sense that
| something like 'update-ca-certificates' requires arguments to
| execute: it's clear from the name it's going to change
| something.
|
| > Do you prefer a script that has a set of default assumptions
| about how it's going to work? If you need to modify that, you
| pass in parameters.
|
| It depends. If there's a "default" way to call the script, then
| yes. For example, in the 'update-ca-certificates' example, just
| use some defaults so I don't need to read more documentation
| about where the certificates are stored or how to do things.
|
| > Do you expect that a script will lay out the changes it's
| about to make, then ask for confirmation? Or should it just get
| out of your way and do what it was written to do?
|
| I don't care too much, but give me options to switch. If it
| does everything without asking, give me a "--dry-run" option or
| something that lets me check before doing anything. On the
| other hand, if it's asking a lot, let me specify "--yes" as apt
| does so that it doesn't ask me anything in automated installs
| or things like that.
| mason55 wrote:
| IMO any script that makes any real changes (either to the local
| system or remotely) should take _some_ kind of input.
|
| It's one thing if your script reads some stuff and prints
| output. Defaulting to the current working directory (or
| whatever makes sense) is fine.
|
| If the script is reading config from a config file or envvars
| then it should still probably get some kind of confirmation if
| it's going to make any kind of change (of course with an option
| to auot-confirm via a flag like --yes).
|
| For really destructive changes it should default to dry run and
| require an explicit ---execute flag but for less destructive
| changes I think a path as input on the command line is enough
| confirmation.
|
| That being said, if it's an unknown script I'd just read it.
| And if it's a binary I'd pass ---help.
| bloopernova wrote:
| Thanks for the reply! I really appreciate being able to pick
| other folks' brains on here :)
| geocrasher wrote:
| I want to like this, but the for loop is unnecessarily messy, and
| not correct. for route in foo bar baz do
| curl localhost:8080/$route done
|
| That's just begging go wonky. Should be
| stuff="foo bar baz" for route in $stuff; do echo curl
| localhost:8080/$route done
|
| Some might say that it's not absolutely necessary to abstract the
| array into a variable and that's true, but it sure does make
| edits a lot easier. And, the original is missing a semicolon
| after the 'do'.
|
| I think it's one reason I dislike lists like this- a newb might
| look at these and stuff them into their toolkit without really
| knowing why they don't work. It slows down learning. Plus, faulty
| tooling can be unnecessarily destructive.
| marginalia_nu wrote:
| These are the types of stuff you typically run on the fly in
| the command line, not full blown scripts you intended to be
| reused and shared.
|
| Here's a few examples from my command history:
| for ((i=0;i<49;i++)); do wget
| https://neocities.org/sitemap/sites-$i.xml.gz ; done
| for f in img*.png; do echo $f; convert $f -dither Riemersma
| -colors 12 -remap netscape: dit_$f; done
|
| Hell if I know what they do now, they made sense when I ran
| them. If I need them again, I'll type them up again.
| masklinn wrote:
| Seems like you could say the same thing about every snippet
| e.g. running jobs is the original purpose of a shell, (3) can
| be achieved using job specifications (%n), and using the `jobs`
| command for oversight.
| lambic wrote:
| Even more correct would be to use an array:
| stuff=("foo foo" "bar" "baz") for route in "${stuff[@]}";
| do curl localhost:8080/"$route" done
| gcmeplz wrote:
| "${stuff[@]}" would be turning it back into "foo bar baz"
| though, right? I think if you were using arrays for this,
| it'd be something like: stuff=("foo" "bar"
| "baz"); array_length=${#stuff[@]}; for i in
| $(seq 0 $array_length); do curl
| localhost:8080/${stuff[i]} done
|
| I'm betting even that isn't right: as soon as bash arrays are
| a thing, I reach for a different language.
|
| [edit]: trying to get formatting correct
| computronus wrote:
| "${stuff[@]}" expands into the elements of the array, with
| each one quoted - so even that first "foo foo" element with
| a space will be handled correctly. That is how I would
| write the loop as well, in general.
|
| However, your technique also works, with some tweaks:
|
| * The loop goes one index too far, and can be fixed with
| seq 0 $(( array_length - 1 ))
|
| * There should be quotes around ${stuff[i]} in case it has
| spaces
| gcmeplz wrote:
| Oh, that's way better for sure! Thanks for the
| explanation
| [deleted]
| cjvirtucio wrote:
| and on the off-chance that `stuff` must be a space-separated
| string for w/e reason: IFS=' ' read -ra
| routes <<<"${stuff}" for route in "${routes[@]}"; do
| curl localhost:8080/"$route" done
| gcmeplz wrote:
| Thanks for the note about the semicolon! Added it
| geocrasher wrote:
| Sure thing. I apologize if my comment came across as overly
| negative. My goal was constructive criticism, but based on
| the downvotes, I'm guessing I missed the mark there.
| usefulcat wrote:
| > Use for to iterate over simple lists
|
| I definitely use this all the time. Also, generating the list of
| things over which to iterate using the output of a command:
| for thing in $(cat file_with_one_thing_per_line) ; do ...
| phone8675309 wrote:
| Why not? xargs -n1 command <
| file_with_one_thing_per_line
| make3 wrote:
| that's like bash 101, not sure why it's in there
| l0b0 wrote:
| > 1. Find and replace a pattern in a codebase with capture groups
| > git grep -l pattern | xargs gsed -ri 's|pat(tern)|\1s are
| birds|g'
|
| Or, in IDEA, Ctrl-Shift-r, put "pat(tern)" in the first box and
| "$1s are birds" in the second box, Alt-a, boom. Infinitely easier
| to remember, and no chance of having to deal with any double
| escaping.
| marginalia_nu wrote:
| Using an IDE kind of handicaps you to only working with your
| IDE though. The shell works everywhere for every use case.
| withinboredom wrote:
| That particular IDE works on Windows though... no idea how to
| use Powershell...
| marginalia_nu wrote:
| WSL2?
| withinboredom wrote:
| From experience, WSL is pretty slow vs. native on fs
| operations.
| marginalia_nu wrote:
| For sure, but for the convenience of being able to use
| bash commands on Windows it's well worth it.
| turbocon wrote:
| Yea, I've yet to come across a regex replacement tool as easy
| to use as jetbrains find & replace. Invaluable for certain
| tasks.
___________________________________________________________________
(page generated 2021-11-23 23:01 UTC)