[HN Gopher] GNU Parallel
___________________________________________________________________
GNU Parallel
Author : todd8
Score : 61 points
Date : 2022-04-02 19:02 UTC (3 hours ago)
(HTM) web link (www.gnu.org)
(TXT) w3m dump (www.gnu.org)
| paskozdilar wrote:
| GNU Parallel is one of my favorite utilities of all time.
|
| I used to write complex and bug-ridden scripts that used bash job
| control to implement parallel execution of batch jobs. When I
| discovered GNU Parallel, I deleted them all and never looked
| back.
|
| Also, the documentation is _awesome_ - there is a whole book [0]
| on GNU Parallel, and the manpage even links to a series of
| youtube videos [1] that explain how it works.
|
| [0]
| https://zenodo.org/record/1146014/files/GNU_Parallel_2018.pd...
|
| [1] http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
| WhatIsDukkha wrote:
| This can be a good entry into the "why" for many people that use
| xargs now -
|
| https://www.gnu.org/software/parallel/parallel_alternatives....
|
| It has a lot of features that can feel excessive at first glance
| but if you have felt some pain in building jobs, most of it is
| pretty sensible and much better than rollyourown.
| cb321 wrote:
| The comparison is not very fair to modern day xargs.
|
| `nproc` is a relatively standard utility (coreutils). So, xargs
| -P$(nproc) gets you core (or core-proportional) parallelism.
|
| Grouping output/Making a safe parallel grep is also easy-ish
| with `--process-slot-var=slot` and sending to `tmpOut.$slot`.
|
| Jobs on remote computers can be done similarly with any kind of
| `arrayVar[$slot]` setup where `arrayVar` has a bunch of `ssh`
| targets, possibly duplicates if you want to run >1 job per
| host. (In pure POSIX sh you could use eval and $1, $2
| positional args with shell arithmetic..)
|
| Anyway, those three are just off the top of my head,
| unfairness-wise. Last I looked at the source for GNU parallel
| it looked like mountains upon mountains of Perl I would rather
| not depend upon, personally, but to each his own.
| forgotmypw17 wrote:
| Perl has retained stable backwards compatibility and no
| breaking changes for 20+ years. What's wrong with Perl?
| ZoomZoomZoom wrote:
| > Anyway, those three are just off the top of my head,
| unfairness-wise. Last I looked at the source for GNU parallel
| it looked like mountains upon mountains of Perl I would
| rather not depend upon, personally, but to each his own.
|
| Well, there was a Rust version with zero Perl, now
| unfortunately archived. It wasn't 100% on a par with the
| original and wasn't really finished. On the other hand, built
| easily for Windows and helped me on a few occasions.
|
| https://github.com/mmstick/parallel
| cb321 wrote:
| Well, some archived project is not so great either. The
| core functionality is not even a 20 line bash script since
| bash grew wait -n, though: #!/bin/bash
| if [ "${1-0}" -lt 1 ]; then # No arg / arg not
| a number >= 1 echo "Usage: $0 <N>"; echo "reads
| cmds from stdin, running up to N at once." exit 1
| fi TMP=`mktemp -t stripen.XXXXXX` trap 'rm
| -f $TMP; exit 0' HUP INT TERM EXIT STRIPE_SEQ=1
| while read cmd; do jobs > $TMP
| # jobs | wc -l does not work if [ $(wc -l <
| $TMP) -ge $1 ]; then wait -n
| # Wait for 1/more jobs to finish fi # Could
| accumulate total $? above, but would need to replace final
| wait. STRIPE_SEQ=$((STRIPE_SEQ + 1))
| ( eval "$cmd" ) < /dev/null & # Run job in a subshell
| in bg done rm -f $TMP wait
| # Wait for all to finish
|
| and if bash ever grows some magic environment variable
| $NUM_BG_JOBS or you don't want auto-help or sequence
| numbers or etc. it can be even simpler.
| dang wrote:
| Related:
|
| _GNU Parallel 2018_ -
| https://news.ycombinator.com/item?id=20726631 - Aug 2019 (68
| comments)
|
| _GNU Parallel Cheat Sheet [pdf]_ -
| https://news.ycombinator.com/item?id=19330356 - March 2019 (63
| comments)
|
| _GNU Parallel_ - https://news.ycombinator.com/item?id=13258142 -
| Dec 2016 (83 comments)
|
| _GNU Parallel Tutorial_ -
| https://news.ycombinator.com/item?id=12943150 - Nov 2016 (65
| comments)
|
| _A Million Text Files and a Single Laptop_ -
| https://news.ycombinator.com/item?id=11248326 - March 2016 (27
| comments)
|
| _GNU Parallel - The command line power tool_ -
| https://news.ycombinator.com/item?id=6209767 - Aug 2013 (28
| comments)
|
| _GNU /Parallel changed my life_ -
| https://news.ycombinator.com/item?id=1894639 - Nov 2010 (8
| comments)
|
| _GNU Parallel - build and execute command lines from standard
| input in parallel_ - https://news.ycombinator.com/item?id=1801186
| - Oct 2010 (36 comments)
| mekster wrote:
| If you want to execute commands on multiple remote hosts over
| SSH, just run a tmux session and launch a tab on each host in a
| loop and execute them.
|
| It's far easier to follow the output and individually deal with
| prompts.
| permalac wrote:
| I do the same, I even have a command that tells me which tmux
| panel I'm in so I can ssh to many servers at once from tmux.
|
| However, when you just want to execute the same command :
| https://clustershell.readthedocs.io/en/latest/tools/clush.ht...
| paskozdilar wrote:
| True, but for a large number of hosts, going manually through
| prompts doesn't scale anymore. Sometimes you just want to check
| whether all hosts have the same configuration file, or
| something like that. That's where GNU Parallel can save your
| gluteus maximus: cat hosts.txt | parallel
| --quote --timeout=10 ssh {} 'echo {} $(md5sum ~/.config/file)'
|
| The above command would take a list of hostnames from the
| hosts.txt file, connect to each one, hash the config file and
| print out hostnames and hashes in one line per host.
| technofiend wrote:
| There's probably a more unix-centric / idiomatic way to do it,
| but using GNU Parallel along with Redis made automating tasks to
| process tens of thousands of systems extremely easy, repeatable,
| pausable, restartable, etc. And the fact you can do atomic
| operations on queues prevented any potential worker collisions.
| dehrmann wrote:
| It's nice, but the citation bit strikes me as very non-free:
|
| > If you use --will-cite in scripts to be run by others you are
| making it harder for others to see the citation notice. The
| development of GNU parallel is indirectly financed through
| citations, so if your users do not know they should cite then you
| are making it harder to finance development. However, if you pay
| 10000 EUR, you have done your part to finance future development
| and should feel free to use --will-cite in scripts.
|
| > If you do not want to help financing future development by
| letting other users see the citation notice or by paying, then
| please consider using another tool instead of GNU parallel. You
| can find some of the alternatives in man parallel_alternatives.
| 6keZbCECT2uB wrote:
| Thinking about the citation notice and whether the software is
| safe to use at work is why I just use xargs.
| petepete wrote:
| I was trying to use parallel properly for the first time last
| week and the nag screen put me off and made me choose an
| alternative.
|
| I ended up with the moreutils one and it appears to work just
| as well for my simple needs.
| Athas wrote:
| It doesn't affect your right to do anything with the software.
| It doesn't even affect your right to remove this citation
| notice (and some distributions do so). Free software is not
| about software not being annoying; merely about your right to
| remove the annoyances.
|
| I'm not certain GNU parallel's approach to obtaining funding is
| a good strategy, but I find it weird that people object to it
| on philosophical or legal grounds.
| CaliforniaKarl wrote:
| > I'm not certain GNU parallel's approach to obtaining
| funding is a good strategy,
|
| Indeed, it's a good question! I know that, for the SGI UV300
| we got through an NIH S10 grant, the usage of the
| supercomputer is tracked by looking for the grant number in
| publication acknowledgements. Yes, we already got the money,
| so you may wonder "what does it matter?", but our ability to
| get future funding (especially as the UV300 nears retirement
| age) depends (at least in part) on showing how well we used
| our previous funding.
|
| In our case, the funding came through an NIH grant, so we ask
| people reference the grant number. But more broadly, and
| especially for software, issuing a DOI for released versions
| (through a service like Zenodo https://zenodo.org), along
| with a request for acknowledgement, gives a way to track
| usage. For example, the 20150322 release of parallel (DOI
| 10.5281/zenodo.16303) has been cited at least 63 times (per
| https://doi.org/10.5281/zenodo.16303).
|
| Looking at
| https://scholar.google.com/citations?user=D7I0K34AAAAJ&hl=en,
| it seems the 2011 version of Parallel has been cited over
| 1,000 times.
|
| The author is at the University of Copenhagen (per the Google
| Scholar link above), so it's entirely possible that at least
| some of the funding for his employment is coming from sources
| that use citation counts as an indicator that they are
| "getting their money's worth" by continuing to fund (at least
| part of) Mr. Tange's employment.
| CaliforniaKarl wrote:
| Your concern about this being non-free is addressed by an
| entire FAQ. Here's a link to the FAQ page from the software's
| Git repo:
|
| https://git.savannah.gnu.org/cgit/parallel.git/tree/doc/cita...
| [deleted]
| stabbles wrote:
| I don't understand how GNU finds this acceptable...
| CaliforniaKarl wrote:
| [edited for formatting and to fix Git link]
|
| According to https://git.savannah.gnu.org/cgit/parallel.git/t
| ree/doc/cita...
|
| > == Is the citation notice compatible with GPLv3? ==
|
| > Yes. The wording has been cleared by Richard M. Stallman to
| be compatible with GPLv3. This is because the citation notice
| is not part of the license, but part of academic tradition.
|
| > Therefore the notice is not adding a term that would
| require citation as mentioned on:
| https://www.gnu.org/licenses/gpl-faq.en.html#RequireCitation
|
| If you are of the view that clearance by RMS is clearance by
| FSF/GNU, then that's how they find it acceptable. If you take
| a different view, then the next part of that section applies:
|
| > If you disagree with Richard M. Stallman's interpretation
| and feel the citation notice does not adhere to GPLv3, you
| should treat the software as if it is not available under
| GPLv3. And since GPLv3 is the only thing that would give you
| the right to change it, you would not be allowed to change
| the software.
|
| There's also an interesting comparison to be made:
|
| > == How do I silence the citation notice? ==
|
| > Run this once:
|
| > parallel --citation
|
| > It takes less than 10 seconds to do and is thus comparable
| to an 'OK. Do not show this again'-dialog box seen in
| LibreOffice, Firefox and similar programs.
| paskozdilar wrote:
| This is an excellent explanation of why "GNU finds this
| acceptable".
|
| Note that the citation message can also be easily silenced
| just by creating an empty file: touch
| ~/.parallel/will-cite
| [deleted]
| ValtteriL wrote:
| Parallel has been (and still is) super useful and simple tool for
| speeding up all kinds of shell tasks during my career.
|
| Still remember the time I wasted days waiting for the completion
| of multiple serial tasks when it would have been an hour with
| parallel :-)
| [deleted]
| arrakeen wrote:
| parallel is so useful and i use it multiple times daily. i wish
| its `:::` syntax was supported at the shell level so i could use
| it for every application
___________________________________________________________________
(page generated 2022-04-02 23:01 UTC)