[HN Gopher] Pipexec - Handling pipe of commands like a single co...
___________________________________________________________________
Pipexec - Handling pipe of commands like a single command
Author : JNRowe
Score : 124 points
Date : 2024-03-10 01:32 UTC (21 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| koolba wrote:
| This is neat, but outside of a contrived ouroboros example,
| what's a real world use case for this?
|
| There's a natural flow of outputs becoming inputs and I'm
| struggling to identify a situation where I would feed things back
| into the source. Also, named pipes kind of solve that already.
| LatticeAnimal wrote:
| Agreed -- their only example is `pipexec -l 2 -- [ LS /bin/ls
| -l ] [ GREP /bin/grep LIC ] '{LS:1>GREP:0}'` which appears to
| be `ls | grep LIC` with more steps. Seems like a (cool)
| solution without a real problem.
|
| (I'd love to be wrong though and see a real use case for some
| cool feedback loop of commands)
| sebosp wrote:
| This reminds me of MIT's open courseware, 601 SC, unit 1 with
| state machines, going all the way to build Fibonacci with them
| without recursion, tbh the moment the teacher translated the
| state machine to bounds to electrical circuit I felt it was a
| leap and I couldn't quite understand their relationship, maybe
| I missed a requirement course. I tried to express that course
| in Rust as one of my first projects learning the language here
| https://github.com/sebosp/rustexercises/blob/develop/ocw601s...
| and I think a similar iteration in the direction of this
| project would be to build the dependencies as drag-and-drop
| boxes over the browser (maybe egui) and connect the state
| machines by clicks, maybe download the generated code as either
| bash or compilable rust code, you know, for kids.
| eichin wrote:
| It looks like it might be able to handle one of the tricks I do
| with dpipe running sshfs (
| https://gist.github.com/bdmorin/5eb17828612e7d1b66e92550f428...
| is probably where I picked it up from) but I'm not sure (and
| dpipe already handles it fine.)
| CGamesPlay wrote:
| I recently wanted this for some scripting over SSH. I basically
| want to run a script on a remote machine and read it back, but
| implement it as a function instead of a wrapper around SSH.
|
| Chatbots, where the bot only needs to be line-driven and you
| can connect it to any CLI chat interface. Or perhaps, run your
| AI agent attached to a shell, and have it treat standard IO as
| a shell session.
| geon wrote:
| Somewhat related: https://github.com/joewalnes/websocketd
|
| > websocketd is a small command-line tool that will wrap an
| existing command-line interface program, and allow it to be
| accessed via a WebSocket.
| lubutu wrote:
| I suppose such feedback could be used for reaching a fixpoint.
| Suppose you have a build system that reads targets to be built
| from stdin and outputs to stdout targets that are dependent on
| that target and must now be rebuilt. With an ouroboros, the
| build system will continue to run, even if the dependency graph
| is dynamically cyclical, until the fixpoint is reached and the
| build terminates.
| geon wrote:
| It can be used as client/server communication locally. I've
| done that (without pipexec) for bots in a multiplayer game.
| That way I could implement the bot ai independently and test
| them against each other.
| 8n4vidtmkvmk wrote:
| I did pretty much this exact same thing too. The game runner
| spawns 2 bots and acts as the middleman/, piping their output
| to each other. I did it in node though. I wouldn't want to
| code this in bash.
| qazxcvbnm wrote:
| It seems to me that this can be achieved by the following bash-
| native way of creating extra file descriptor pipes:
| pipe_path="$(mktemp -u)" mkfifo "$pipe_path" exec
| 3<>"$pipe_path" rm -f "$pipe_path"
|
| Here, exec associates the file descriptor (3 here, replace with
| any desired descriptor) with the pipe created by mkfifo. The
| filesystem path to the pipe is removed immediately after we
| obtain a file descriptor to it, so that the the only remaining
| reference to the pipe in the system would be from this script,
| and thus when the script dies, the kernel will automatically free
| the pipe.
|
| An example use case would be like so:
| https://unix.stackexchange.com/a/216475/585293
| pxeger1 wrote:
| Worth noting that POSIX sh only specifies that file descriptors
| up to 9 be supported, and many shells stick with this minimum.
| junon wrote:
| Maximum, you mean? Or am I misunderstanding you? Written a
| lot of shell scripts in my days and somehow this fact escaped
| me. Thanks!
| js2 wrote:
| Rephrased for clarity: POSIX specifies at least (a minimum
| of) 9. POSIX does not specify a maximum. Shells may provide
| more than 9. From the spec:
|
| > Open files are represented by decimal numbers starting
| with zero. The largest possible value is implementation-
| defined; however, all implementations shall support at
| least 0 to 9, inclusive, for use by the application.
|
| https://pubs.opengroup.org/onlinepubs/9699919799/utilities/
| V...
|
| And from the bash manual:
|
| > Redirections using file descriptors greater than 9 should
| be used with care, as they may conflict with file
| descriptors the shell uses internally.
|
| https://www.gnu.org/software/bash/manual/bash.html#Redirect
| i...
|
| Despite writing shell scripts for nearly three decades, I
| also was unaware of what POSIX had to say but I can't
| recall ever needing more than a couple extra FDs at most.
| junon wrote:
| Ah, thanks! Makes more sense :)
| pipeline_peak wrote:
| Isn't this exactly what shell scripting is for?
| chubot wrote:
| Hm interesting, also see dgsh, the directed graph shell
|
| https://www2.dmst.aueb.gr/dds/sw/dgsh/
|
| https://github.com/dspinellis/dgsh
|
| https://news.ycombinator.com/item?id=21700014
|
| dgsh uses Unix domain sockets, not pipes. I don't remember
| exactly why, but it's in the paper, perhaps to avoid deadlocks
| compared to pipes.
|
| I'd also be interested in some more examples with pipexec or
| dgsh!
| JNRowe wrote:
| Yep deadlocks, you're correct. From the paper:
|
| "Modern Unix systems offer named pipes, also known as fifos,
| which can be used to hand-craft arbitrary process communication
| topologies. However, if combined one-to-many and many-to-one
| piping are setup by using named pipes, another problem will
| occur. Due to the limited buffering offered by typical
| programs, deadlocks can easily occur when a process consuming
| data from many producers with more than one input, blocks
| waiting for input from one of the processes feeding it. This
| can cause a second feeding process to block, waiting to send
| its output to another one of the consumer process's inputs,
| and, thereby, blocking the upstream process feeding both
| processes that provide data to the consumer one."
|
| dspinellis has commented on another dgsh discussion1(along with
| you). Interestingly, with a light comparison to pipexec2.
|
| I stumbled upon pipexec trying to find a battle tested solution
| to extend a data munging task where I was relying on zsh's
| multios3, mostly because orchestrating the interactions with a
| coproc'd jq for output were fighting me. There is something
| both frustrating and soothing about finding a seven year old
| comment pointing out why my path was doomed before I'd even
| started; people have solved the problem already, plus people
| far smarter than me also found the trap.
|
| 1 https://news.ycombinator.com/item?id=13352659
|
| 2 https://news.ycombinator.com/item?id=13358090
|
| 3
| https://zsh.sourceforge.io/Doc/Release/Redirection.html#Mult...
| chubot wrote:
| Oh wow, I had stumbled on the zsh behavior of
| echo hi > *.txt # writes ALL of the files!
|
| No other shell does that.
|
| But I didn't know it was called MULTIOS until now. (I guess
| that's read "mult I/O's"? I have a hard time not reading it
| as "multi-OS" :) )
|
| It seems a bit niche to be honest, but it's possible to
| support in Oils.
|
| ---
|
| Oils also uses Unix domain sockets already for the headless
| shell protocol
|
| https://github.com/oilshell/oil/wiki/Headless-Mode
|
| We could do something like dgsh, but so far I haven't seen a
| lot of uptake / demand. Every time it's mentioned, somebody
| kinda wants it, and then it kinda peters out again ... still
| possible though.
|
| I think flat files work fine for a lot of use cases, and once
| you add streaming, you also want monitoring, more control
| over backpressure/queue sizes, etc.
| JNRowe wrote:
| I use mutlios and even I'm not that attached to it. The
| majority of my use is combined with process substitution,
| and could be replaced with common-ish tools like pee1(or
| pipexec for more complex cases). The only occasion when I'm
| thankful for it is if I want to use a shell function as a
| target, but there are workarounds for that too.
|
| As a noclobber user the footgun is largely hidden to me,
| but I feel its presence. multios without globbing support
| would be less useful, but would still work for _most_ of
| _my_ use cases. Scanning my shell history I see various
| cases of relying on zsh 's ability to apply sorting and
| filtering to globs with multios' input redirection, but
| only a couple where I want that in output redirection. The
| input instances could easily be rewritten using cat and
| globbing.
|
| Even with multios unset the behaviour is different between
| zsh and bash. For example, nomultios disables all the
| expansion, so zsh behaves like more like dash with ':
| >t{1,2}' creating a file instead producing an error like
| bash does.
|
| [FWIW, I google'd mult _i_ ios to link the option in mt
| original comment. It really feels like it needs double-i,
| and I read the single i name the same way you do.]
|
| ---
|
| I'd be one of those people whose desire for dgsh-like
| functionality wanes. If it was slight DSL that I could
| "upgrade" pipelines to I'd probably use it, but not enough
| to warrant working on it or switching other tooling to
| support it.
|
| The end of result of this morning's pipeline was breaking
| my jobs up, and applying some judicious use of nq2 to keep
| track of it. I'd follow your advice and move on to more
| specialist tools if the job grew significantly or if it
| became a regular occurrence.
|
| 1 https://joeyh.name/code/moreutils/
|
| 2 https://git.vuxu.org/nq/about/
| snthpy wrote:
| This is great, thanks!
| wwalexander wrote:
| In the category of "command line representations of graphs" see
| also ffmpeg's filtergraphs [1].
|
| [1] https://ffmpeg.org/ffmpeg-filters.html#Filtering-
| Introductio...
| notpushkin wrote:
| First thing I thought about when seeing the Pipexec's syntax,
| too.
| keithalewis wrote:
| > Nobody will tell you:
|
| > stdin, stdout and stderr are artificial definitions.
|
| Unless you RTFM.
| bravetraveler wrote:
| Yea ~ $ man stdin | grep stdin,
| stdin, stdout, stderr - standard I/O streams The
| input stream is referred to as "standard input"; the output
| stream is referred to as "standard output"; and the error
| stream is referred to as "standard error".
| These terms are abbreviated to form the symbols used to refer
| to these files, namely stdin, stdout, and stderr.
| On program startup, the integer file descriptors associated
| with the streams stdin, stdout, and stderr are 0, 1, and 2,
| respectively.
|
| This alignment is, indeed, very much deliberate. Take a peek at
| this:
|
| https://pubs.opengroup.org/onlinepubs/9699919799/functions/s...
| _(also, C11)_
|
| They are optional - emphasis mine:
|
| > three streams _shall_ be predefined and _need not_ be opened
| explicitly
|
| They're artificial in the sense that I don't _have to_ follow
| laws /rules/customs, I guess. Interesting things may or may not
| happen if I don't.
|
| In the same way, interesting things happen if I wire stdout to
| stderr.
| soygem wrote:
| 42 pipex, we meet again
| gorgoiler wrote:
| What a beautifully designed tool. In our Python codebase we end
| up reaching for inline sh scripting a lot whenever we need to
| pipe between processes. In a way it feels ok -- after all, no one
| has any qualms about reaching for inline SQL to get things done,
| so what's wrong with a little shell script in the middle of a
| Python module?
|
| Just as there are efforts -- both wise and misguided -- to
| represent the building of an SQL query with Python syntax, what
| Python tools are there to build sh pipelines between processes
| with a more pythonic syntax? Do they provide value in excess of
| the novelty tax one has to pay for using a non standard library?
| erisinger wrote:
| I used to use plumbum for Pythonic(ish) shell scripting. It was
| great for a specific use case.
| DebtDeflation wrote:
| Read the first word of the title and assumed this would be about
| some new bot/method for automating layoffs.
| k3vinw wrote:
| It would be cool to have a tool similar to this, but for
| composing a graph of commands similar to aws step functions (or
| Jenkins pipelines). I'd call it mapexec :)
| agumonkey wrote:
| Considering how airflow/dagster are trendy these days,
| concurrency too.. I assume a leaner, os/language agnostic
| solution for this problem might emerge not too far in the future.
| mmgutz wrote:
| Wasn't understanding the graphics in the README until I used
| light mode.
| diekhans wrote:
| It's an interesting package, and I have some of the use cases it
| appears to address. However, the documentation is inadequate to
| quickly understand how to robustly build some of the more complex
| cases. In particular, how to build bash-style process
| substitution. Robust here is the pipeline exits non-zero if any
| of the substituted processes fail, as demonstrated by this
| example: #!/bin/bash set -beEu -o
| pipefail cat <(date) <(false) echo did not exit
| non-zero
|
| If this is addressed, it would be worth more time to figure out
| Pipexec.
___________________________________________________________________
(page generated 2024-03-10 23:01 UTC)