[HN Gopher] UP: A tool for writing Linux pipes with instant live...
___________________________________________________________________
UP: A tool for writing Linux pipes with instant live preview
Author : ducktective
Score : 439 points
Date : 2021-03-31 06:28 UTC (16 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| foreigner wrote:
| Wow I literally just thought about building a tool like this
| yesterday! Spooky.
| whizzter wrote:
| Feels like a perfect tool for taco-bell style
| programming(googleable), and i think there needs to be more in
| this style.
|
| Powershell for example has an IDE(MS style) that is damn helpful
| by providing breakpoints/intraspection but this sits at a
| lower/faster level for just inspecting the initial pipings.
| pjmlp wrote:
| Powershell is probably the closest shell to the REPL experience
| of Xerox PARC and ETHZ workstations.
|
| UNIX shells could approximate it with support for shared
| objects, D-Bus/XPC/IPC, but few make use of it.
| qrbLPHiKpiux wrote:
| "Running around with a chainsaw"
|
| Lol!
| dan-robertson wrote:
| This looks quite good, and I've wanted something like this for a
| while now. I wonder if it could be improved by incremental
| versions of commands like grep, e.g. making the change grep net
| -> grep network faster.
|
| Another system with some different but related capabilities is
| userland: https://hisham.hm/userland/
| akavel wrote:
| A variation on userland is the next project I'm trying to work
| on, though there are... distractions. In a random case that
| some benevolent millionaire wants to sponsor me to work full
| time on things like that, I'm certainly open to being
| contacted, my email is in the commits ;)
|
| As to another _huge_ and _absolutely awesome_ emerging
| contender in this space, see https://luna-lang.org
| thitcanh wrote:
| Wow this is almost exactly what I've been looking for for
| years. I don't want a tool like up that works on a fixed entry
| command; I want a full terminal that takes care of everything,
| showing me the output at every step. What's missing is some
| caching (think HTTP requests) and limitation (e.g. evaluate the
| first 4 HTTP calls, and wait for me to approve the rest if
| they're good)
| rakoo wrote:
| That is some impressive Proof of Concept and it's not even
| surprising considering who created it. Using a "terminal" like
| that would probably change a lot of how we work on the command-
| line
| mtbkvc wrote:
| very good idea. thanks akavel
| zachlatta wrote:
| This seems amazing. I can't wait to try this.
| amarshall wrote:
| Somewhat similar is fzz https://github.com/mrnugget/fzz
| rgrau wrote:
| For fast iteration with pipes, using only basic unix tools, and
| arguably less risky to fat-fingering, I write it in a file, and
| use watch -n1. Easy to modify, temporarily comment parts of it,
| etc...
|
| Also, I wrote this plog function (for pipe-log, or something):
| plog() { local msg=${1:-plog} tee >(sed -e
| "s/^/[$msg] /" | cat 1>&2 ) }
|
| That allows me to see what's happening in the middle steps of the
| pipeline.
|
| I ended up compiling a list of tricks like that in
| https://raimonster.com/scripting-field-guide/ , in case anyone is
| interested (and feedback is appreciated)
| rustyminnow wrote:
| This is super cool. There's so much good stuff in there I'm
| gonna have to come back a few times just to digest it all! Also
| looking forward to some of the links at the bottom.
|
| Do you have a standard template you use for your scripts? You
| kind of touch on this in section 4.20.1 but don't go too deep.
|
| Also have you done any work with writing zsh (or bash)
| completions? I feel like having good completions would make my
| scripts magnitudes more handy, but I've had trouble finding
| good starter resources.
| rgrau wrote:
| Most of my scripts start with that template on 4.20.1. That's
| the baseline. But soon enough they have 4.22 and 4.25.
|
| I never liked those "bare minimum templates" that are 100
| lines, because usually the scripts I write start being just a
| couple dozen lines. If they grow, I add things on a need
| basis, but the ratio of logic/template has to be >0 always.
|
| wrt completions, doing smart completions is hard, and the zsh
| ones are not very intuitive to me. I try to make my scripts
| work with "compdef _gnu_generic ". It goes a long way, and
| it's an incentive to write proper --help
| arendtio wrote:
| If you don't mind, this is my base template for Bash scripts:
|
| https://pastebin.com/FYMmnZ0B
|
| At the start it provides a stacktrace function which can also
| be used to notify you if your script has failed (e.g. via
| xmpp, e-mail, etc.). It is not my own creation, as I found it
| somewhere a few years ago, but found it super useful over
| time.
|
| In addition, there are two versions for checking for
| dependencies. The first lets you communicate instructions for
| every tool that is required while the second can be used if
| you simply want to communicate the list of programs that is
| missing.
|
| The third part is about named parameters, which can be
| helpful if you don't want to use positional parameters.
| jolmg wrote:
| You can skip cat. tee >(sed -e "s/^/[$msg] /"
| 1>&2)
| maxwelljoslyn wrote:
| rgrau, I'm halfway through your field guide. I appreciate how
| concise the text portions are, and how, although each section
| teaches one thing at a time, you are sometimes willing to leave
| the reader with "here are all the major incantations, figure
| out the details of each one and appreciate them" - such as in
| section 4.4.
|
| Learned several new bashisms which i will put into my computer
| tip memorization deck. Thank you for writing.
| rgrau wrote:
| Thanks for giving it a read. Glad it was useful!
| kmarc wrote:
| Yes, I also found it very good. Opened a PR[1] to "Awesome
| Shell" to include it.
|
| [1]: https://github.com/alebcay/awesome-
| shell/pull/424/files
| rippeltippel wrote:
| I started a very similar side project (in Rust) at the beginning
| of lockdown, but didn't manage to keep up as I had to spend most
| of my "free" time parenting.
|
| Although the self-proud part of me feels a bit disappointed that
| someone else got there first, I'm actually glad that it happened
| (it also means that my idea was sound). I didn't even realise
| that something like that existed already. Kudos to both the
| developer and the OP!
| dmingod666 wrote:
| Just saying, the tool is about 3 years old, most recent commit
| is 12 months ago..
| michaelmior wrote:
| Actually, 5 months ago :)
| rippeltippel wrote:
| I noticed, that's why I wrote "I didn't even realise that
| something like that existed already" :) And I'm grateful to
| the OP who brought this tool to HN's attention.
| akavel wrote:
| Don't worry, when I published _up_ I also learned about a
| previous similar project from commenters ;) probably wouldn 't
| start if I knew earlier ;) fortunately I didn't know :D just in
| case if you still have some mojo left, _up_ is very open to an
| idea of a RIIR contender, it 's actually even kinda encouraged
| in the readme :)
| Siira wrote:
| I find `fzf --bind "change:reload:some-shell-command {}"
| --disabled` a better alternative.
| dakra wrote:
| I use a similar tool in Emacs called piper
| https://gitlab.com/howardabrams/emacs-piper
|
| And here's the EmacsConf video for those interested:
| https://www.youtube.com/watch?v=HKJMDJ4i-XI
| marco_craveiro wrote:
| Piper looks amazing, shame its not on a package repo.
| akavel wrote:
| I didn't manage to grasp it from quickly skimming the video and
| the gifs: is it possible to retry a command in piper until I
| find the right combination of flags, and only then press `|` to
| push it to the buffer? Ideally, keeping the incomplete command
| visible for editing all the time? Also, can I get back to older
| contents of the buffer, i.e. a couple `|`s back? If yes, that
| might be basically what I wanted _up_ to be from the beginning.
| ducktective wrote:
| The developer, Mateusz, did a "show HN" some 2.5 years ago:
|
| https://news.ycombinator.com/item?id=18291302
| akavel wrote:
| Hi ducktective, thanks for submitting again! :) It was a
| pleasant surprise seeing it here today, on my regular HN
| addiction^H^H^H daily read :) I completely didn't connect the
| dots that it's ok to repost periodically to share the fun and
| serve as a refresher for the lucky 10k, kudos to you for being
| here to do that :)
| kreetx wrote:
| There was this same idea realized via fzf preview, also posted to
| hn I think.
| WalterGR wrote:
| fzf has been submitted many times -
| https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
| - but most recently:
|
| Improving Shell Workflows with Fzf (jambor.dev)
|
| https://news.ycombinator.com/item?id=26634419
|
| 221 points by todsacerdoti 22 hours ago | 51 comments
| orf wrote:
| Slightly related, I wrote a tool called ptail[1] that only shows
| the last N lines of output from a pipe. Quite useful in some
| situations
|
| 1. https://github.com/orf/ptail
| [deleted]
| mraza007 wrote:
| I was looking for a very similar tool for my project. Anyhow does
| some one know about a tool that allows you to render your
| commandline output to html
| uberduper wrote:
| When whittling down text output like the examples on the project
| page, I've found that just piping into vim and using a
| combination of vim commands and external commands like `:%!grep
| -v foo` to be far easier and much more capable.
| thesh4d0w wrote:
| I thought I'd seen almost everything in bash, but I've never seen
| |& before. My attempts at googling it aren't coming up with
| anything, can anyone advise what it does compared to a normal
| pipe?
| amarshall wrote:
| From https://www.gnu.org/software/bash/manual/html_node/Pipelin
| es...:
|
| > If '|&' is used, command1's standard error, in addition to
| its standard output, is connected to command2's standard input
| through the pipe; it is shorthand for 2>&1 |. This implicit
| redirection of the standard error to the standard output is
| performed after any redirections specified by the command.
| thesh4d0w wrote:
| Ah ha, I suspected it was that but couldn't find the relevant
| docs. Thanks!
| alpaca128 wrote:
| Just recently I thought I could use a live checking tool for
| regexes but without the horrible UI of the typical website made
| for this purpose.
|
| I will definitely try this and see how it combines with
| ripgrep/rg.
| dmingod666 wrote:
| regex101.com is pretty comprehensive, my go to for complex
| regexes.. not sure if that's what you're complaining about..
| alpaca128 wrote:
| That's what I tried recently among others, it's the website
| equivalent of an IDE. And when I actually started entering
| some regex it started throwing errors and timeout messages
| for inputs that standard grep will happily chew through in
| fractions of a second.
|
| On subpar hardware that site is a single collection of
| annoyances, including window-filling loading screens.
|
| That said I tried UP and it works very nicely for what I
| need.
| crvdgc wrote:
| > WARNING: Please be careful when using it! It could be
| dangerous. In particular, writing "rm" or "dd" into it could be
| like running around with a chainsaw. But you'd be careful writing
| "rm" anywhere in Linux anyway, no?
|
| Side effects always seems like potential disasters for me.
| mimimi31 wrote:
| I remember reading about a project that let you "preview" the
| effects (e.g. file changes) of terminal commands by using
| ptrace to intercept the syscalls and just logging them instead.
| Maybe you could do something similar here to prevent accidental
| file deletions.
| nabakin wrote:
| My first thought was to sandbox the executable but this
| sounds even better. Is it hard to implement the interception
| of all syscalls? And can file reads be distinguished from
| file writes?
| simcop2387 wrote:
| Not terribly hard to do, and yes you can distinguish reads,
| writes, etc. A tool to log syscalls to see what might be
| possible is strace, it's incredibly useful for debugging a
| misbehaving application that either you don't have all the
| source for or it's gigantic codebase that you don't
| understand. Seeing which files a program is looking for is
| often all you need to figure out why it's behaving the way
| it is (i.e. locating an undocumented/badly
| documented/incorrectly documented config file)
| minxomat wrote:
| That's maybe: https://github.com/p-e-w/maybe
| akavel wrote:
| Please note that _up_ respects $SHELL, so if you find a way
| to set $SHELL to point to _maybe_ (possibly wrapped in some
| helper script) in a useful manner, it might actually work!
| I 'd highly appreciate an issue with a report/guide for
| that then on _up_ , others might find that helpful!
|
| For details, see:
| https://github.com/akavel/up/releases/tag/v0.3
| minxomat wrote:
| I'm not affiliated with maybe, and it's also no longer
| maintained. I don't know if there's a newer, better tool
| out there.
| adictator wrote:
| Wow. That sounds like a really handy tool. Can it be used
| to debug programs, maybe?
| alvarlagerlof wrote:
| Wow that would be cool
| haolez wrote:
| Is there a way to start a new instance of bash with read-only
| permissions only? This could easily fix the side-effects issue.
| jackhalford wrote:
| if linux had openbsd's pledge(2) then you could disable
| filesystem writes, which you don't need for grepping. I know
| linux probably has orher mechanisms to achieve the same result
| but in this situation pledge seems natural.
| dan-robertson wrote:
| Probably that's right. But note that sort, for example, may
| need to write to (temporary) files. So disabling writes
| entirely is probably not a sufficient solution
| 10000truths wrote:
| You can achieve the same thing in Linux with seccomp.
| onli wrote:
| Why side effects? It's about the main effect of tools like rm
| or dd. Some things just don't fit to live previews like
| provided by UP.
| nabakin wrote:
| You don't want to accidentally delete or write to things and
| this command seems like an accident waiting to happen. A
| command whitelist would be very helpful but it doesn't look
| like UP has one.
| dmingod666 wrote:
| Yes, even a blacklist with obvious ones listed would be
| cool. Imagine just typing the characters rm and you're
| gone..
| akavel wrote:
| One of the reasons for not doing this is that I decided I
| don't want to lure the users into a false sense of
| security. A comment by _hnlmorg_ basically sums this up
| better than I could:
| https://news.ycombinator.com/item?id=26645875 (The other
| being, that I was cutting corners on my initial ideas
| _brutally_ trying to push the tool out of the door, and
| it 's still kind of a miracle I eventually managed to do
| that.)
| michaelmior wrote:
| Good point. That's a big problem. I could conceivably
| have a perfectly harmless tool where a prefix of the name
| is something dangerous.
| nonbirithm wrote:
| Chrome's developer tools have live preview, and they apparently
| accomplish this at the VM level by blacklisting a set of impure
| bytecodes when running the code in a separate evaluation
| context. If one of the bytecodes is found then it throws an
| exception.
|
| I kind of wish LuaJIT had something like that.
| alexf95 wrote:
| I didn't realise I needed this until I saw this post. Now I'm
| wondering why this isn't the standard already for every command
| line out there. Seems like a no brainer improvment with non to
| few downsides.
| hnlmorg wrote:
| I've been working on a way to implement this into my own shell
| (murex). The shell already has a builtin support for known
| "safe" commands, that is commands that are generally safe for
| background execution. The reason murex has this feature is
| because it supports executing the pipeline to generate more
| relevant autocompletion suggestions (eg when picking out nodes
| from a JSON or YAML file). Thus implementing an `up` like
| feature _should_ be a pretty easy addition....however I ran
| into a bug that stalled the development of said feature and it
| 's been in that stalled state for the last ~2 years.
|
| The reason development stalled for so long is because
| personally I'm not sure this is something that should be widely
| supported. Even with murex, which has an internal catalogue of
| known safe(ish) commands, the dangers of tools like this is
| pretty high. Running this against a regular shell which doesn't
| even do that seems like potential suicide.
| rocqua wrote:
| How about spinning up a container with the entire host file-
| system mounted in a Copy-on-Write configuration. This way you
| can trash the container as much as you want without affecting
| anything on the outside.
|
| Combine it with syscall logging and you might actually have a
| great tool for figuring out exactly how a command is
| affecting your system.
| hnlmorg wrote:
| That'll certainly help but you'd still have issues with any
| connected devices (USB, network API alls, etc).
|
| I don't think it is a solvable problem though. There is no
| way to know ahead of time which usage is safe and which is
| destructive because of the variety of UNIX-like
| configurations out there, variety of coreutils and variety
| of optional flags. Not to mention all the 3rd party tools
| out there. And some seemingly destructive usages are vital
| for safe usage (eg tmp files). Worse still, by the time you
| build a safe container you've have spent more time spinning
| up a sandbox than you've had spent using the CLI preview
| tools anyways (yeah ZFS could help here but few people run
| ZFS on their dev machines and you still haven't solved
| mocking APIs for connected devices).
|
| I just can't see how this can be achieved safely without
| completely redesigning how SYSCALLs work from the ground up
| to follow a method more akin to iOS or FireFox's on-demand
| permissions. At which point you've gone well beyond the
| realm of "yak shaving"
| rocqua wrote:
| Network API calls seems like a scary one. But for things
| like USB and sys-calls, I'd suggest a "block and log"
| method.
|
| Like "running this command shows this output (with some
| extra error messages" and tried to delete these files and
| do these system calls.
| hnlmorg wrote:
| But my point is even the safest of coreutils will make
| dozens of SYSCALLs. The moment you start pipelining
| commands there will be several writes since that's how
| pipelining works and if you break even one of those
| writes the entire pipeline fails and the usefulness of
| this pipeline preview utility falls flat.
|
| There isn't a way you can make this utility safe _and_
| useful.
|
| You can make it safe, but then you're drifting into a
| whole new field of computing with regards to analysing
| suspicious binaries. Unfortunately you then don't have
| reliable, accurate, real time pipeline previews. Or you
| can have the real time previews but you then have to
| accept there is some risk involved. But you can't have it
| both safe and risk free.
|
| This is why murex took the approach of having a safe list
| of trusted executables. It doesn't remove the risk but it
| at least reduces the risk to a subset of commands that
| are _typically_ read only. However even that is far from
| a perfect solution.
| rocqua wrote:
| Thinking about it more, the kind of guarantees you want
| are the kind given by a VM for running untrusted
| binaries. The only difference is that you don't mind
| read-access to the host system.
|
| Yes, there will be certain things that involve system-
| calls or network activity that your previewer will not
| accurately show. But those cases fall far outside the
| goal of "live preview of text output of data processing
| pipelines".
| hnlmorg wrote:
| No they don't. Believe it or not a lot of people make
| network API calls from the command line.
|
| I know this because I'm one of them. I've even published
| guides I'm on how to make said API calls and parse them
| all as one command pipeline.
| michaelmior wrote:
| Sounds like SELinux might do the trick here :)
| hnlmorg wrote:
| How do you differentiate between a write which is safe
| and a write which isn't? File system location gives a
| hint but it's no guarantee. Likewise for network
| connections, SELinux couldn't protect against an unsafe
| HTTPS POST request vs side-effect free GET request.
|
| And the amount of SELinux tuning you'd need to cover
| every single executable in every single configuration
| available on SELinux enabled machine (and even then, your
| limiting your tool to only work on systems with SELinux)
| is monumental, manual, inevitably error prone.
|
| There isn't a practical way of making this safe.
| michaelmior wrote:
| You're right, it wouldn't be perfect. I was being a bit
| facetious :) A perfect solution is never really possible
| and in this case, if you can't have pretty good
| guarantees of getting it right, probably best not to try.
| The solution here might just be to be _very_ careful when
| using such a tool.
| rocqua wrote:
| This thread was originally about using a container /
| overlay FS that has read access to the underlying
| filesystem, and writes happen into a temporary overlay.
|
| Network connections and certain system-calls might still
| be scary. However, VM / container systems that are
| supposed to defend against hostile hosted machines should
| also be safe here. The only difference is that here
| information leaking from the host into the VM is the
| point, instead of dangerous.
| hnlmorg wrote:
| No, this thread was originally about a version of said
| tool that I built and the dilemma I had about making it
| safe.
|
| A VM would be too slow to spin up to be practical for a
| live preview. A container would be too host specific. And
| neither solutions fully solve the problem because all
| you're addressing is file system access and, as mentioned
| several times already, that's only scratching the surface
| of the interactions a command line tool can make.
|
| My shell, murex, is designed around making networking
| access as seamless as writing to a local file. Bash does
| this too with its /dev/$datagram/$port pseudo file system
| too. So you can't just dismiss non-local changes as
| irrelevant.
| Saint_Genet wrote:
| Maybe I'm just old and grumpy, but what do you gain by this
| compared to just building up your pipe directly in the shell? Not
| having to hit enter once in a while?
| jolmg wrote:
| You still have to press Enter.
|
| I'm with you. I can't see myself preferring this over using my
| shell's interactive prompt, history, and a use of head/less at
| the end.
|
| Thanks to the author and OP for sharing and it's good for other
| people if they find this useful, though.
|
| EDIT: Maybe I could find this useful when the command that
| provides `up`'s input is not something that can be repeated for
| the same input. But even then, you can put it in the background
| with its output redirected to a file, then make the command
| that you would build inside `up` from a pipe like `tail -fn+1
| bg-job.out | cmd`.
| akavel wrote:
| Not much, really. Yet somehow those little things add up to
| something... faster :) Basically "just":
|
| - not having to append `| head -n $ROWS` or something similar
| on every command (yet having the view as-if-you-typed-it, but
| at the same time being able to scroll down further through the
| results; and having them visible as you continue tweaking,
| unlike `less` which you have to quit to tweak the pipeline),
|
| - not having to press up-arrow-key, then alt-b / alt-f to a
| position in the pipeline you want to tweak after every Enter
| (e.g. just before the `| head...`, but sometimes a bit more
| back),
|
| - sometimes also not having to dump the initial input from a
| command into a file, and to wait until it's finished, but being
| able to immediately start filtering even while the input is
| still being generated.
|
| Once I _just had those ideas_ , I felt pain every time I
| repeated the dance, enough that it made me push on _up_ even
| when I was super tired of working on it. The pain of _not
| having it when I know it could exist_ was even bigger.
___________________________________________________________________
(page generated 2021-03-31 23:02 UTC)