[HN Gopher] JavaScript for Shell Scripting
___________________________________________________________________
JavaScript for Shell Scripting
Author : gigel82
Score : 326 points
Date : 2021-05-07 05:28 UTC (17 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| christiansakai wrote:
| Lol. Anything that can be written in JavaScript will be written
| in JavaScript.
| Bombthecat wrote:
| It's getting really painful..
| devilduck wrote:
| Except for Javascript itself, since it can't do that
| chaostheory wrote:
| It seems like Steve Yegge predicted it 10 years before it
| happened. Spot on.
| bronlund wrote:
| Jesus Christ. As if Node wasn't bad enough.
| graderjs wrote:
| This is cool. _To the front page you go..._
| graderjs wrote:
| But then it happened. Don't hate. I was upvoter no. 2 and I
| just knew this thing was gonna be on PAGE1. I know it's an "non
| substantial comment", but still! :P ;) xx
| justsomeuser wrote:
| As soon as I hit an "if" branch in a shell script I move to JS,
| Python or Ruby
| dfinninger wrote:
| I think if's are generally alright. My personal bar is arrays.
| Anything beyond for x in a b c; ...
|
| is getting written in Python.
| justsomeuser wrote:
| It's not the "if" in particular, it's more of a signal that I
| am moving from a simple set of commands to something more
| complex with logic that will need objects, arrays and
| possibly a package manager.
| dahfizz wrote:
| Do you have node installed on all Linux boxes you interact
| with?
| justsomeuser wrote:
| I typically deploy my code in docker containers. Often bash-
| like scripts are for my dev machine.
|
| But if portability was an issue I would still probably move
| to something that outputs a single binary (Go, Rust).
|
| I just think bash is not a very good language compared to the
| alternatives.
| joncp wrote:
| Well, the whole reason to choose bash in the first place is
| that it just works on every single machine I have access to..
| Ruby and js aren't installed everywhere. As for python, I have
| to be willing deal with versioning hell and the whole
| virtualenv vs venv vs conda mess.
|
| Maybe C could work, but I have to make sure it's statically
| linked.
| szhu wrote:
| Compare to Deno, which also makes JS friendlier for shell
| scripting! https://deno.land/
|
| Coming from Deno, the single biggest advantage I see here is the
| handy tag function $: await $`cat package.json
| | grep name`
|
| Hoping someone will write a Deno port of this.
| maga wrote:
| That's rather trivial with Deno.run, here I just wrote a gist
| for that out of curiosity:
| https://gist.github.com/zandaqo/93004fb265146a95aadb28ec851a...
| japanuspus wrote:
| Thank you for posting -- I had never heard of tagged template
| literals!
|
| Reading the docs [1], it appears there is a `raw` property
| which might make this even simpler?
|
| [1]: https://developer.mozilla.org/en-
| US/docs/Web/JavaScript/Refe...
| maga wrote:
| `raw` would allow us to pass the input as is without
| escaping it using JS rules, but otherwise the code would
| remain the same, we still have to "stitch" both arrays
| together with the loops.
| Raed667 wrote:
| It would be an interesting CS problem to properly type the
| return values of random bash commands.
| orta wrote:
| This was kinda explored at the start of April this year:
| https://github.com/microsoft/TypeScript/pull/43480
| sneak wrote:
| Why would you want a shell script to be async? All this means is
| that you're going to type await 47 times. This isn't a network
| server, I'm not sure why this is the right tool for this job.
|
| Edit: that parallel rsync example answers my question nicely. I
| should have read more carefully.
| maxandersen wrote:
| Proper type safe Java scripts are nice too :)
|
| https://jbang.dev
| moocowtruck wrote:
| looks cool, but just want to drop this here
| https://book.babashka.org/ https://github.com/babashka/babashka
| iLemming wrote:
| What's the point? Most people choose to suffer, even when you
| offer them something that requires learning the thing just
| once. No matter how nice the thing is.
| c0l0 wrote:
| This embodies so much of what is wrong with some JavaScript
| programmers' (that I've known personally) mindset that I find it
| hard to distinguish from satire...
| christiansakai wrote:
| Welcome to GME/Doge era.
|
| Very bullish on JavaScript.
| classified wrote:
| Yep, the quest continues to rewrite every software there ever
| was in JS.
| the_cat_kittles wrote:
| nothing wrong with wanting to avoid the insane syntax of bash
| imo.
| c0l0 wrote:
| But you don't get to have that! You get the "insane" syntax
| oh bash, encapsulated in a ridiculously verbose and frankly
| quite unintuitive JavaScript contraption that seems to
| provide little, if any, benefit.
|
| Also, Shell scripting is mostly about correctly and safely
| interacting with other CLI-based tools - which this new
| thing wrapping a shell (and badly at that, see other
| comments in this thread) won't help you get right, either.
| the_cat_kittles wrote:
| i feel like you could avoid some of the kind of annoying
| syntax like array iteration and conditional syntax. would
| it be better to know that stuff 100%, and write it in
| pure bash? yeah. is it one of those todos that people
| like me never seem to get around to, and this is kind of
| a stop gap? maybe. it doesnt seem entirely useless
| kube-system wrote:
| There are some projects which I've migrated from bash to
| js, in the middle of the migration, we had a mix of both.
| This project would have been really helpful to make that
| migration quicker. Of course, it wasn't ideal, but
| there's always a journey before the destination.
| johndoe42377 wrote:
| This is getting beyond rediculous .
|
| It is already crossed into embarrassment.
| alpaca128 wrote:
| Neat idea, but what's the point of async-await when you just put
| await before 100% of calls like in the first example? Now you've
| got more to type for no gain.
|
| I don't get this popularity of async-await, especially in JS
| where I find its combination of syntax and absence of pre-run
| checks overly confusing and error-prone.
|
| And this, seriously? await $`exit 1`
| hutrdvnj wrote:
| It depends on how sequential your script is. It's not very
| uncommon to start a background task, do something in the
| meantime and continue when the bg task has finished. These
| types of async control flows are very easy to model in JS.
|
| And as far as I can see you don't have to await every single
| statement, because you can do multiple statements await $`echo
| 1; mkdir test; exit 0`
| intergalplan wrote:
| IMO it was always a fundamental mistake to force the programmer
| to deal with the event loop by default. Run async in the
| background, but present as sync to the developer unless
| otherwise requested, would have been a much saner design.
| Unless you're developing a framework or library, odds are good
| your ratio of leveraging async versus contorting yourself to
| make it (from your script's perspective) go away will be 1:10
| at best.
|
| JS keeps coming up with new ways to make this less painful, but
| it's ridiculous every time because it's a fundamental problem
| with how Node presents itself. A comical sea of "await"s in
| front of damn near every call is the modern version of this,
| and is utterly _typical_ in real JS codebases, but before it 's
| been callback hell, or screwing around with promises all over
| your codebase when 90+% of the time you just wanted things to
| execute in order (from your perspective), and so on.
| alpaca128 wrote:
| I think it's also made much worse by how JS doesn't care
| about whole classes of bugs. Forgot an `await` somewhere or
| called an async function assuming it's sync? Now you've got a
| bug and in some cases no idea where to look for it.
| TypeScript is also blind to it (although now that I think of
| it a linter might flag it?).
| intergalplan wrote:
| A really good point, and all the more reason to make
| developer-visible async behavior something the developer
| has to to _ask for_ , even if the call is in fact async
| under the hood and might let, say, code handling another
| request run while it's waiting on I/O.
|
| I think a pattern where there are one or two great places
| at the lowest level of a Node program for program flow to
| act async, and then a bunch of business logic where it
| _rarely_ is (probably running "under" the part where async
| makes sense, if you take my meaning) is far more common
| than those where async-friendly flow is what you want for
| over 50% of calls. "Call this chunk of code async, but run
| everything in it exactly in order" is super-common, and the
| interface to achieve that is exactly backwards in Node.
| deckard1 wrote:
| I can't even count the number of times I've seen a unit
| test giving a false positive (or, perhaps more accurately,
| not even run) because the developer forgot to properly use
| async/await or use the done callback.
| jiofih wrote:
| If you want to have the same code structure as this, but
| without await, you're gonna implement your own queuing system
| (every call adds an operation to a queue). Many old tools
| operated this way, before ES6.
|
| The problem with that it doesn't meld together with most third
| party libs, since they will have no knowledge of your queue and
| will just execute immediately or out of order.
| CGamesPlay wrote:
| I'm really unclear on what repository you're commenting on. The
| first example shows an example of executing commands in
| parallel, and does 3 jobs with 1 await. await
| Promise.all([ $`sleep 1; echo 1`, $`sleep 2;
| echo 2`, $`sleep 3; echo 3`, ])
|
| The only `exit 1` in the README is an example on how to handle
| cases where your job fails, so don't really understand what
| your complaint is.
| alpaca128 wrote:
| Okay, but why would I do that instead of just
| sleep 1 && echo 1 & sleep 2 && echo 2 & sleep
| 3 && echo 3 & wait
|
| and be done with it? Now I don't need to prepend 'await
| $`...`' in front of every other command.
| confiq wrote:
| I totally get your point! As someone who is not coming from
| JS, I agree with you!
|
| But you must understand that I/O in JS is async, it's how
| the language is build! People who are advanced in async
| programming find this very comfortable.
| jbverschoor wrote:
| What load of bs.
| jiofih wrote:
| that's just ignoring all the other benefits, or pitfalls of
| bash that you have to worry about. This project is not
| about enabling better parallelism.
|
| (I've written massive bash scripts, and now strongly prefer
| Ruby / python / whatever is available in the system)
| blablabla123 wrote:
| Also I think you need to add a trap to make sure the
| processes exit on ^C.
| alpaca128 wrote:
| I agree that for complex scripts bash isn't ideal. But my
| criticism was about unnecessarily verbose syntax. Bash is
| synchronous by default, this project is the exact
| opposite.
| mirekrusin wrote:
| If process in the middle exists with non zero exit code,
| your whole script won't exit with non zero exit code. You
| have to collect exit codes and check them.
| Normal_gaussian wrote:
| when I write scripts in js or python over bash, its to
| leverage their access to data structures, error handling,
| and libraries. Often I use TypeScript, because types are
| good and concurrent i/o is a first class feature.
|
| This means that promise.all probably looks something more
| like const lastWeek =
| DateTime.now().minus({ days: 7 }).toISOString() const
| pods = JSON.parse(await $`kubectl get pods -o json`)
| useValidate(pods, isKubeCtlPodList) await
| Promise.all( pods.filter(pod =>
| !protectedPods.includes(pod.name)) .filter(pod
| => target.test(pod.name)) .filter(pod =>
| pod.created < lastWeek) .map(pod => $`kubectl
| delete pod ${pod.name}`) )
|
| Id certainly encapsulte my calls, parsing, and validation
| differently in real code, but you get the gist.
|
| In shell I can wrangle jq, date, xargs and get the same
| result; only this is waay easier to write, gives me
| validation and usable error messages, and can be altered
| much more easily than shell.
|
| I write complex bash scripts somewhat often. They can be
| quicker to write, they are great for one-offs. But if I'm
| coming back to something non-trivial or if its getting
| deployed into production I want a language like js or
| python (or nearly any other language. C++, OCaml, Java all
| help more than they hinder).
| devit wrote:
| Yeah, feels like they should use the Haskell IO monad "do"
| notation style where the await is implicit unless you use let.
|
| This would require the scripts to no longer be vanilla
| JavaScript, but it seems easy to extend sucrase/babel to do
| that transpiling.
| efortis wrote:
| In plain JS: pipe(a, b, c) is equivalent to
| a().then(b).then(c) function
| pipe(...functions) { (async () => { let
| accum; for (const fn of functions)
| accum = await fn(accum); }()); }
| eyelidlessness wrote:
| All IO in JS is async, you either have a Promise-based API
| (which can be sugared with await) or callbacks. The only
| exceptions are Node APIs which are explicitly synchronous in
| their names and CommonJS require. All of which are either
| discouraged or being generationally phased out (CJS->ESM).
|
| That it extends to the $ function (anything preceding a
| backtick in JS is a "tagged literal" but also a function call)
| is just API consistency because it can't know whether you're
| calling exit versus something which actually performs IO. So it
| always returns a Promise.
| megous wrote:
| > All IO in JS is async, you either have a Promise-based API
| (which can be sugared with await) or callbacks.
|
| No it's not. There are all kind of *Sync functions even in
| nodejs and in browsers (there was sync variant of XHR), and
| JS engine generally doesn't care if you block in your
| functions or not. JS engines don't even have an event loop,
| that's just a construct within the app that uses the engine,
| like nodejs or whatever.
| shepherdjerred wrote:
| In this case it's probably fine to block, but in most
| others it's absolutely not.
| Raed667 wrote:
| I'll give you 2 examples and tell me which is more readable:
| console.log('start') doA() .then(() => {
| doB() .then(() => { doC()
| .then(() => { console.log('finish')
| }) }) }) .catch((error) => {
| console.log(error) })
|
| Or console.log('start') try {
| await doA() await doB() await doC()
| console.log('finish') } catch (error) {
| console.log(error) }
| leipert wrote:
| No need to nest promises:
| console.log('start') doA() .then(() => doB())
| .then(() => doC()) .then(() => console.log('finish'))
| .catch((error) => { console.log(error) })
| BeefWellington wrote:
| Is this not less typing and as clear?
| console.log('start') doA() .then(doB)
| .then(doC) .then(() => console.log('finish'))
| .catch((error) => { console.log(error) })
| mdaniel wrote:
| It's been my experience that doing it that way can fiddle
| with the `this` value of those B and C functions,
| although I do in general agree there's usually no need
| for the outer wrapping arrow function if the interior one
| doesn't care about `this`
|
| I believe your .catch can similarly be
| `.catch(console.log)` for the same reason
| Raed667 wrote:
| i was being a bit facetious. "Real-life" code is usually
| more complex and nesting is needed when you have
| conditional logic (call doC or doD depending on the return
| value of the previous call etc..)
| trog wrote:
| Here's a lazy third option:
| console.log('start') try { doA()
| doB() doC() console.log('finish')
| } catch (error) { console.log(error) }
|
| It seems to me async/await is a desperate attempt to try to
| make JavaScript less painful to deal with because of its
| asynchronous execution nature.
|
| I am not a JS expert and have not had many opportunities to
| work with it extensively, but I confess I have always
| struggled with this when dealing with JS; I have often
| wondered if the (alleged?) performance gains from executing
| JavaScript like this outweighs [what I have always perceived
| as] the significant extra verbosity and complexity required
| to manage simply executing things in order.
| jmull wrote:
| I agree with this syntax, but it's too late now (for
| JavaScript).
|
| To clarify, JavaScript _could have_ made await the default
| for async functions, so that if you called an asynchronous
| function without putting any keyword in front of it,
| execution would block until the operation completed. In
| that design, there would be some keyword you would use when
| you did _not_ want to wait.
|
| What they did: you have to _opt-out_ of async execution by
| adding the await keyword.
|
| What they could have done: you _opt-in_ to async execution
| by adding a keyword.
|
| They just chose the wrong default (IMO). It's not a big
| deal -- it's easy enough to type "await" here and there.
| It's bad, though, that you don't really see the basic flow
| of control from examining the code making function calls.
| You also need to consult the function definitions to see
| which ones are async.
|
| Another option would be to have no default, and instead
| require that async functions be called with an explicit
| indication of whether they are to execute sync or async.
| That too heavy-handed, IMO. Fine for a statically checked
| language but not a dynamic one.
| trog wrote:
| Thanks, this is basically what I was trying to say but is
| written so it actually makes sense.
| Raed667 wrote:
| I wasn't attempting to defend JavaScript design choices.
| I'm just stating that working with blocking vs non-blocking
| code is now MUCH easier (and less error prone) since we
| have async/await.
|
| Callback-hell and Promise-hell were real issues that
| plagued any project of significant size.
| trog wrote:
| Oh yeh sorry I didn't mean to give the impression I
| thought you were wrong. I 100% agree with you.
|
| I remember the first time I experienced "callback hell"
| (2016, for me, but I'm sure it was a huge problem for
| others before then) when I was doing some JavaScript
| stuff implementing Keybase's library for GPG support - I
| learned they'd built a whole separate JavaScript thing
| called IcedCoffeeScript[1] specifically to add
| await/defer support to get rid of those huge callback
| pyramids.
|
| 1. http://maxtaco.github.io/coffee-script/
| yoz-y wrote:
| I don't understand your example, if would only work if your
| doA,B,C functions were synchronous, which completely
| changes the paradigm and use case. It's simpler yes, but
| very different. (simple example: if that function were to
| be run when you click on a button, now you have a frozen
| UI.)
| trog wrote:
| Yep. I am just wondering if the paradigm of JS being
| asynchronous is just not worth the added effort in code
| complexity and all the workarounds that have had to be
| added in over the years to try to make it manageable.
|
| I personally find it a chore to work in JS compared to
| other languages that work the opposite way - everything
| is synchronous and things become a chore when you want to
| deal with async stuff. Again I have limited JS experience
| and have never enjoyed working with it (I just am not
| interested in front end stuff) so I'm sure it's stuff
| people get used to.
| noahtallen wrote:
| Could you clarify what the paradigm of JS being
| asynchronous means? Usually, async calls are actually
| "async" in that you're doing a network request or
| something actually asynchronous. The reason they have to
| be async is that you cannot just block all execution or
| else other things can't happen while the network request
| is happening. Which means your entire application grinds
| to a halt. Other languages solve this in a similar way
| IMO, with async/await and actual threads. JS doesn't have
| threads, so you can't fork a process to handle network
| stuff apart from UI. I personally think that async/await
| is a lot more straightforward to use than forming a
| process.
|
| Additionally, async/await allows you to wait when you
| need to. You don't have to await a promise -- you can
| just call it, and then it will work in the background.
| You'll just not be able to respond when it finishes work.
| (Because you aren't "wait"ing for it.)
|
| What are some other examples with easier async in other
| languages? I'm just confused what you're trying to get
| at. Everything in JS is synchronous. You only have async
| when you have async. Which is a lot, because a lot of
| stuff in the web is async.
| trog wrote:
| Yep, sorry & thanks for your polite post in response to
| my basically unhinged ramblings (late night after a tough
| week, always a bad time to try to do anything).
|
| The poster jmull above basically wrote[1] what I was
| trying to say much more succinctly; this other post[2] is
| also more clear.
|
| Really I was just being snarky about the idea of shell
| scripting where you have to type 'await' in front of
| every command you want to run in a three-line file, like
| in the examples presented for this tool.
|
| 1. https://news.ycombinator.com/item?id=27080734
|
| 2. https://news.ycombinator.com/item?id=27077752
| bryanrasmussen wrote:
| The complaint really is that if everything is sequential it
| is a fault in the language to make you explicitly say it
| every time, not that await is somehow worse than promises.
| andresgottlieb wrote:
| Is everything sequential?
| alpaca128 wrote:
| When I need to run something async in Bash I can just add
| a `&`. Done. The majority of my scripts are purely
| sequential, though, as the language lends itself well to
| sequential data processing/piping.
|
| I'm not seeing a benefit by wrapping almost every single
| command in 'await $`...`'. I get why you'd want to wrap
| Bash in a different language, especially when handling
| numbers. But I'd rather use something like Python than
| this verbose trickery.
| vagrantJin wrote:
| I agree. My scripts are also rigidly sequential and
| verbose which is easier for me and other devs after me to
| have a simple mental model of what is going on at a
| glance. Which I think saves time in the long run because
| any dev can easily understand and make intelligent
| additions to make it better. For complex scripts, Python
| is the goto and isn't terribly difficult to grasp either.
| bryanrasmussen wrote:
| I didn't say it was, but in the example given basically
| everything was, hence the complaint the parent made was
| that in JS we are often writing a lot of await, or
| promises or what have you because we have a lot of things
| that need to be done sequentially in a particular part of
| the program, or, in a small script, where everything
| needs to be sequential.
| Raed667 wrote:
| It's not really a JS issue here, just the ZS project
| decided to make `$` function async (for some good reason
| I'd think) but they could have gone the other way and
| make it synchronous by default.
|
| I don't believe this is a language specific choice, they
| could have done the same with C#
| _ZeD_ wrote:
| well, almost?
| chetangoti wrote:
| This is good, enabling people with Javascript proficiency write
| complex scripts IMO.
|
| There is also https://github.com/shelljs/shelljs which is
| implemented on top of nodejs APIs.
| niffydroid wrote:
| I pretty much got rid of my bash scripts and just replaced them
| using shelljs. Makes it a lot easier and quicker to maintain,
| it means any developer can jump in fix it and add to it.
| jeswin wrote:
| If you like shelljs, then check out https://bashojs.org (mine).
| # Example, list all dev dependencies: cat package.json |
| basho --json 'Object.keys(x.devDependencies)'
| rattray wrote:
| Honestly the more appealing aspect than the "await $`cmd`" syntax
| is that it imports a bunch of handy libs by default.
|
| For some reason there's just something that rubs me the wrong way
| about having a bunch of requires at the top of a bash script (or
| cluttering my package.json with handy tools I use once in a
| script). But you're gonna want things like (promisified) fs, os,
| chalk, readline, node-fetch, etc quite a lot of the time.
|
| Definitely wish they'd included shelljs though; almost strange
| not to.
|
| I hope they add (or already have) syntax highlighting as .mjs
| without the file extension (just the shebang) for GitHub, VSCode,
| etc.
| tln wrote:
| Currently, your scripts have to use the .mjs syntax to work
| because zx uses import() to actually execute the script.
| stunt wrote:
| Javascript is just the new utility knife that all devs have in
| their pocket now mostly because you don't need to master it to
| build something with it, but also because it's acceptable to
| write improper JS code.
|
| PHP used to be the utility knife for a while for the same
| reasons. It was easy to learn, and run it. And it was ok to write
| bad PHP code. I remember most of the exploits and remote shells
| were written with PHP because it was the easiest language for
| hackers to learn.
| Toutouxc wrote:
| JS as the utility knife? No thanks, I'll write my quick scripts
| in a language with an actual useful standard library.
|
| That would be Ruby for me, but I also accept Python and maybe
| dozens of others.
| nicbou wrote:
| Python is a much better utility knife in my opinion, because
| it's used in more domains than just web development, and comes
| bundled with most operating systems.
| devilduck wrote:
| python is Way better than js but js people can't cope with
| other languages for some reason
| [deleted]
| [deleted]
| parhamn wrote:
| Curious. When will JS tooling ecosystem support executing
| typescript by stripping the type information? Are there any plans
| to add generic soft type annotations to the spec? It'd be great
| to write these in TS without changing the runtime tooling.
| rockwotj wrote:
| I've found esbuild to be great for this:
| https://esbuild.github.io/content-types/#typescript
| campers wrote:
| What about using ts-node? You can run typescript files directly
| with that. I use it for my scripting purposes.
| parhamn wrote:
| It's a great tool. It wouldn't work in this case would it?
| Also, IIRC it checks types, doesn't just strip (sometimes I
| prefer to just run the thing and do my type checking at the
| IDE/build level). Also still leaves the multiple tools
| problem.
| campers wrote:
| You can use the --transpile-only option to skip type
| checking. I'm going start using that myself for scripts I
| know are already good! Just tried it and makes a good
| difference to startup time.
| CGamesPlay wrote:
| Well `tsc` is the program that "strips" types after
| checking them. To be clear what you're asking for is a
| JavaScript runtime that supports parsing types but ignoring
| any errors it finds? Is there any computer programing
| language that has this behavior?
| goodside wrote:
| Python, pretty much exactly.
| demurgos wrote:
| Babel can do this if you wish to only strip TS extensions
| without typechecking.
| jeswin wrote:
| You can already do that with deno.
| lucasmullens wrote:
| > JavaScript is a perfect choice
|
| I mean I love JS as much as the next person, but _perfect_? No
| way.
| juddlyon wrote:
| Better than bash but nothing beats Python for scripting IMO. I'm
| a mediocre programmer and search results are rich with Python
| examples for anything under the sun.
| louis-lau wrote:
| To be fair, the same goes for javascript.
| franciscop wrote:
| I was hoping for an article on how to use Node.js for normal
| scripting, since it's already pretty close to what it's shown in
| this library. I've written two libraries to help with scripting
| in Node.js:
|
| `files`: https://github.com/franciscop/files/
| import { read, walk } from 'files'; // Find all of
| the readmes const readmes = await walk('demo')
| .filter(/\/readme\.md$/) // Works as expected!
| .map(read); console.log(readmes); // ['#
| files', '# sub-dir', ...]
|
| `atocha`: simplest cli runner (no escaping though!)
| https://github.com/franciscop/atocha/ import
| cmd from 'atocha'; // Any basic command will work
| console.log(await cmd('ls')); // Using a better
| Promise interface, see the lib `swear` console.log(await
| cmd('ls').split('\n')); // Can pipe commands as
| normal console.log(await cmd('sort record.txt | uniq'));
|
| Both of them are wrapped with Swear, a "promise extension"
| (totally compatible with native promises!) so that's why the
| first example works. You can build operations on top of the
| return of the promise, so that these two are equivalent:
| // Without `swear`: const list = await walk('demo');
| const readmes = list.filter(item => /\/readme\.md$/.test(file));
| const content = await Promise.all(readmes.map(read);
| // With `swear`: constn content = await
| walk('demo').filter(/\/readme\.md$/).map(read);
| hackerbrother wrote:
| Deno is great for this also! (https://deno.land/)
|
| This short wrapper is great for doing asynchronous SSH
| commands. https://github.com/gpasq/deno-exec
| pwdisswordfish8 wrote:
| I hope the $ interpolator at least performs escaping instead of
| blindly concatenating shell scripts?
|
| Ah, who am I kidding.
|
| https://github.com/google/zx/blob/5ba6b775c4c589ecf81a41dfc9...
| function substitute(arg) { if (arg instanceof
| ProcessOutput) { return arg.stdout.replace(/\n$/, '')
| } return arg } // [...]
| let cmd = pieces[0], i = 0 for (; i < args.length; i++)
| cmd += substitute(args[i]) + pieces[i + 1] for (++i; i <
| pieces.length; i++) cmd += pieces[i]
|
| Sigh... yet another code injection vector. And to think the whole
| template literal syntax was specifically designed with this in
| mind. Have people not learned from 20 years of SQL injection?
| akst wrote:
| This feels like a pretty dramatic response when this is already
| a problem with child_process.exec and most shell scripting
| languages in the first place.
|
| I'm not sure what you're expecting if you're taking arbitrary
| IO output to build up a process with arguments separated by
| spaces. A lot of things had to go wrong before you get to this
| point.
|
| If we're being realistic here, this library's likely intended
| use is for this is smallish scripts anyways... not large pieces
| of software that are creating commands on the fly to from
| arbitrary IO
| goshx wrote:
| > If we're being realistic here, this library's likely
| intended use is for this is smallish scripts anyways... not
| large pieces of software that are creating commands on the
| fly to from arbitrary IO
|
| If we're actually being realistic here, we know users will
| use this for whatever scenario, regardless of the author's
| intent.
| pwdisswordfish8 wrote:
| Ordinary variable substitution in shells splits on spaces,
| which is still bad, but at least doesn't _immediately_ lead
| to arbitrary code execution. I'm expecting at the very least
| an equivalent of Python's shlex.quote. This is supposed to be
| an improvement on the status quo, not a regression.
| lhorie wrote:
| But with child_process.exec you can at least pass values via
| env and have the shell script come from a file (which you can
| throw shellcheck at)
|
| Also, spawning node from shell to spawn shell to spawn
| something like ls is madness. Node has fs.readdir already and
| there are util packs on NPM like fs-extra and friends.
| seniorsassycat wrote:
| You should use child_process.execFile or the execve equiv in
| your language. Shell has expansion issues but rigorous
| quoting helped by shellcheck make it safe.
|
| And zx's `$` could make better use of tagged template
| literals.
|
| Something like this, tho it isn't correct
| function $(strings, ...args) { const cmd = [];
| for (const part of strings) {
| cmd.push(...part.split(/\s+/)); if (args.length >
| 0) { cmd.push(args.shift()); }
| } return child_process.execFile(cmd[0],
| cmd.slice(1)); }
|
| https://developer.mozilla.org/en-
| US/docs/Web/JavaScript/Refe...
| jiofih wrote:
| are you bringing in user input from a web form into your shell
| scripts?
| judofyr wrote:
| It also means that accidentally adding a space somewhere
| ("$HOME/go" -> "$HOME /go") can have catastrophic effect. I
| wouldn't dare write a single "rm" if I'm not 100% sure the
| argument is being quoted.
| gnfargbl wrote:
| The developers of a not-insignificant portion of IoT firmware
| absolutely are bringing user input in from web forms and
| chucking it into shell scripts. And unfortunately, it's that
| same class of developers that are disproportionately likely
| to pick up _zx_ and run with it.
|
| The point is, at this stage in the evolution of internet
| security, we pretty much know where the bugs come from.
| Injection attacks are still a huge practical problem. It
| would be nice if new scripting languages reduced that attack
| surface rather than increasing it.
| pwdisswordfish8 wrote:
| Who knows, maybe? Or maybe I just want to process file names
| with spaces in them? Maybe I don't want to worry about
| apostrophes in people's surnames?
| tutfbhuf wrote:
| How do you deal with it, when you are writing bash scripts?
| selfhoster11 wrote:
| This is usually answered with "poorly", or "with great
| difficulty".
| pwdisswordfish8 wrote:
| By not using eval (and sh -c, and everything equivalent)
| unless it's absolutely unavoidable and always quoting
| variables. The $ construct acts basically like (the
| shell's) eval. Proper escaping is an absolute must here.
| 40four wrote:
| Indeed, I'm not a bash expert, but I've heard multiple
| times that using eval is a bad idea. Your point is a good
| reminder of why.
| TechBro8615 wrote:
| I'm pretty sure that "eval is a bad idea" was mentioned
| in the "thought terminating cliches" topic on Ask HN a
| few days ago :)
|
| In bash, `eval` is a footgun like any other. You can use
| it, but you just need to be aware of where your toes are
| when it shoots.
|
| It's usually a "bad idea" in the sense that if you think
| you need to use it, you probably don't, and 90% of the
| time, there is an easier way to accomplish what you want
| to do. The next 5% of the time, using `eval` might be
| easier but will also create maintenance debt with its
| overgeneralization. And the final 5% of the time might be
| actual legit use cases for `eval`.
|
| I just grepped my codebase for `eval` and I almost never
| use it. One example of the "overgeneralized" 5% might be
| when I realized I could use `eval` to set "variable
| variables" (i.e. the name of the variable is itself a
| variable, taken from a function argument). It was cool,
| but I ended up deleting it in favor of a more concrete
| solution.
|
| Personally, if I'm hesitating to use `eval`, it's usually
| not for any security reasons. In general, my bash scripts
| only exist in dev machines and CI runners, and I don't
| copy them into the application containers that are
| exposed to a live runtime environment with untrusted
| users. So for CI/dev scripts, I can safely assume the
| code will only run in CI/dev, and therefore I can trust
| arbitrary user input (which I can of course still
| validate).
| DonHopkins wrote:
| By immediately invoking Python and getting the hell out
| of bash.
| FranchuFranchu wrote:
| Or xonsh
| Spivak wrote:
| Python is my absolute favorite language but it's not
| suitable for the kinds of things you would use bash for.
|
| This is the real code that a Ansible uses to run a shell
| command correctly and is 350 lines and is still a small
| subset of the features of a single line of bash. https://
| github.com/ansible/ansible/blob/a2776443017718f6bbd8...
|
| The Python code to do what a single mv invocation does is
| 120 lines https://github.com/ansible/ansible/blob/a277644
| 3017718f6bbd8...
|
| People always focus on the footguns that exist in Bash
| the language but ignore how much systems programming is
| abstracted away from you in the shell environment.
|
| In Bash you can enter a Linux namespace with a single
| nsenter invocation. If you want to do the same in Python
| you have use ctypes and call libc.clone manually.
| nonameiguess wrote:
| How is that a remotely real comparison? The ansible mv
| function deals with preserving SELinux context, which mv
| doesn't do, and it automatically deals with a whole lot
| of common error conditions, whereas mv just fails. If you
| just want to replicate mv, Python has shutil.move, one
| line of code. Ansible is trying to do a lot more.
|
| By the way, I don't know if this is the canonical
| implementation, but FreeBSD mv is 481 lines of C:
| https://github.com/freebsd/freebsd-
| src/blob/master/bin/mv/mv...
| BiteCode_dev wrote:
| The code you are showing does a lot more than MV:
|
| - it's portable accross OSes (and keep flags if the OS
| supports it, deal with encoding, etc)
|
| - it ensures selinux context is saved if there is such a
| thing
|
| - it has proper and rich error communication with the
| calling code
|
| - it's includes documentation and comments
|
| - it outputs json par parsing and storage
|
| No to say "mv" is not awesome, because it is. There is
| much more boiler plate in python, and is why I'll often
| do subprocess.check_call(['mv', 'src', 'dst']) if my
| script is linux only.
|
| But you are pushing it
| overtomanu wrote:
| i think its not fair comparison. mv command
| implementation in 'C' might have more lines of code.
| Maybe we should complain that there are no OOTB library
| functions in python to move the file.
| Spivak wrote:
| I don't really care how many lines of code there are in
| an implementation. I care how many lines of code I
| actually have to write.
|
| Python has shutil.move and os.rename but the Ansible
| example is to illustrate that there's a lot of code that
| needs to surround those calls to make them useful and
| they're not 1-1.
| mirekrusin wrote:
| It's called shell microservice, you don't need b/e developers
| anymore.
| jerf wrote:
| "are you bringing in user input from a web form into your
| shell scripts?"
|
| This is shell scripting. We have literally decades of
| experience with these. We know for a positive fact that
| simple concatenation is dangerous. The contents of files, the
| names of files on the file system, and all the other things
| that shell scripts normally encounter are _perfectly
| sufficient_ to wreck your day if you are too casual about
| them. Should you reply with something incredulous about this
| claim, be prepared for dozens of people to jump in with their
| war stories about how a single space in a filename in an
| unexpected place cost their business millions of dollars
| because of their shell script screwup. (Obviously, they are
| not that consequential on average, but the outliers are
| pretty rough in this case.)
|
| Shell scripting is frankly dangerous enough just with what is
| on your system already; actually hooking up user input to it
| is borderline insane. Shell scripting would have to level up
| quite a bit to only be something to be concerned about when
| "user input" was being fed into it.
|
| Learning to do half-decent shell scripting in most shells
| consists about half of learning the correct way to do things
| like pass through arguments from a parent to a child script,
| because for backwards compatibility reasons, all the easy
| ways are the wrong ways from the 1970s. It's nice when a more
| modern take on shell scripting is nicer than that.
|
| I will also say when I'm evaluating libraries for things like
| shell scripting, I look for things like this, and it
| definitely doesn't score any points when I see stuff like
| this.
| chrisfinazzo wrote:
| Just using `set -euo pipefail` will prevent many stupid
| things, but then again, conventional wisdom these days just
| seems to be to not use a shell if you can help it.
|
| https://sipb.mit.edu/doc/safe-shell/
| jerf wrote:
| That's another example of what I mean, learning the magic
| invocations that amount to "Oh, please shell, act
| _halfway_ like this is the 21st century, please? " It's
| like how I still have "#!/usr/bin/bash NL use strict; NL
| use warnings;" still burned into my fingers for Perl.
| (AIUI that's obsolete now but I never got to upgrade to
| the versions where that became obsolete, and now I'm just
| out of it.)
| lhorie wrote:
| > We have literally decades of experience with these. We
| know for a positive fact that simple concatenation is
| dangerous.
|
| Yes and no. I agree with the overall premise that the
| footguns are well documented, but at the same time,
| projects like this show that there are still large segments
| of developers who will gleefully shoot themselves in the
| foot because they never took the time to learn shell, or
| they just never had the opportunity to earn the battle
| scars.
|
| At least Google has a bug bounty program.
| lobstrosity420 wrote:
| Escaping in the $ interpolator is a planned feature. You
| definitely have a point, but the project was made public 4 days
| ago and deserves a little bit of slack don't you think?
| pwdisswordfish8 wrote:
| Injection vulnerabilities are one of the most pervasive and
| well-known kinds of bugs, _especially_ in shell scripting.
| This should have been considered on day 1 and implemented
| before anything else started depending on it being otherwise.
| Fixing bad designs after the fact is long and painful. Just
| look at PHP.
| lobstrosity420 wrote:
| We are talking pre alpha stages of development here,
| nothing that depends on this can reasonably expect for the
| API to not break several times over in the near future.
| This software is still figuratively on it's day 1 of
| development. Weather you or someone else would have
| implemented this feature faster matters very little to me.
| pwdisswordfish8 wrote:
| Then it should have carried prominent notices to that
| effect or at least used a different version number.
| Version 1.0.2 hardly screams 'pre-alpha'.
|
| https://github.com/google/zx/blob/5ba6b775c4c589ecf81a41d
| fc9...
| maxgee wrote:
| no?
| cutemonster wrote:
| Looks nice I think :-)
|
| I wonder if it can work with Deno, and maybe even with Deno's
| security features:
| https://deno.land/manual/getting_started/permissions
| cltsang wrote:
| Atwood's law is still accurate after more than a decade:
|
| "Any application that can be written in JavaScript, will
| eventually be written in JavaScript." - Jeff Atwood
|
| source: https://blog.codinghorror.com/the-principle-of-least-
| power/
| ilyash wrote:
| What if I told you there is a modern programming language
| designed specifically for DevOps?
|
| https://ngs-lang.org/
|
| Born out of frustration with bash and Python.
|
| and ... nope, never considered JS for that type of scripting.
| austinshea wrote:
| Do people actually find this more convenient than bash?
|
| Their own readme isn't trying to suggest that it's an
| improvement, just some sort of convenience.
| mewpmewp2 wrote:
| More productive for folks who are comfortable with JS, but not
| bash, which would be very common for front end folks.
| ttt0 wrote:
| Will people crucify me for suggesting that maybe you should
| also learn a language that's not JavaScript?
| mewpmewp2 wrote:
| I should, but not sure if learning bash specifically is
| best for productivity. Especially if I don't use it daily,
| I will forget syntax etc, will still have issues
| understanding what is going on quickly, more likely to
| introduce bugs when trying to improve something etc.
| gen220 wrote:
| The shell is a long-term investment, and decades of
| programmers are happy to quietly testify to its
| continuing dividends.
|
| POSIX shells (and OS) are the common thread underlying
| all *nix-based software.
|
| Bash prioritizes terseness and backwards compatibility;
| which is occasionally contradictory with UX, but are
| valid prioritizations nonetheless.
|
| Also, learning something new gives you a more nuanced
| appreciation of what you already knew. It's a mind-
| expanding experience, even if it's occasionally
| frustrating.
|
| TLDR. I'd advocate for it. It's OK if things break or
| fall apart on the road, this is the price of knowledge
| acquisition.
| vedantroy wrote:
| This reminds me of my `jsmv` tool, which I use for manipulating
| the file-system in place of Bash / find:
|
| https://github.com/vedantroy/jsmv
| zmix wrote:
| Nashorn (Oracle's, now discontinued ECMAScript implementation for
| the JVM and Rhino's successor) can be used for Shell scripting,
| too, when started with the `-scripting` parameter. Here is a
| little overview[1]. I think, GraalVM's new ECMAScript
| implementation is compatible with both NodeJS and Nashorn,
| including the scripting features.
|
| [1]:
| https://docs.oracle.com/javase/8/docs/technotes/guides/scrip...
| brabel wrote:
| GraalVM's JS engine[1] is an implementation of ECMAScript 2021
| and as such, it's a drop-in replacements for Node.js and npm.
|
| It also supports some Nashorn features for interop with Java,
| but for many things to work, you need to enable it explicitly,
| especially the compatibility mode flag (`-Dpolyglot.js.nashorn-
| compat=true`).
|
| See
| https://github.com/oracle/graaljs/blob/master/docs/user/Nash...
|
| [1] https://github.com/oracle/graaljs
| ilaksh wrote:
| Maybe there could be a shorthand for the await keyword. Such as
| instead of await mycall()
|
| Maybe mycall()!
|
| Or something. Or perhaps a flag you could set at the top of the
| script that would transform every call into an await call.
| vedantroy wrote:
| Does anyone know why Bash has low usability?
|
| It seems like Bash has tons of footguns and unintuitive syntax.
| Is this just because the language grew organically?
| [deleted]
| ape4 wrote:
| Would the $ conflict with jQuery.
|
| (Of course jQuery's main job is DOM manipulation which isn't
| needed here)
| lobstrosity420 wrote:
| You can load JQuery into a different global variable if you
| want. There are cases where you might want to parse a DOM in
| the context of shell scripts, such as scrapping some web page.
| Although JQuery wouldn't be the tool I'd personally reach for
| in this case.
| LocalPCGuy wrote:
| This isn't a browser-based script, so no real fear of conflict.
| Plus, it's basically past time to retire jQuery anyways,
| outside of maintaining legacy apps.
| Diti wrote:
| Why is Javascript a << perfect choose >> for shell scripts?
| Shacklz wrote:
| I had very strong anti-javascript opinions for the longest of
| times, and still feel that writing any non-weekend-project in
| JS is a mistake.
|
| That being said, I have grown very fond of writing
| "plumbing"-scripts in nodeJS. The syntax is so much more sane
| than bash, the documentation of nodeJS is actually pretty good
| in my opinion, and once nodeJS is installed and available, its
| super easy to mash something together to get working. For small
| scripts, the uglier parts of JS do not really matter, and while
| nodeJS does have some APIs that feel a bit odd to use at times,
| it's at least well-documented with examples/guides all over the
| web.
|
| Some folks use python for that exact purpose, but I always felt
| that getting something going takes me more effort than nodeJS
| (the fact that I'm just not fond of Python as a language might
| also have something to do with this).
|
| All in all... for folks still using bash, I really recommend to
| give JS a try.
| golergka wrote:
| Have you tried Typescript? It has one of the best type
| systems among mainstream (not FP) languages, IMO.
| Shacklz wrote:
| TS is probably my favorite language out there, so yes, I
| definitely did :)
|
| But just for "one-off"-scripts, JS usually does the job
| pretty well. For bigger scripts or scripts where I want to
| reuse/share parts, typescript is an option - but it comes
| with a few hurdles: Most of the scripts I'm talking about
| work with zero npm-dependencies, while typescript adds at
| least one.
|
| Furthermore, I either have to take care of compiling it, or
| use ts-node - one more dependency; and in both cases, I
| can't avoid the usual es6 vs. commonJS shenanigans. It's a
| bit of overhead to get going; always worth it if I spend
| more than a few hours on a script, but I can do without if
| it's just a few lines that I will hardly ever see again.
| dmux wrote:
| I think Deno may be a good choice in this use case:
| single binary that has a good standard-lib makes it seem
| like a good candidate for simple shell and pipeline
| scripting.
| dehrmann wrote:
| I prefer Python for this because more OSes ship with it, it
| has a fairly rich standard library, and SREs are more
| comfortable with it, so you're more likely to find others
| using it as glue.
| throw_m239339 wrote:
| > I prefer Python for this because more OSes ship with it,
| it has a fairly rich standard library,
|
| I mean nodejs =/= javascript, the problem is nodejs and
| some of the people who designed it who basically made it so
| that it would rely heavily on NPM, a commercial solution,
| thanks to the virtually non existing standard library. Now
| Microsoft virtually owns Nodejs since it owns NPM. Nodejs
| creator himself regretted this decision, publicly.
| Shacklz wrote:
| For what I called "plumbing"-scripts above, I rarely ever
| need any dependencies via npm. What nodeJS delivers out
| of the box suffices completely.
|
| Sure, doing things like http directly with node is a bit
| cumbersome and having a library that does some
| abstraction comes in handy if you're working on something
| that goes way beyond what you'd ever consider doing in
| bash. But especially for that very purpose of things
| where I used nodeJS instead of using bash, nodeJS always
| had everything I needed out-of-the-box, almost never
| required any dependencies.
|
| That being said, if you re-use code, plan to share
| scripts etc. with other people and all these things -
| using something like typescript (and possibly some
| testing-libraries) comes in very handy for
| maintainability, and there you can't get around npm. But
| for one-off-"plubming-scripts", nodeJS had all I ever
| needed.
| deergomoo wrote:
| This will probably not be particularly popular with the HN
| crowd, but PHP is also a surprisingly capable general
| purpose scripting language.
|
| It's pre-installed on a lot of systems, backwards
| compatibility is good, and despite the inconsistencies, the
| standard lib can do a lot of stuff.
| tored wrote:
| Agree, but it would be nice to have access to the entire
| composer community without setting up a composer
| environment, so not to toot my own horn, but I wrote
| proof of concept script to download PHP composer
| dependencies by parsing comments from the PHP script
| itself.
|
| https://gist.github.com/tored/b500eb7c10fbabbe2043126e51c
| af2...
|
| It would be nice to have something like this more
| integrated within the shell or maybe composer itself.
| _ZeD_ wrote:
| have a look at perl :)
| goatlover wrote:
| Javascript all the things.
| artificial wrote:
| When the only tool you have is a hammer :)
| bryanrasmussen wrote:
| You look like Thor!
| Mauricebranagh wrote:
| More like Jeremy Clarkson hitting thing with a hammer
| instead of the right tool for the job.
| kristopolous wrote:
| This is purely performative and has no other benefit than to
| bow to a set of aesthetics.
|
| It's merely doctrinal orthodoxy full of needless and
| ceremonious frivolities for the adherents to chin stroke in
| approval.
|
| It is not, in any way, a reasonable way to do things, to
| exercise engineering, write maintainable or reliable code or
| otherwise accomplish tasks.
|
| It is just spectacle.
| [deleted]
| throw_m239339 wrote:
| > Why is Javascript a << perfect choose >> for shell scripts?
|
| It isn't, that project looks like perl re-invented but worse,
| since it mixes Javascript via nodejs and shell script syntax.
|
| Shell scripts already support loops, functions, conditions,
| variables, ... it makes more sense to write a nodejs script
| that works as a regular shell process and include it in a shell
| script than the other way around...
| giorgioz wrote:
| JS is the most used language for web development both client
| and server side. It's easy to find fullstack repos with only JS
| code and a bunch of shell scripts to launch the tooling like
| webpack bundler and cypress e2e tester. By wrapping the tooling
| also in JS it's possible to take advantage of shared
| configuration written in JS and write reusable functions to
| share across scripts. Currently I have all my tooling scripts
| in bash (not yet in JS) and it's becoming hard to manage.
| Sharing functions and constant values across folders in shell
| bash script is very cumbersome.
| EvilEy3 wrote:
| > and server side.
|
| Cool story.
| abiro wrote:
| I thought that was weird too. I much prefer Python to JS for
| scripting.
| Mauricebranagh wrote:
| Or even Perl - I have only done some scripts in Python to use
| beautiful soup, or for scripts that may be used by less
| experienced developers.
| gitgud wrote:
| Would be good to have a function which outputs all stdout from a
| sub command in real time, rather than after it's finished.
|
| I generally use "spawn" from child_process instead of "exec"
| (which this tool uses), which can pipe output to the terminal as
| it happens. Great for creating build scripts.
| lobstrosity420 wrote:
| Yes, you could use node's streams to implement this. It would
| be really cool.
| exdsq wrote:
| Honestly I'd rather use Bash than JavaScript. I've been using Go
| for scripting recently which has been nice.
| Ashanmaril wrote:
| Do you compile binary(s) or just execute a `go run` and rely on
| the fast build times?
| cafard wrote:
| I occasionally have used JScript on Windows, mostly because I
| don't much like VBScript, and PowerShell want you to sign its
| scripts.
| vips7L wrote:
| Just set your powershell execution policy to unrestricted?
| Thats the default on non-windows machines.
___________________________________________________________________
(page generated 2021-05-07 23:01 UTC)