[HN Gopher] Jo - a shell command to create JSON (2016)
___________________________________________________________________
Jo - a shell command to create JSON (2016)
Author : robfig
Score : 178 points
Date : 2022-02-05 18:27 UTC (4 hours ago)
(HTM) web link (jpmens.net)
(TXT) w3m dump (jpmens.net)
| pydry wrote:
| >Bam! Jo tries to be clever about types and knows null, booleans,
| strings and numbers.
|
| I'm very skeptical of this. If I put x=001979 in as a value I
| dont think I want you trying to guess if that's supposed to be an
| integer or a string.
|
| This sounds like the Norway Problem waiting to happen.
| mkdirp wrote:
| This kind of thing has its place and can be useful, but I think
| there should be options to enable/disable such magic.
| Personally I'd lean towards it being opt-in, but I think with
| the cli it's a lot harder to not have it opt-out. Everythint is
| a string in the cli, but not so much when it comes to json.
| That's why I think it makes sense, providing you can disable
| the magic.
| matja wrote:
| then use -s : jo -- -s opaque_id=001979
| {"opaque_id":"001979"}
| pydry wrote:
| jo opaque_id=$(command used in prod that returns digits +
| letters for three months and everything works just fine and
| then suddenly at 3am on a Sunday it returns 1979 and breaks
| everything)
|
| Avoid using this command in prod.
| matja wrote:
| jo -- -s a=123 -s b=00123 -s c="$(date)"
| {"a":"123","b":"00123","c":"Sat 5 Feb 21:29:08 GMT 2022"}
|
| What's the issue?
| pydry wrote:
| The problem is avoiding your surprise ruined Sunday night
| three months from now is predicated on you not forgetting
| to put the -s in.
|
| Implicit typing _sucks_ :
| https://www.destroyallsoftware.com/talks/wat
| matja wrote:
| Agreed, luckily there is a solution ;)
| alexander-litty wrote:
| It looks like you can specify the value type per property, for
| those cases where it matters.
| jrochkind1 wrote:
| It is reminiscent of a feature in YAML that has bitten people.
|
| But, clearly there is a use-case for producing json with
| integer, null, and boolean values.
| pydry wrote:
| Of course there is, but it should be done explicitly - i.e.
| treat as a string by default but use -n if it's a number
| rather than having a guess.
| [deleted]
| higgins wrote:
| next to "copy as cURL" in dev tools, this might make the regular
| rotation
| borgchick wrote:
| Awesome, thank you. Lovely counterpart to jq :)
| mirekrusin wrote:
| Could be simply called `json`.
| peakaboo wrote:
| Good luck in search results.
| tedmiston wrote:
| Direct link to repo:
|
| https://github.com/jpmens/jo
| seletskiy wrote:
| Some time ago I wrote zsh helper [1] that uses jo to quickly
| construct complex JSON requests, mainly for testing and quering
| services from console.
|
| Paired with httpie [2] aliases [3] it produces concise APL-like
| syntax: POST https://httpbin.org/post test:=j`a=b
| c=`e=3` l=`*1 2 3``
|
| Which translates to: http POST
| https://httpbin.org/post test:="$(jo a=b c="$(jo e=3)" l="$(jo -a
| 1 2 3)")"
|
| Or, in other words, sending POST with the following body:
| {"a":"b","c":{"e":3},"l":[1,2,3]}
|
| [1]:
| https://github.com/seletskiy/dotfiles/blob/78ac45c01bdf019ae...
| [2]: https://httpie.io/ [3]:
| https://github.com/seletskiy/dotfiles/blob/78ac45c01bdf019ae...
| jkbr wrote:
| HTTPie creator here. We've recently[0] added support for nested
| JSON[1] to the HTTPie request language, so you can now craft
| complex JSON request directly: $ http
| pie.dev/post test[a]=b test[c][e]:=3 test[l][]:=1
|
| [0] https://httpie.io/blog/httpie-3.0.0
|
| [1] https://httpie.io/docs/cli/nested-json
| matja wrote:
| I like it, very concise but retains all the features you need.
|
| `jq` can construct JSON "safely" from shell constructs, but is
| rather more verbose - e.g. with the same examples:
| $ jq -n --arg name Jane '{"name":$name}'
| $ jq -n \ --argjson time $(date +%s) \
| --arg dir $HOME \ '{"time":$time,"dir":$dir}'
| $ jq -n '$ARGS.positional' --args spring summer winter
| $ jq -n \ --arg name JP \ --argjson
| object "$( jq -n \ --arg fruit Orange
| \ --argjson point "$( jq -n \
| --argjson x 10 \ --argjson y 20 \
| '{"x":$x,"y":$y}' \ )" \ --argjson
| number 17 \
| '{"fruit":$fruit,"point":$point,"number":$number}' \
| )" \ --argjson sunday false \
| '{"name":$name,"object":$object,"sunday":$sunday}'
| mbrock wrote:
| There's also jshon which is a simple stack-based DSL for
| constructing JSON from shell scripts.
|
| http://kmkeen.com/jshon/
|
| It's written in C and is not actively developed. The latest
| commit, it seems, was a pull request from me back in 2018 that
| fixed a null-termination issue that led to memory corruption.
|
| Because I couldn't rely on jshon being correct, I rewrote it in
| Haskell here:
|
| https://github.com/dapphub/dapptools/tree/master/src/jays
|
| This is also not developed actively but it's a single simple ~200
| line Haskell program.
| lordleft wrote:
| The more I work with JSON, the more I crave some kind of
| dedicated json editor to easily visualize and manipulate json
| objects, and to serialize / deserialize strings. This is
| especially the case with truly massive JSON objects with multiple
| layers of nesting. Anyway, cool tool that makes one part of the
| process of whipping up JSON a little less painful
| fiddlerwoaroof wrote:
| I wrote something like this for emacs: a couple functions
| "fwoar/dive" and "fwoar/return" that let you navigate JSON
| documents in a two-pane sort of paradigm.
|
| https://youtu.be/qbRNmk-malw
| 8organicbits wrote:
| I built a JSON editor for Android a while back as part of a
| tool for kicking off AWS Lambda functions. I was planning on
| pulling the JSON editor to it's own reusable package, but lost
| momentum. I imagine it could be useful in many sorts of apps
| that use JSON.
|
| https://play.google.com/store/apps/details?id=com.alexsci.an...
|
| https://github.com/ralexander-phi/android-aws-lambda-runner
| qbasic_forever wrote:
| Add a json LSP to your editor, or use VS code which includes it
| natively: https://www.npmjs.com/package/vscode-json-
| languageserver Configure a json schema for the document you're
| editing and suddenly you get suggestions, validation, etc. as
| you type. It's pretty magical.
|
| There's one for yaml too that works well in my experience:
| https://github.com/redhat-developer/yaml-language-server
| videogreg93 wrote:
| Looks neat, but the documentation was hard to follow. A lot of
| typos, and some things didn't make sense. For example, I think 2
| paragraphs were swapped because the example code below the
| paragraph taking about using square brackets has no square
| brackets (but nested objects) while the paragraph right after
| talks about nested objects but it's example doesn't show it (it
| does however seemingly demonstrate the square bracket feature
| mentioned previously)
| EdSchouten wrote:
| I remember that when I worked at Google about a decade ago, there
| was this common saying:
|
| "If the first version of your shell script is more than five
| lines long, you should have written it in Python."
|
| I think there's a lot of truth in that. None of the examples
| presented in the article look better than had they been written
| in some existing scripting/programming language. In fact, had
| they been written in Python or Javascript, it would have been far
| more obvious what the resulting output would have been,
| considering that those languages already use {} for objects and
| [] for lists.
|
| For example, take this example: jo -p name=JP
| object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17)
| sunday=false
|
| In Python you would write it like this:
| json.dumps({"name": "JP", "object": {"fruit": "Orange", "point":
| {"x": 10, "y": 20}, "number": 17}, "sunday": False})
|
| Only a bit more code, but at least it won't suffer from the
| Norway problem. Even though Python isn't the fastest language out
| there, it's likely still faster than the shell command above.
| There is no need to construct a fork bomb just to generate some
| JSON.
| naniwaduni wrote:
| For programs this trivial, startup time so dominates runtime,
| and Python's startup time is so incredibly awful, that you can
| often fork 10-20 low-overhead processes before Python even
| starts executing user code.
| [deleted]
| zimpenfish wrote:
| > Even though Python isn't the fastest language out there, it's
| likely still faster than the shell command above.
|
| Taking these two command lines: jo -p name=JP
| object=$(jo fruit=Orange point=$(jo x=10 y=20) number=17)
| sunday=false >/dev/null python -c 'import
| json;print(json.dumps({"name": "JP", "object": {"fruit":
| "Orange", "point": {"x": 10, "y": 20}, "number": 17}, "sunday":
| False}))' >/dev/null
|
| For jo (x86_64, Rosetta2), python2 (x86_64, Rosetta2), jo
| (arm64), and python3 (arm64), running 1000 iterations, with
| `tai64n` doing the timing. 2022-02-05
| 21:25:38.357228500 start-jo-x86 2022-02-05
| 21:25:45.319337500 stop-jo 2022-02-05
| 21:25:45.319338500 start-python2-x86 2022-02-05
| 21:26:18.876235500 stop-python2-x86 2022-02-05
| 21:26:18.876235500 start-jo-arm 2022-02-05
| 21:26:22.316063500 stop-jo-arm 2022-02-05
| 21:26:22.316064500 start-python3-arm 2022-02-05
| 21:26:40.379063500 stop-python3-arm
|
| I make it: 7s for jo-x86, 33.5s for python2-x86, 3.5s for jo-
| arm, 18s for python3-arm.
|
| Test script is at https://pastebin.com/4tTVrDia
| nousermane wrote:
| python3 is (relatively) slow to startup, and this is
| something that got significantly worse with 2->3 migration:
| $ time python3 -c '' real 0m0.029s $ time
| python2 -c '' real 0m0.010s $ time bash
| -c '' real 0m0.001s
|
| Which means - you probably don't want to have a python script
| on a busy webserver, being called from classic cgi-bin (do
| people still use those?), or run it as -exec argument to a
| "find" iterating over many thousands files. Maybe a couple
| more of such examples. For most use-cases though, that's
| still fast enough.
| EdSchouten wrote:
| Now put a thousand of those JSON objects in a list, invoking
| jo for every element.
| KronisLV wrote:
| I actually did that for a more realistic comparison.
|
| Example for jo: docker run --rm -it debian
| bash apt update && apt install -y jo nano nano
| bash-loop.sh && chmod +x bash-loop.sh
| #!/bin/bash for ((i=0;i<1000;i++)); do
| jo -p name=JP object=$(jo fruit=Orange point=$(jo x=10
| y=20) number=17) sunday=false done time
| ./bash-loop.sh >/dev/null
|
| Example for Python 3: docker run --rm -it
| debian bash apt update && apt install -y python3 nano
| nano python-loop.py import json for i in
| range(1000): print(json.dumps({"name": "JP",
| "object": {"fruit": "Orange", "point": {"x": 10, "y": 20},
| "number": 17}, "sunday": False})) time python3
| python-loop.py >/dev/null
|
| Versions: Debian GNU/Linux 11 (bullseye)
| jo 1.3 Python 3.9.2
|
| Results for jo: real 0m2.230s user
| 0m1.106s sys 0m1.076s
|
| Results for Python 3: real 0m0.027s
| user 0m0.021s sys 0m0.005s
|
| So it seems like you're probably right about how individual
| invocations scale for larger amounts of invocations in non-
| trivial cases!
|
| Note: jo seems to pretty print because of the "-p"
| parameter, which is not the case with Python, might not be
| a 1:1 comparison in this case. Would be better to remove
| it. Though when i did that, the performance improvement was
| maybe 1%, not significant.
|
| Admittedly, it would be nice to test with actually random
| data to make sure that nothing gets optimized away, such as
| just replacing one of the numbers in JSON with a random
| value, say, the UNIX timestamp. But then you'd have to
| prepare all of the data beforehand (to avoid differences
| due to using Python to get those timestamps, or one of the
| GNU tools), or time the execution separately however you
| wish.
|
| Edit to explain my rationale: Why bother doing this?
| Because i disagree with the sibling comment:
|
| > The claim was that the Python snippet would be quicker
| than the jo snippet.
|
| In my eyes that's almost meaningless, since in practice
| when you'll actually care about the runtimes will be when
| working with larger amounts of data, or alternatively
| really large files. Therefore this should be tested, not
| just the startup times, which become irrelevant in most
| real world programs, except for cases when you'd make a
| separate invocation per request, which you sometimes
| shouldn't do.
|
| Edit #2: here's a lazy edit that uses the UNIX time and
| makes the data more dynamic, ignoring the overhead to
| retrieve this value, to get a ballpark figure.
|
| Use time value for jo: jo -p name=JP
| object=$(jo fruit=Orange point=$(jo x=10 y=20)
| number=$(date +%s)) sunday=false
|
| Use time value for Python 3: import time
| ... print(json.dumps({"name": "JP", "object":
| {"fruit": "Orange", "point": {"x": 10, "y": 20}, "number":
| int(time.time())}, "sunday": False}))
|
| Results for jo: real 0m2.794s user
| 0m1.422s sys 0m1.313s
|
| Results for Python 3: real 0m0.027s
| user 0m0.020s sys 0m0.006s
|
| Seems like nothing changed much.
|
| Edit #3: probably should have started with a test to verify
| whether the initially observed performance differences
| (Python being slower due to startup time) were also
| present.
|
| Single iteration results for jo: real
| 0m0.003s user 0m0.000s sys 0m0.002s
|
| Single iteration results for Python 3: real
| 0m0.022s user 0m0.017s sys 0m0.004s
|
| Seems to also more or less match those results.
| [deleted]
| zimpenfish wrote:
| That wasn't the claim made in the original post though, was
| it? The claim was that the Python snippet would be quicker
| than the jo snippet.
|
| "Even though Python isn't the fastest language out there,
| it's likely still faster than the shell command above."
|
| Which is most definitely is not - it's 5x slower.
|
| (Probably not a huge issue in the real world if you're
| writing a shell script, mind, given that bash itself isn't
| a performance demon. But claims have to be tested.)
| EdSchouten wrote:
| That's because you're making a false assumption about the
| environment prior to executing the statement.
|
| If you are in a shell session and have to choose between
| executing python -c or calling jo, the latter is faster
| as you've demonstrated. But that's not a realistic
| assumption.
|
| Statements like these are almost certainly part of some
| combined work. The data you're feeding to jo comes from
| somewhere. Its output is written somewhere.
|
| You can't convince me that if you're already inside some
| Python script, that invoking json.dumps() is slower than
| calling jo from within a shell script.
|
| At no point did I claim that launching Python AND running
| that json.dumps() is faster than running that shell
| command. I only stated that the json.dumps() is.
| mmgutz wrote:
| If you're going to use string literal, then
| echo '{"name": "JP", "object": {"fruit": "Orange", "point":
| {"x": 10, "y": 20}, "number": 17}, "sunday": false}'
|
| is fine as a script
| pkulak wrote:
| Sure, but using Python gives you 100 more problems. Would you
| like to set pip and your package manager on a fight to the
| death? Or is today the day you learn all about venv? Might as
| well use Docker. Oh nice, my 600MB script is ready!
| detaro wrote:
| That's not really an issue for small scripts, no.
| motoboi wrote:
| Hard to write oneliners in python, though.
|
| And this lets you write json without (or with less) braces,
| commas and quotes. That alone is already a big win.
| pwdisswordfish9 wrote:
| Does it suffer from the Norway problem?
| 0xbadcafebee wrote:
| I think you're thinking of YAML; JSON doesn't interpret "no" as
| a boolean. Scroll down here to see the JSON grammar:
| https://www.crockford.com/mckeeman.html
|
| I actually think the "Norway problem" is a PEBKAC from users
| not learning the data format. But this tool may confuse some
| people or applications who don't know what a boolean, integer,
| float or string are, and try to mix types when the program
| reading them wasn't designed to. Probably the issue will come
| up whenever people mix different kinds of versions ("1", "1.1",
| "1.1.1" should be parsed as an int, float, and string,
| respectively)
| motoboi wrote:
| jo a=12
|
| Should "a" be number or a string in the resulting JSON?
|
| And if it's number, how can I tell it to output a string?
| montroser wrote:
| This is cool.
|
| In the spirit of "do one thing well", I'd so rather use this to
| construct JSON payloads to curl requests than the curl project's
| own "json part" proposal[1] under consideration.
|
| [1]: https://github.com/curl/curl/wiki/JSON#--jp-part
| matja wrote:
| Agree, I was surprised that the cURL feature was considered as
| it seems to go against the "Do One Thing" and composability
| points of the UNIX philosophy.
| oblio wrote:
| curl? Do one thing? In the same sentence?
|
| curl looks like it has hundreds of flags.
|
| https://danluu.com/cli-complexity/
| nerdponx wrote:
| Curl does like 100 "things" already by that standard. The
| Unix philosophy doesn't have to be reductionist.
|
| Curl _does_ do one thing: make network requests. This feature
| is making it easier to make network requests, i.e. it makes
| it better at doing the one thing that it does.
| nikolay wrote:
| I've been using it for years and it's a great companion of jq and
| jp (JMESPath).
| ysk73 wrote:
| And I just use Nushell. You have built-ins to create (and parse)
| not only json but also url, xml and more...
| https://github.com/skelly37/Reject-POSUCKS-embrace-Nushell#b...
| kazinator wrote:
| Five minute job: $ ./jo foo=1 bar=2 obj=$(./jo -a
| 1 2 3 "</script" '"')
| {"foo":1,"obj":[1,2,3,"<\/script","\""],"bar":2} $
| ./jo foo='abc > def > ghi'
| {"foo":"abc\ndef\nghi"} $ cat jo
| #!/usr/local/bin/txr --lisp (define-option-struct
| jo-opts nil (a array :bool "Produce
| array instead of object") (nil help :bool
| "Print this help")) (defvarl jo-name *load-path*)
| (defun json-val (str) (match-case str ("true"
| t) ("false" nil) ("null" 'null)
| (`{@nil` (get-json str)) (`[@nil` (get-json str))
| (@else (iflet ((num (tofloat else))) num
| else)))) (let ((o (new jo-opts))) o.(getopts
| *args*) (when o.help (put-line "Usage:\n")
| (put-line ` @{jo-name} [options] arg*`) o.(opthelp)
| (exit 0)) (if o.array (let ((items
| [mapcar json-val o.out-args])) (put-jsonl (vec-list
| items))) (let ((pairs [mapcar (lambda (:match)
| ((`@this=@that`) (list (json-val this) (json-val that)))
| ((@else) (error "~a: arguments must be name=obj pairs" jo-name)))
| o.out-args])) (put-jsonl ^#H(() ,*pairs)))))
| ilyash wrote:
| Depending on your exact needs, you might also want to try Next
| Generation Shell. It's a fully featured programming language for
| the DevOps-y stuff with convenient "print JSON" switch so "ngs
| -pj YOUR_EXPR" evaluates the expression, serializes the result as
| JSON and prints it.
|
| Disclosure: I'm the author
|
| Project link: https://github.com/ngs-lang/ngs
|
| Enjoy!
___________________________________________________________________
(page generated 2022-02-05 23:00 UTC)