[HN Gopher] JC - JSONifies the output of many CLI tools
___________________________________________________________________
JC - JSONifies the output of many CLI tools
Author : pgl
Score : 361 points
Date : 2022-11-03 07:55 UTC (15 hours ago)
(HTM) web link (kellyjonbrazil.github.io)
(TXT) w3m dump (kellyjonbrazil.github.io)
| berkes wrote:
| A slightly related pet-peeve: I don't like it when "random"
| commands "squat" the two-letter domain, or worse, the one-letter
| domain: t, jq, jc etc.
|
| In my perfect world (which, obviously doesn't exist), commands
| from tools "in the wild" are at least three letters long. With
| historical exceptions for gnutools: preferably they'd take the
| three-letter space, but two-letters (cd, ls, rm etc) is fine.
|
| Two letter space outside of gnutools, is then reserved for my
| aliases. If jsonquery is too long to type, AND I lack
| autocomplete, then an alias is easy and fast to make. alias
| jq=jsonquery.
|
| In the case of this tool, it will conflict with a specialised
| alias I have: `alias jc=javac -Werror`. Easy to solve by me with
| another alias, but a practical example of why I dislike tools
| "squatting" the two letter namespace.
| artemisart wrote:
| I don't believe it's really an issue in practice, your `jc`
| alias will just take the priority and you can easily add one
| for jsonquery -> `path/to/bin/jc`. I think good short command
| names can help adoption (like for ripgrep, fd) but it's true
| that we should have a race to squat all the 2 letter names.
| Sharparam wrote:
| Instead of typing the whole path, you can also use `command`
| or a backslash to bypass aliases
|
| E.g.: alias jsonquery='command jq'
| ycombobreaker wrote:
| I perceive an element of hubris when a tool claims a two-
| character name. That is a very small namespace, so the tool
| is effectively staking a claim on mental or emotional real-
| estate.
| [deleted]
| WorldMaker wrote:
| This seems a general symptom of unix/POSIX command naming where
| commands have "always" been short names, and commands are
| expected to be short names.
|
| It's something I appreciate about the powershell naming
| conventions. A lot of people mock the verbosity of the names of
| powershell commands and commandlets which require the "proper"
| name to be Verb-Noun qebab case monstrosities, but this was
| chosen for exactly the reasons of your pet peeve: short command
| names should be user aliases for work in a shell, and longer
| command names are great for avoiding namespace clashes in
| scripts and between users. The verbs and nouns create large
| (discoverable) namespaces.
|
| For instance, this tool might be powershell named ConvertTo-
| ParsedJson. (ConvertTo-Json is an out of the box command that
| converts any powershell object to JSON.) It might suggest the
| user alias it by adding `Set-Alias -Name jc -Value ConvertTo-
| ParsedJson` but generally commands in powershell only offer
| such aliases as _suggestions_ rather than defaults. (Though
| there are a lot of out of the box aliases for common powershell
| commands.)
|
| It makes sense to me that powershell encourages long names
| first and allows and encourages users to have far more control
| over short aliases.
| anecdotal1 wrote:
| Can you trust it? Cli tool output is not exactly stable. I
| thought that's why libxo exists?
|
| https://github.com/Juniper/libxo
| gorkish wrote:
| Another vote for something like libXo as the better solution.
| This thing is just passing the parsing problem of interacting
| with an unstable API to someone else and hoping they maintain
| theirs better than you would maintain yours.
| exabrial wrote:
| Have a parseable output is great. What would be more incredible
| is to have a parseable output with a schema definition and/or
| formal grammar of some sort.
| vincnetas wrote:
| Schemas are provided on linked page near definitions of
| parsers.
| kbrazil wrote:
| You can also run `jc -h --dig` to get the parser details that
| include the schema.
|
| Having true JSON Schema[0] is being considered, but on the
| back-burner due to the sheer number of parsers to build
| schemas for. Also, it is more difficult to accurately define
| the schema for a small subset of parsers since their command
| output are so variable.
|
| [0] https://json-schema.org/
| mg wrote:
| dig example.com | jc --dig
|
| Seems a bit redundant. Maybe it should be the other way round?
| jc dig example.com
|
| Similar to how you do time dig example.com
| chii wrote:
| `time` needs to be the one to exec the command because they
| need to know when the command they are timing starts.
| Therefore, `time` cannot use pipes, as by then, the command
| being timed would've already started!
|
| `jc` doesn't need to know anything about the command producing
| the output - just the format of the output. So using a pipe and
| stdin makes a lot of sense.
| DougBTX wrote:
| Yes, that works, there are examples on the page like:
| $ jc dig example.com | jq -r '.[].answer[].data'
| 93.184.216.34
|
| It uses the first argument to infer the command output type.
| berkes wrote:
| Is "redundancy" that you have to provide --dig?
|
| I can imagine `jc` having some detection built in, from which
| it determines the command/content it's being parsed. Doesn't
| seem to have it, yet, and I'm generally no big fan of "magic"
| like this, but it would remove the redundancy.
|
| Having it as a pipe, allows for much more, though.
| some_expensive_command > out.log jc --expensive-cmd <
| out.log
|
| Or hourly_dig.sh > example_com_records_$(date
| +%F+%s) cat example_com_records_* | jc --dig
| kbrazil wrote:
| Hi there - author of `jc` here. I originally intended to have
| auto-detection but put that on the backburner to focus on
| creating parsers, especially after introducing the magic
| syntax.
|
| I did implement auto-detection for `/proc` file parsers so
| you can just do: $ cat /proc/foo | jc
| --proc
|
| or $ jc /proc/foo
|
| But you can specify each procfile parser directly if you want
| to as well.
| enriquto wrote:
| In the opposite direction, if you want to recover a human-
| editable text stream from json data, you run gron.
| asicsp wrote:
| Link: https://github.com/tomnomnom/gron
|
| Discussion: https://news.ycombinator.com/item?id=25006277 _366
| points | Nov 6, 2020 | 91 comments_
| zikduruqe wrote:
| Came looking for gron recommendations.
| codedokode wrote:
| I think that JSON is a bad choice here.
|
| It is obvious that CLI commands should produce machine-readable
| output because they are often used in scripts, and accept
| machine-readable input as well. Using arbitrary text output was a
| mistake because it is difficult to parse, especially when spaces
| and non-ASCII characters are present.
|
| A good choice would be a format that is easily parsed by programs
| but still readable by the user. JSON is a bad choice here because
| it is hard to read.
|
| In my opinion, something formatted with pipes, quotes and spaces
| would be better: eth0: ip:
| 127.15.34.23 flags: BROADCAST|UNICAST mtu:
| 1500 name: """"Gigabit" by Network Interfaces Inc."""
|
| Note that the format I have proposed here is machine-readable,
| somewhat human-readable and somewhat parseable by line-oriented
| tools like grep. Therefore there might be no need for switches to
| choose output format. It is also relatively easy to produce
| without any libraries.
|
| Regarding idea to output data in /proc or /sys in JSON format, I
| think this is wrong as well. This would mean that reading data
| about multiple processes would require lot of formatting and
| parsing JSON. Instead or parsing /proc and /sys directly,
| applications should use libraries distributed with kernel, and
| reading the data directly should be discouraged. Because
| currently /proc and /sys are just a kind of undocumented API.
|
| Also, I wanted to note that I dislike jq utility. Instead of
| using JSONPath it uses some proprietary query format that I
| constantly fail to remember.
| onion2k wrote:
| If you use something other than JSON you'd have to wait until
| every app you want to use chooses to update to support your
| preferred format. That might take a while. Wouldn't it be
| better to use JSON for the output as that's an acceptable input
| to lots and lots of applications already, and if you want to
| read the output just pass it to an app that converts from JSON
| to "something formatted with pipes, quotes and spaces".
| codedokode wrote:
| On the other hand, if machine-readable format is adopted, it
| will be used for many years or even decades. So instead of
| making a quick hack that everybody will regret later it might
| be better to spend some time comparing different options.
| feanaro wrote:
| Some alternative ideas for making JSON more readable:
|
| - Pipe into gron (https://github.com/tomnomnom/gron) to get a
| `foo.bar.baz = val` kind of syntax.
|
| - Pipe into visidata (https://www.visidata.org/) to get a
| spreadsheet-like editable view.
| kbrazil wrote:
| Hi there - `jc` author here. `jc` can also output in YAML
| format with the `-y` flag. It is fairly trivial to add other
| options in the future since `jc` just turns the text into
| objects which can be serialized to many different formats.
|
| For example: % jc -y date ---
| year: 2022 month: Nov month_num: 11
| day: 3 weekday: Thu weekday_num: 4
| hour: 9 hour_24: 9 minute: 0 second: 22
| period: AM timezone: PDT utc_offset:
| day_of_year: 307 week_of_year: 44 iso:
| '2022-11-03T09:00:22' epoch: 1667491222
| epoch_utc: timezone_aware: false
| codedokode wrote:
| Great, by I would prefer to avoid YAML because it is very
| complicated and difficult to parse.
| thesuperbigfrog wrote:
| >> something formatted with pipes, quotes and spaces would be
| better
|
| How well would this format handle deeply nested structures? It
| seems like it would require a lot of space characters compared
| to nesting open and close characters: {} or () or []
|
| How would escaping pipes, quotes, and spaces work to represent
| those character literals?
|
| There are already numerous structured text formats: JSON, XML,
| S-expressions, YAML, TOML, EDN, and many more. Wouldn't this be
| yet another format? (https://xkcd.com/927/)
| djedr wrote:
| Dare I suggest Jevko[0] as yet another alternative?
| eth0 [ ip [127.15.34.23] flags
| [[BROADCAST][UNICAST]] mtu [1500] name
| ["Gigabit" by Network Interfaces Inc.] ]
|
| This is one of the things it was designed with in mind.
|
| It's even simpler and more flexible than S-expressions.
|
| Handles deeply nested structures perfectly well. Has only 3
| characters to escape (brackets and the escape character).
|
| (I am the author)
|
| [0] https://jevko.org/
| artemisart wrote:
| How do you differentiate types with jevko (numbers,
| strings, boolean)? Your examples on jevko.org appear lossy
| as they encode in the same way things that are different in
| JSON and I don't know how you would then differentiate
| between true and "true", 27 and "27", etc.
| djedr wrote:
| A plain Jevko parser simply turns your unicode sequence
| into a tree which has its fragments as leaves/labels.
|
| No data types on that level, much like in XML.
|
| Now above that level there is several ways to
| differentiate between them.
|
| The simplest pragmatic way is a kind of type inference:
| if a text parses as a number, it's a number, if it's
| "true" or "false", it's a boolean. Otherwise it's a
| string. If you know the implicit schema of your data then
| this will be sufficient to get the job done.
|
| Otherwise you employ a separate schema -- JC in
| particular has per-parser schemas anyway, so that's
| covered in this case. If it wouldn't, you'd need to write
| a schema yourself.
|
| Or you do "syntax-driven" data types, similar to JSON,
| e.g. strings start w/ "'".
|
| Here is a shitty demo:
| https://jevko.github.io/interjevko.bundle.html
|
| It shows schema inference from JSON and the schemaless
| (syntax-driven) flavor.
|
| Jevko itself is stable and formally specified:
| https://github.com/jevko/specifications/blob/master/spec-
| sta...
|
| It's very easy to write a parser in any language (I've
| written one in several) and from there start using it.
|
| However, I am still very much working on specifications
| for formats above Jevko. I have some recent
| implementations of the simplest possible format which
| converts Jevko to arrays/objects/strings:
|
| * https://github.com/jevko/easyjevko.lua
|
| * https://github.com/jevko/easyjevko.js
|
| The schema-driven format that was used in the demo is
| implemented here:
|
| * https://github.com/jevko/interjevko.js
|
| * https://github.com/jevko/jevkoschema.js
| bonzini wrote:
| > In my opinion, something formatted with pipes, quotes and
| spaces would be better:
|
| Just pipe it into a JSON-to-YAML script like this:
| #! /usr/bin/python3 from ruamel import yaml
| import json, sys, io
| print(yaml.dump(json.load(sys.stdin)))
| ape4 wrote:
| JSON is the lazy choice. I particularly dislike quoting keys
| (variable names). Relaxed JSON, for one, allows unquoted keys
| http://www.relaxedjson.org But that's just a small step - I am
| sure we (the community) could do better.
| rjzzleep wrote:
| I also dislike jq, but this is a bit of a non issue IMHO. You
| could in theory add any kind of output transformer in theory.
| The codebase doesn't seem to be optimized for that yet, but it
| should be trivial to add.
| psychoslave wrote:
| Following the project documentation, you easily come to:
| jc dig example.com | jq [ {
| "id": 30081, "opcode": "QUERY", "status":
| "NOERROR", "flags": [ "qr",
| "rd", "ra" ], "query_num": 1,
| "answer_num": 1, "authority_num": 0,
| "additional_num": 1, "opt_pseudosection": {
| "edns": { "version": 0, "flags":
| [], "udp": 4096 } },
| "question": { "name": "example.com.",
| "class": "IN", "type": "A" },
| "answer": [ { "name": "example.com.",
| "class": "IN", "type": "A", "ttl":
| 56151, "data": "93.184.216.34" }
| ], "query_time": 0, "server":
| "192.168.1.254#53(192.168.1.254)", "when": "Thu Nov
| 03 14:06:40 CET 2022", "rcvd": 56,
| "when_epoch": 1667480800, "when_epoch_utc": null
| } ]
|
| Rather readable to my mind. And you can rather easily transform
| it to your preferred human readable output format I guess.
| codedokode wrote:
| For me there are too many quotes and brackets. My proposed
| format can also be converted to JSON if necessary.
| WorldMaker wrote:
| > A good choice would be a format that is easily parsed by
| programs but still readable by the user.
|
| I think the powershell approach is a good one here too:
| powershell commands output binary streams of objects rather
| than text and it is powershell itself that has several standard
| ways of human readable outputs, most of which are automatic
| (but easily tweaked with an extra pipe or two). Standard human
| readable forms are nice, _and_ even standardized there 's no
| need to rely on parsing them back out into objects because they
| are already passed as objects so they can focus a bit more on
| "pretty" over "parse-able" (such as including human useful
| things like ellisions `...` on long columns).
| zokier wrote:
| The thing to understand with PowerShell is that the way it
| pipelines objects is enabled by the fact that it is all
| happening in-process within one .net runtime. It is
| significantly more difficult to achieve anything similar with
| several independent processes being piped together
| WorldMaker wrote:
| Well, yes, powershell takes some shortcuts and has the
| advantage that .NET has a strong object system.
|
| If you were to build it from scratch with the idea of
| "shared nothing" applications similar to the unix model
| with text files, it's not that much more difficult with
| just about any sort of object or message broker. You could
| easily imagine a world with a dbus based "REPL"/shell, for
| instance. Or a different approach easily imaginable if you
| still want to focus on unix-style streams/files between
| processes would be something like BSON streams (thought it
| would still have some serialization/deserialization
| overhead).
| DonHopkins wrote:
| Not binary streams of serialized objects, but pointers to
| live COM objects in one process's memory. No serialization /
| deserialization, binary or text, or piping between processes.
| That's much more efficient than serializing and deserializing
| every step in different processes.
| [deleted]
| khiqxj wrote:
| pgl wrote:
| Command line output as JSON. Very handy and has parsers for tons
| of common utilities, written in Python.
|
| Blog post with examples here:
| https://blog.kellybrazil.com/2020/08/30/parsing-command-outp...
| ducktective wrote:
| Then pipe it into jq [1] to query parameters or build up a
| formatted string.
|
| Or pipe it into rq [2] to convert the format to yaml, toml etc.
|
| [1]: https://stedolan.github.io/jq/tutorial/ [2]:
| https://github.com/dflemstr/rq#format-support-status
| asicsp wrote:
| There's also https://github.com/TomWright/dasel (supports JSON,
| TOML, YAML, XML and CSV)
| samuell wrote:
| I would wish for jq to be a really generic tool for working
| with structured data on the commandline, but I have a really
| hard time figuring out how to do e.g. conditional-based editing
| etc. Can't get my head around that, and don't find any info
| about it on the net. Seems even something that just supports
| SQL (upon JSON) would be better in this regard.
| kbrazil wrote:
| Hi there - I'm the author of `jc`. I also created `jello`[0],
| which works just like `jq` but uses python syntax. I find
| `jq` is great for many things but sometimes more complex
| operations are easier for me to grok in python.
|
| [0] https://github.com/kellyjonbrazil/jello
| ddulaney wrote:
| What do you mean by conditional-based editing? I've found its
| language to be pretty concise and readable, especially with
| the // operator.
| psadri wrote:
| Is anyone aware of something similar but for the args? A database
| that maps a command to a schema for all it's possible cli
| arguments?
| uvesten wrote:
| As someone who uses `jq` almost daily, this looks like a great
| tool I didn't know I needed. Thanks for the tip!
| naikrovek wrote:
| This seems very fragile, to me, without support from the
| application whose output is converted to JSON.
|
| minor updates to command-line tools can and do subtly alter the
| textual output of the tool, and the outputs of these tools are
| not standardized.
|
| This is a step towards "objects passing messages" as originally
| conceived by Alan Kay, if my incomplete understanding of what
| he's said is correct, and that's a good thing, I think. Objects
| passing messages around is a very solid model for computing, to
| me. Note that I am stupid and don't understand much, if I'm
| honest.
| synergy20 wrote:
| to make this great tool truly universal, it has to be written in
| c instead of python these days, then provide
| python|javascript|etc bindings if possible.
|
| I'd like to use it on embedded systems, where python is too large
| to fit. this tool can be widely deployed just like awk|sed|etc
| but it has to be in C for that.
| montroser wrote:
| I would welcome POSIX shell or at least bash as an alternative
| to a compiled C/Rust/Go binary.
|
| But yeah, for these types of utilities, relying on an external
| language runtime like Python/Node is pretty rough.
| bitwize wrote:
| Replace C with Rust.
| synergy20 wrote:
| on embedded rust is still much larger than c because rust
| links its stdlib statically.
| bitwize wrote:
| There is such a thing as no-stdlib Rust deployments.
| synergy20 wrote:
| which is for MCUs only and is like a bare metal language
| without any libraries, its use is very limited.
|
| all I need is that rust's stdlib can be linked
| dynamically just like c/c++/java/whatever, if that
| happens I'm ready to switch to it.
| makapuf wrote:
| Agreed but I would change it with "any compiled language that
| has no external runtime" and common shared libs dependencies. I
| don't care that a utility is written in Go, Zig, C++ or Pascal
| ;)
| kbrazil wrote:
| `jc` is available today as a compiled binary with all
| dependencies self-contained. The binaries and OS package
| installers are available under the GitHub Releases:
|
| https://github.com/kellyjonbrazil/jc/releases
|
| This is still python under the hood and not as small of a
| binary as I would like, but it does work.
| maxbond wrote:
| Seems to me like anywhere Python is too bulky to work, a layer
| serializing and deserializing JSON at every pipeline step is
| likely to exhaust your memory too.
| ElectricalUnion wrote:
| But python is also universal? You can just bundle jc inside
| https://github.com/jart/cosmopolitan/tree/master/third_party...
| and it's as universal as it gets.
| HyperSane wrote:
| The greatest thing about powershell is that all commands returns
| structured output like this.
| WorldMaker wrote:
| Also, this tool is useful for using old unix commands _in_
| powershell: `jc ... | ConvertFrom-Json` easily gives you
| powershell objects to work with.
| HyperSane wrote:
| That is a great idea!
| friendzis wrote:
| Sounds a lot like Powershell, which returns rich objects instead
| of a text blob
| WorldMaker wrote:
| Yup, powershell is mentioned in the references. Also, this tool
| can be used to bridge older unix commands into powershell
| objects (ConvertFrom-Json).
| sindoc wrote:
| OMG, I was looking for exactly something like this. I'd like to
| create immersion between Logseq and CLI. This can very useful.
| hinkley wrote:
| I discovered process.send() in Node a couple years ago and it
| made the decision to fork a child process a lot easier. No need
| to sanitize command line output when you can do direct IPC over a
| connection that uses JSON under the hood.
|
| The itch I can't seem to scratch is how to run tasks in parallel
| and have logs that are legible to coworkers. We do JSON formatted
| logs in production and I'm wondering if something like this would
| help solve that set of problems.
| visarga wrote:
| love it, json could be the best way to pipe data between
| processes because it is text, but structured
| drunkpotato wrote:
| So is xml. Json is relatively unstructured compared to other
| formats.
|
| I think json has several advantages though. It's a relatively
| lightweight and widely known serialization standard, rich
| enough for most cases and extensible in others, and it has easy
| to use parsers in all major programming languages.
|
| Also jsonlines is a simple addition that make it easy for json
| to play well with non-json aware older Unix tools.
|
| It has a few shortcomings but I think its advantages outweigh
| them, and it's become a pretty widely used standard in a short
| time.
| tgv wrote:
| I just learned about gron (elsewhere in the comment section:
| https://news.ycombinator.com/item?id=33448471). That
| overcomes some of jsonlines problems.
| fimdomeio wrote:
| There's a clash of names between this jc and autojump
| (https://github.com/wting/autojump) jc (jump to child)
| menjaprunes wrote:
| I stumbled upon this few weeks ago to parse the output of nmcli
| and now it is in my toolset for scripting
|
| I love it
| pastage wrote:
| I Love using this in streams of data, but there is a lot to be
| said about the pit falls. Some of them; first you might want to
| filter with grep first and that leads to missing metadata, second
| error handling is a good thing but people tend to ignore errors
| and then not handle them. This is basically what
| filebeat/logstash from Elastic does, which is a beast at parsing
| (and impossible to use from command line).
|
| The power of plain text pipes is that you do not interpret them
| and that makes them fast, that is usefull because you handle both
| 100 bytes, 1MB and 1TB as input. You choose what you parse
| keeping it simple, fast and usually error free. This tool miss
| the, fast, simple and human readable part of debugging pipes.
| Which is fine!
| nixcraft wrote:
| Last year I looked for JSON output with the dig command on Linux
| but found the yaml option while reading the man page. It was
| handy. I always wondered why yaml/JSON output is not standard
| with Linux and Unix utilities for scripting needs. Anyway:
| dig +yaml google.com
| vidarh wrote:
| Because neither yaml or JSON existed when most of these tools
| were written, and then people got used to the current outputs,
| and so it's very varying whether or not anyone has felt the
| pain enough to decide it's worth adding the option for another
| output format.
| esskay wrote:
| Been using this a while to pull out a bunch of server stats for a
| monitoring dashboard and it's been great. I just wish there were
| more supported services. The total mess of different outputs and
| config types for Linux packages is extremely annoying to deal
| with.
| [deleted]
| deafpolygon wrote:
| Bringing a powershell feature to the Unix CLI.
| Kipters wrote:
| Since it's now cross platform, technically powershell brought
| powershell features to Unix first
| dmoura wrote:
| This is great!
|
| I am the author of SPyQL [1]. Combining JC with SPyQL you can
| easily query the json output and run python commands on top of it
| from the command-line :-) You can do aggregations and so forth in
| a much simpler and intuitive way than with jq.
|
| I just wrote a blogpost [2] that illustrates it. It is more
| focused on CSV, but the commands would be the same if you were
| working with JSON.
|
| [1] https://github.com/dcmoura/spyql [2]
| https://danielcmoura.com/blog/2022/spyql-cell-towers/
| sbt567 wrote:
| Wow, this looks super useful. Will definitely check this out.
| Thanks
| spinningslate wrote:
| Didn't know about this, the HN dividend pays out again!
|
| When wrestling with sed/awk in trying to parse results of a shell
| command, I've often thought that a shell-standard, structured
| outpout would be very handy. Powershell[0] has this, but it's a
| binary format - so not human-readable. I want something in the
| middle: human- and machine-readable. Without either having to do
| parsing gymnastics.
|
| jc isn't quite that shell standard, but looks like it goes a long
| way towards it. And, of course, when JSON falls out of fashion
| and is replaced by <whatever>, `*c` can emerge to fill the gap.
| Nice.
|
| [0]: https://learn.microsoft.com/en-us/powershell/
| vbezhenar wrote:
| Shell would benefit from Content-Type/Accept headers. Like you
| can specify that cat accepts text and jq accepts Json. Then `ip
| a` would output corresponding type automatically.
| mprovost wrote:
| Someone on this site suggested that programs open another
| filehandle along with stdout and stderr (stdjson) for their
| json output which struck me as a way to make this work in a
| backwards-compatible fashion.
| dotancohen wrote:
| Even just a generic stdmeta would go a long way to defining
| _what_ is being output. Curl is the worst about this.
|
| https://unix.stackexchange.com/questions/197809/propose-
| addi...
| thesuperbigfrog wrote:
| >> Shell would benefit from Content-Type/Accept headers. Like
| you can specify that cat accepts text and jq accepts Json.
| Then `ip a` would output corresponding type automatically.
|
| That seems unnecessary. Traditionally, shells have always
| used text streams. JSON is just text that follows a given
| convention. Couldn't what you are describing be implemented
| by setting environment variables or using command line flags?
|
| For example:
|
| PREFERRED_OUTPUT_FORMAT="JSON"
|
| --output-format="JSON"
|
| --input-format="JSON"
|
| Tools that can generate and consume structured text formats
| are a good idea, but they should be flexible enough that they
| can even work with other tools that have not been written
| yet.
|
| "This is the Unix philosophy: Write programs that do one
| thing and do it well. Write programs to work together. Write
| programs to handle text streams, because that is a universal
| interface." --Doug McIlroy
| vbezhenar wrote:
| I don't follow. JSON is not really readable. I don't want
| to see JSON output ever except for script debugging. I want
| to see well formatted output. But at the same time I want
| to be able to write something like ip a |
| filter "[].address like 192.*"
|
| So when I'm typing `ip a` I expect to get output for human
| and when I'm piping it to `filter` program, I expect for
| those programs to exchange with JSON (and ideally `filter`
| should use some tabular formatting as its human-readable
| output).
|
| You suggesting that I should write
| `PREFERRED_OUTPUT_FORMAT=JSON ip a | filter "[].address
| like 192.*"` but that's really verbose and error-prone. It
| might work for scripts, but for ad-hoc shell I don't like
| this approach. Ideally programs should be able to
| communicate between pipes for their preferred formats.
| thesuperbigfrog wrote:
| I was saying that Accept Headers or "format negotiation"
| concepts that are typically used in client-server
| communications are a bit overkill for command line tools
| and shell pipelines.
|
| I agree that human-readable text formats should be the
| default output formats for command line tools, but that
| easy-to-parse structured text output formats should be
| easy to specify with either environment variables or
| command line flags.
|
| If I am writing a script and I am using tools that
| support a given structured output format and use
| environment variables or command line flags for output
| configuration it could work something like this:
| #!/bin/env script-interpreter export
| PREFERRED_OUTPUT_FORMAT="JSON" query-cli-tool |
| filter-cli-tool --output-format=json | combinator-cli-
| tool --input-format=json | pretty-formatter-tool >
| output_file
|
| This would mean that the command line tools default to
| human-readable formats, but can still generate JSON or
| some other structured text format when configured to do
| so.
| bonzini wrote:
| FWIW ip already has JSON support: ip -j
| a s | jq 'map(select((.addr_info |
| .[].local)|startswith("192.168."))) | map(.ifname)'
| teddyh wrote:
| "ip" has the -json option; i.e. "ip -json a" gives you
| straight JSON; no need for JC.
| brodo wrote:
| Nushell has this too. I've tried is as daily driver for a
| while. It's not there yet, but almost. After it hits 1.0 I'm
| going to switch for good and leave the duck tape solutions
| behind.
| hnlmorg wrote:
| Shameless plug but maybe give murex a look. That's been
| stable for a while now and does the same thing too.
|
| HTTPS://GitHub.com/lmorg/murex
| jerjerjer wrote:
| > Powershell has this, but it's a binary format
|
| Well, yes - powershell passes binary objects but as you can
| always:
|
| 1) access their properties 2) pass them downstream 3) serialize
| to json/csv 4) instantiate from json/csv
|
| I think this is both human- and machine-readable enough (even
| through internal format is binary, but working with Powershell
| you are never really exposed to it).
|
| How do you think it can be improved?
|
| In my opinion object io IS the best part of powershell - it
| allows us to ditch results wrangling with sed/awk/grep
| entirely. I'm super interested if there's an even better way
| forward.
| otikik wrote:
| This is a good step in the right direction, which IMHO should be
| that all those cli tools should have a ---json flag.
| asicsp wrote:
| See also:
|
| * "Bringing the Unix philosophy to the 21st century (2019)"
| (https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-ph...)
| - https://news.ycombinator.com/item?id=28266193 _238 points | Aug
| 22, 2021 | 146 comments_
|
| * "Tips on adding JSON output to your CLI app"
| (https://blog.kellybrazil.com/2021/12/03/tips-on-adding-json-...)
| - https://news.ycombinator.com/item?id=29435786 _183 points | 11
| months ago | 110 comments_
| dotancohen wrote:
| If you're bringing that up, then this is the place to spread
| the word about my dream of a stdmeta file descriptor.
|
| https://unix.stackexchange.com/questions/197809/propose-addi...
|
| Just like we have stdout and stderr, header lines such as those
| produced by `ps` should be printed to stdmeta. Curl is the
| worse offender here, outputing meta lines to stderr instead of
| stdout. A stdmeta file descriptor would make it clear what is
| data, what is an error, and what is _describing_ the data.
| anonymoushn wrote:
| If we're adding standard output fds, maybe it would be a
| useful time to define any mechanism for consumers of those
| fds to discern the total ordering of bytes written to the
| three of them.
| dotancohen wrote:
| WC has a "character" flag that really just counts bytes:
| $ echo dvtn | wc -c 9
|
| Note that each letter is two bytes, and the newline is an
| additional byte. So you could pipe (or tree) to wc to count
| bytes. For a hypothetical stdmeta on fd 3, that might look
| like this (piping stdout to devnull): $ foo
| 3>&1 > /dev/null | wc -c
| anonymoushn wrote:
| I'm not sure what you are answering. I am writing about
| the thing where some programs intentionally write
| different byte streams to stderr and stdout that will be
| interleaved a certain way if stdout and stderr turn out
| the be the same thing, but if the user wishes to
| distinguish which bytes were written to which stream,
| they can no longer recover the correct interleaving of
| the bytes.
| dotancohen wrote:
| Yeah, I see now what you wrote. My coffeeless brain must
| have just seen "total ... of bytes". Carry on...
| lstamour wrote:
| The best compromise I can think of would be to prepend
| the line source to the combined output, so lines in the
| combined output can be distinguished. But depending on
| how you do that, you might have timing issues. In fact,
| any form of interleaving the outputs of stderr and stdout
| are technically subject to timing issues because even the
| source application can't specify that bytes in one stream
| should follow bytes in another stream.
| https://stackoverflow.com/questions/12517519/how-to-
| redirect... talks a bit about the problem, and I'm sure
| someone somewhere has written a general-purpose tool, but
| there are timing edge cases here and often the solution
| is seen as keeping errors messages exclusively in stdout
| (or logs) or more practically, not worrying about
| preserving order and writing errors exclusively to stderr
| without worrying as much about strictly when errors
| interleave. If you add timestamps to your line outputs,
| or add some form of line numbers to the output, you can
| restore order after logging but it would be application-
| specific to do so. From a practical perspective,
| listening to both streams and quickly appending to a list
| with a data structure containing the source and message
| text is probably your closest approximation if writing
| code to handle output streams and you want an combined
| output that can still source which stream came from which
| source. (Again subject to timing errors)
| WorldMaker wrote:
| You could require orderable sequence numbers before or
| after each line. It would be easiest as yet another VT
| escape sequence, but if you are pie-in-sky adding more
| standard "files" anyway maybe you wish for lines to be
| newline-delimited JSON CRDTs while you are at it.
| rjh29 wrote:
| Any metadata mechanism should be extensible IMO. For
| example sort and join care about ordering but they also
| care about column names.
| runlevel1 wrote:
| That reminds me of something I've wanted for quite a while:
|
| A ringbuffer filetype. Similar to a named pipe file (see:
| fifo(7)[^1]), but without consuming the contents on read and
| automatically rotating out the oldest lines.
|
| Of course, there would be some complexities around handling
| read position as lines are being rotated out from under you.
|
| [1]: https://linux.die.net/man/7/fifo
| naikrovek wrote:
| > Of course, there would be some complexities around
| handling read position as lines are being rotated out from
| under you.
|
| that seems solvable to me, but multiple simultaneous
| readers... that seems like it might be a bit more
| challenging...
| wooptoo wrote:
| Unrelated tool but also very useful in combination with JQ: gron
|
| https://github.com/tomnomnom/gron
| rob74 wrote:
| Great idea, but sounds like a maintenance nightmare to me. Not
| only that many users will complain that their favorite CLI tool
| isn't supported, but also a new release of any of the supported
| CLI tools might break the support without any kind of warning, as
| I don't think changes to the (human-readable) output are
| considered major changes.
| vidarh wrote:
| Changes to the human-readable output of most standard Unix
| tools are a big deal given the amount of scripts which depends
| on them. They commonly are seen as fairly stable to the point
| that even some non-Posix tools like e.g. apt, which doesn't
| have a stable CLI warns about it to stderr if you try to
| redirect it's output to a pipe[1].
|
| If anything, though, that's a good reason for a tool like this
| to exist rather than have every script that depends on these
| tools use their own, often hacky, parsing of the output.
|
| [1] "WARNING: apt does not have a stable CLI interface. Use
| with caution in scripts."
| slightwinder wrote:
| It doesn't even need a new release. jc can already fail because
| of details like the system-language. With my local language, on
| a simple output of ls -l, it's parsing
| {"filename":"drwxr-xr-x 16 root root 4096 Oct 4 11:21 ."}
|
| instead of {"filename":".","flags":"drwxr-xr-
| x","links":16,"owner":"root","group":"root","size":4096,"date":
| "Oct 4 11:21"}
|
| with LANG=US. This makes it really hard to trust such a tool.
| kbrazil wrote:
| Hi there! `jc` author here. Yes, it is a documented caveat[0]
| that the `C` or `en_US.UTF-8` locales should be used for best
| results.
|
| It's not unheard of for tools to require `C` locale for
| proper parsing: $ LC_ALL=C ls -l | jc --ls
|
| This is one of many inherent issues with using unstructured
| text as an API. That's why I believe there should be a JSON
| (or at least some other widely used format[1]) option for
| tools that have output that would be useful in scripts.
|
| [0] https://github.com/kellyjonbrazil/jc#locale
|
| [1] formats should have good library support across many
| languages and nice filter/query capabilities from the
| command-line
| simonkagedal wrote:
| This is true. However, the issue is with the tool outputting
| unstructured data to begin with. As the `jc` author says, the
| best would be if tools supported structured output formats to
| begin with.
|
| Any custom parser of ls output would potentially have the
| same problem. Of course, it can be improved though - for
| example by looking at LANG - and it would be nice for such
| improvements to get into `jc`, so that other tools can rely
| on it at least more than doing the parsing directly
| themselves.
| vesinisa wrote:
| Seems to be part of the Arch repository:
| https://archlinux.org/packages/community/any/jc/
|
| Why does this site recommend using "paru", "aura" or "yay" to
| install it on Arch? I have been using Arch for a decade or so but
| have never even heard of such tools. They don't even have pages
| in the Arch wiki, and only yay ("Pacman wrapper and AUR helper
| written in go") is available via the standard repository.
|
| Begs the question: what is so wrong with plain pacman?
|
| EDIT: Okay so seems they were previously on AUR and once accepted
| to community repository they just forgot to stop recommending an
| AUR wrapper for installing:
| https://github.com/kellyjonbrazil/jc/commit/f2dd7b8815edc92e...
|
| EDIT2: Created a PR with the GitHub.dev editor .. Absolutely
| blown away by how easy it was! Feels like the future of
| development.. https://github.com/kellyjonbrazil/jc/pull/310
| tazjin wrote:
| It's a program to wrap stable CLI programs, makes sense that
| they recommend a wrapper around a stable program like pacman
| kangalioo wrote:
| Is this a snarky comment implying that the CLI programs and
| pacman are better left unwrapped?
| bennyp101 wrote:
| Looking at github, it was only moved from AUR in May this year,
| but the commit only changed the description, not the commands.
|
| But this is 100% going in my toolbox - I can think of a couple
| of scripts that I can update to use this right of the bat!
| pessimizer wrote:
| I don't know why I never considered this sort of option to
| powershell out bash. My only problem with it is that it's in
| python (also why I don't really use jq), and that it's not
| something that just sets aliases behind the scenes.
|
| If this were written in a performant language, if it simply
| aliased (i.e. invisibly) all common cli commands to a wrapper
| which would obviate the need for all of the text processing
| between steps in command pipelines, if it were versioned and I
| could include the version number in scripts, and finally if I
| could run versioned scripts through it to compile them into
| standard bash scripts (a big ask), I'd give it a 3 month test
| starting today. There'd be nothing to lose.
|
| Just putting that out there for people who like to rewrite things
| in Rust. A slightly different version of this concept could allow
| for nearly friction-free adoption.
| freedomben wrote:
| Why is the python an issue for you? I also dislike and avoid
| tools written in python, unless they have an rpm available. If
| there's a package then I don't so much care what language it's
| in.
| cstrahan wrote:
| > also why I don't really use jq)
|
| You also don't like software written in C (the language jq is
| written in)?
___________________________________________________________________
(page generated 2022-11-03 23:02 UTC)