[HN Gopher] yq: command-line YAML, JSON, XML, CSV and properties...
___________________________________________________________________
yq: command-line YAML, JSON, XML, CSV and properties processor
Author : based2
Score : 105 points
Date : 2023-02-04 17:09 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| encryptluks2 wrote:
| There is also another tool named yq that is Python based, and
| passes everything through jq. The Go-based yq is pretty awesome,
| but does have some limitations.
| notorandit wrote:
| This is really a great tool! Thanks Mike!
| voytec wrote:
| The yaml document from hell[1] needed three changes ("*.html",
| "*.png", "!.git") to be parsed by yq at all. "Norway problem" is
| not a problem as _no_ was converted to a quoted string. Unquoted
| strings in "allow_postgres_versions" part were not quoted by yq.
|
| [1] https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-
| fr...
| MuffinFlavored wrote:
| i wonder if the people who worked on the YAML spec regret not
| string wrapping stuff? i'm sure it was by design "at the time"?
| tyingq wrote:
| The spec says this:
|
| _" The plain (unquoted) style has no identifying indicators
| and provides no form of escaping. It is therefore the most
| readable, most limited and most context sensitive style."_
|
| Which reads to me that they expected people to treat the
| plain style as a convenience that had notable downsides.
|
| Edit: Note that this is the "plain" style, where there's also
| single and double quoted styles.
| tgv wrote:
| A reasonable choice (in contrast to "no" or "null"). YAML
| is simply not a format for all use cases. It's good enough
| for many tasks, and more readable than most other formats
| where it fits.
| dragonwriter wrote:
| > YAML is simply not a format for all use cases.
|
| Maybe, but _unquoted strings_ not being the right choice
| for all use cases (or, similarly, structure-by-
| indentation not being) doesn't show that, since YAML
| supports unquoted and quoted strings, and supports both
| indent-sensitive "block style" and delimiter-based "flow
| style".
| ehPReth wrote:
| honourable mention for HTML: https://github.com/mgdm/htmlq
| [deleted]
| VWWHFSfQ wrote:
| Looks very cool! I don't care so much about YAML, but I do a ton
| of processing of JSON and csv/tsv. Any word on the performance
| relative to jq and xsv [1]?
|
| [1] https://github.com/BurntSushi/xsv
| 0cf8612b2e1e wrote:
| I am all for faster tools, but I am curious as to your use case
| where the jq speed would be limiting. I only ever cleanup a
| maximum of a few megabytes at a time, where the jq response is
| close enough to instant that it has never been a concern.
| snacktaster wrote:
| Being written in Go, I would be very surprised if it's anywhere
| close to as fast as either of those tools.
| user3939382 wrote:
| I recently dug into the docs of jq and was surprised to find
| that, contrary to my prior belief based on shallow experience
| with it, jq's expression aren't merely a path syntax but
| apparently a turing complete language. I was blown away.
|
| I wish MySQL and AWS could have figured out a way to adopt it, or
| a subset of it, rather than each using different ones. Now I have
| varying levels of knowledge for 4-5 variations of JSON path
| semantics/standards, it's annoying.
| nequo wrote:
| User odnoletkov[1] has solved several years' worth of Advent of
| Code in jq:
|
| https://github.com/odnoletkov/advent-of-code-jq
|
| [1] https://news.ycombinator.com/user?id=odnoletkov
| dragonwriter wrote:
| > I wish MySQL and AWS could have figured out a way to adopt
| it, or a subset of it, rather than each using different ones.
|
| For AWS CLI, you can just output unfiltered JSON and pipe the
| results through jq; the filtering is client-side anyway, so
| it's not like you are losing anything doing external filtering
| vs. filtering within the AWS CLI.
| justin_oaks wrote:
| The JSON expressions used by MySQL and the AWS command line
| tool (JMESpath) are extremely limited compared to jq.
|
| So not only do we end up learning multiple JSON path
| variations, but most of them are nearly useless for anything
| but the simplest use cases.
|
| I appreciate the intention of including JMESpath in awscli, but
| I quickly dropped it in favor of piping the JSON results to jq.
| mdaniel wrote:
| I have a similar complaint but I'd guess there are (at least)
| two problems standing in the way of awscli getting jq
| language support: a python impl of the language with a
| license that awscli tolerates, and awscli being (in general)
| very conservative about changes. There are innumerable open
| issues about quality of life improvements that are "thank you
| for your input" and I'd expect that change to be similarly
| ignored
| eru wrote:
| It's not just a Turing complete language, but a well designed
| one, too.
| WesolyKubeczek wrote:
| There has been a jq written in jq here on the front page.
| worldsavior wrote:
| Looks cool, but disappointing it's written in Go. Go is fast, but
| not as fast as Rust or C. I'm sure with large streams, you
| probably can see the difference in time if it's written in Rust
| or Go.
| ihucos wrote:
| Can we stop this programming language bashing.
| theonemind wrote:
| gojq works great with YAML and reimplements jq itself in Go. I
| use gojq with --yaml-input or --yaml-output (or sometimes both)
| and flip back and forth between JSON and YAML promiscuously and
| have 100% jq UI compat, which helps because I use jq a lot. First
| thing I looked at on yq is '-s', which is 'slurp' for jq, but
| different for yq. Slightly altered semantics would just trip me
| up, and it seems like you can make a nearly straight bijection
| between YAML and JSON so you can just do exactly the same things
| with either one (with some minor exceptions.)
|
| https://github.com/itchyny/gojq
| 0cf8612b2e1e wrote:
| gojq does not preserve key order or offer option to sort keys.
| Which is a non-starter for me. The majority of my jq use is to
| cleanup API responses for easier human review.
| tenken wrote:
| yup. this. It's portable! ... But nerfed.
| karmakaze wrote:
| They should be feature requests to gojq. There must be
| libraries for maps with sorted keys or preserving insertion
| order to use in place of the std 'map'.
| pokstad wrote:
| I believe gojq is also bundled with Benthos. Benthos is a great
| Swiss army CLI tool for various data manipulations.
| filereaper wrote:
| yq has a merge function that gojq does not.
|
| We use both tools equally where ease of use is the primary
| driving factor.
| CathalMullan wrote:
| Another tool in this space is Dasel[1], which can handle
| querying/modifying JSON, YAML, TOML, XML and CSV files.
|
| [1] https://github.com/TomWright/dasel
| AndyKluger wrote:
| I do prefer that to jq syntax, as well as alternatives jello
| and yamlpath.
| EdwardDiego wrote:
| `yq` is invaluable when working in the K8s world, love it.
___________________________________________________________________
(page generated 2023-02-04 23:00 UTC)