[HN Gopher] Show HN: gq - like jq or zq, but you use Go
___________________________________________________________________
Show HN: gq - like jq or zq, but you use Go
Hi HN. I've gotten pretty tired of needing to learn a custom
programming language for a tool I use once a week or less. So I
figured, it might be easier to pick up if `jq` used a programming
language I already know. Voila, gq.
Author : errnil
Score : 90 points
Date : 2022-04-27 16:15 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| pensatoio wrote:
| I love that you wrote this last night in response to the recent
| front page. :D What a cool project!
| errnil wrote:
| :) Thank you! It was fun to write.
| eatonphil wrote:
| Nice work! I also learned about gojq [0] yesterday too. A Go
| clone of jq that you can also embed in Go code.
|
| [0] https://github.com/itchyny/gojq
|
| Edit: oh unlike jq/gojq your tool doesn't implement the jq
| language it allows you to script in Go. Cool! And interesting to
| see your use of the Yaegi interpreter for interpreting the Go.
|
| I'm curious, did you find it faster to interpret with Yaegi vs
| just writing the Go code to a temp file and build/running that? I
| imagine at some point for larger datasets the latter approach
| would be faster?
| tedunangst wrote:
| yaegi is pretty fast. Not compiled fast, but it can chug
| through some pretty large tree walks without trouble. (Using it
| for walking html processing.) Half the work is parsing, which
| is using the compiled library.
| eatonphil wrote:
| Curious what is your scenario for using yaegi and not Go
| itself?
|
| You allow users to run arbitrary Go somewhere?
| tedunangst wrote:
| Instant reloads for changes.
| errnil wrote:
| Precisely. The first pass generated a temp file and
| compiled/ran it, but that lead to annoying pauses when trying
| to run simple scripts (probably 500ms latency? Maybe a tad
| more?). Yaegi cut that latency dramatically. I think you're
| right that this may make it slower for larger scripts or for
| sufficiently large inputs, however the biggest bottleneck for
| me with jq isn't performance but learning it's language
| repeatedly.
| [deleted]
| macintux wrote:
| Discussion on zq from yesterday:
| https://news.ycombinator.com/item?id=31166956
| cabirum wrote:
| Do we even need a tool like that? I mean, just use the language
| you know to do something with the json you have.
|
| Alternatively: the tool should use sql.
| dhagz wrote:
| Just load the JSON into Mongo/Cassandra and query that. /s
| voidon wrote:
| This is cool and I agree about not needing to learn yet another
| language. I love jq, but I always have to look at the docs. I
| recently learned about ctrl-x-e (after 25 years of shell-use!) to
| edit the one-liners in $EDITOR, so verbose go code might actually
| work.
| ricardobeat wrote:
| Go is a fine language, but not great for scripting. The verbosity
| shows in the examples, it's basically skipping five or six lines
| of code to read from stdin and import a functional library.
|
| Learning the jq syntax will have a much better payoff in the long
| run.
| nh23423fefe wrote:
| | gq 'j.Filter(func(n *Node) bool { return
| strings.Contains(n.String(), "good/") })' | jq
| 'map(select(contains("good/")))'
| throwaway894345 wrote:
| I like jq, but the parent is right that it's hard to wrap
| one's head around the way jq thinks about things and how that
| model maps to its grammar. Even in this example, I couldn't
| tell you why `map()` wraps everything else, and I always get
| tripped up on when to use `|` or how to go back and forth
| between a list and a stream. I don't think the answer is
| `gq`, but I certainly understand why people try out other
| languages.
| nh23423fefe wrote:
| every filter operates on its input. in this case the input
| was an array.
|
| every map implementation ive seen has type function(a,b) ->
| list(a) -> list(b)
|
| select(contains("good/")) is a function from string ->
| string|void
|
| so map plus a voidable function equals a filter
|
| | just means connect 2 filters
|
| if you have a stream you turn it into a array with the
| array constructor []
|
| if you have an array you turn it into a stream with the
| producer operator .[]
|
| the api is super orthogonal and well designed. but you do
| have to understand the model
| throwaway894345 wrote:
| Right, I already believe it all makes sense when you
| understand the model, but the model isn't super obvious
| and I often find myself Googling things for reference
| because I don't use it very often; however, I do use
| regular programming languages and I can reason about
| those more easily (although while I like Go, I would be
| more inclined to used a Python- or JS-like language for
| something like this).
| mccanne wrote:
| zq good/ -
| errnil wrote:
| No doubt this tool is lacking. However I will say that for me
| the most important quality in my `jq`-like is not conciseness.
| My main frustration is that I always forget the small details
| of how `jq`'s language works and spend 10 minutes googling. E.g
| doing operations within arrays of objects, using select, and so
| on. Maybe I'm just dumb, but I'm fond of tools that accommodate
| my dumbness!
| sleepybrett wrote:
| Also this seems to mean my pipeline needs to know how to
| compile go if i want to transform some json .. boo hiss
| errnil wrote:
| Actually `gq` does not require a locally installed Go
| compiler. It interprets the supplied Go using Yaegi.
| spacemanmatt wrote:
| i can get the same jq from multiple package installers that
| will ensure it is in my execution path. this covers windows and
| linux, and likely osx if i needed it.
| bachmeier wrote:
| Looks nice. I might even use it if I wrote Go programs.
| johnday wrote:
| I'd give it maybe a day and a half before `hq` makes its way onto
| the front page.
|
| No guesses which language the `h` stands for :)
| ldh wrote:
| Surely `rq` can't be far behind.
| wey-gu wrote:
| Haha 100% sure there will be rq...
| eatonphil wrote:
| I think the joke was that it already exists.
|
| https://github.com/dflemstr/rq
| ducktective wrote:
| rq is criminally underrated. It is a universal convertor
| between json, avro, cbor, message-pack, toml, yaml and
| csv.
|
| It misses XML though, and activity in development ;)
| adolph wrote:
| Two maybe: one for the systems folk and another for data
| scientologists
| mynameisash wrote:
| I was going to post this the other day, but my project[0]
| isn't quite the same as jq. I didn't even know about jq when
| I had a need for some easy JSONL text extraction. No
| aggregation or complex querying -- just extracting from
| arrays/objects in my data.
|
| Sorry to disappoint? :)
|
| [0] https://github.com/aeshirey/jsx
| bicx wrote:
| HTML!
| sleepybrett wrote:
| Firstmover goes to `jq` all these other tools might be great but
| honestly it might just be better to use the dev time to add
| features to jq.
|
| For instance json is a subset of yaml, maybe internally jq they
| could use yaml libraries, consume either, output either json or
| input format by default and give me a flag for yaml output if I
| want it.
|
| As a guy who does a bunch of devops and works with a bunch of
| teams to improve their pipelines etc. jq is a tool they all have
| and i always find in pipelines having to install the new jq
| flavor of the month every six months is terrible.
| kitd wrote:
| _For instance json is a subset of yaml, maybe internally jq
| they could use yaml libraries, consume either, output either
| json or input format by default and give me a flag for yaml
| output if I want it._
|
| As it happens, gojq, mentioned elsewhere in this thread, adds
| this feature, and quite a few others.
|
| Jq has served the industry admirably, but there's always room
| for improvement.
| [deleted]
| dmitriid wrote:
| Ok. Is this the new "todo list" or "hacker news clone" trend, but
| for tools? :)
|
| We need competing implementations in more languages!
|
| (Ignore my snarky tone because: why not?)
| zamalek wrote:
| > Im tired of learning new programming languages.
|
| Hear hear! Building bona-fide compilers that have interesting
| concepts, or just for fun, is a worthwhile thing to do. But these
| one-off languages are becoming ridiculous.
|
| * buildah shows just how much of a headache Dockerfiles are. We
| already have a whole ecosystem of awesome scripting languages,
| such as bash, fish, zx, etc. Instead, we have to deal with the
| incredibly constrained (and harmfully magic) behavior of docker
| build.
|
| * the whole implerative YAML ecosystem. I have been playing
| around with Dagger (CUE, a language built to solve real
| problems), and what a joy compared implerative CI.
|
| * lest we forget how JavaScript came to be:
| https://thenewstack.io/brendan-eich-on-creating-javascript-i...
| yessirwhatever wrote:
| > Hear hear! Building bona-fide compilers that have interesting
| concepts, or just for fun, is a worthwhile thing to do. But
| these one-off languages are becoming ridiculous.
|
| I don't understand this mentality. Some programmers enjoy
| writing programs and some enjoy learning about programming
| languages. No one is telling all programmers they they need to
| learn about every programming language, but why paint the whole
| endeavor as frivolous?
| dijit wrote:
| I've been a sysadmin (now, devops, SRE, whatever) for over 15
| years now, so when you say:
|
| > No one is telling all programmers they they need to learn
| about every programming language
|
| I cringe a little.
|
| There are about 25 different DSL's in my head from
| configuring various desired state configuration systems or
| services which use DSLs, there's countless config file
| formats ranging for ZONE files to TOML to YAML to HCL and on
| and on and on.
|
| Makefiles, Dockerfiles, Ninja files, Jenkinsfiles
| etc;etc;etc;etc;
|
| DevOps of Yore had to know bash and perl, then it was Ruby,
| then python, now Go. The treadmill continues, it's
| _necessary_ to learn those things because you _must_ interact
| with those things.
|
| That's before we get into special languages like AWK which
| had significant prominence when I was beginning life as a
| sysadmin too.
|
| Nobody is forcing programmers to know every language, but
| honestly, there's a lot that sysadmins had to learn, and with
| the rise of DevOps, developers are the SysAdmins of tomorrow:
| so you have to know what we knew.
| yessirwhatever wrote:
| I feel your tiredness.
|
| But I don't understand why you'd say it'd make you cringe.
|
| You can have a discussion about how the industry (usually
| motivated by non-technical reasons) is moving towards. I
| was talking specifically about those "one-off" languages
| that some people find fun to investigate and/or work on.
| dijit wrote:
| I cringe because you said that programmers don't have to
| learn many languages.
|
| But in order to do a good job a sysadmin has to learn the
| semantics of dozens.
|
| So you're right, in a way, but it's not right at all for
| sysadmins.
|
| Since sysadmins are being essentially replaced by
| programmers, it won't be right for programmers soon
| either.
| zamalek wrote:
| From what you quoted:
|
| > Building bona-fide compilers that have interesting
| concepts, or just for fun, is a worthwhile thing to do.
| cpuguy83 wrote:
| I agree that Dockerfile has some constraints, but I'm not sure
| I get the "magic" part. It's also not like it's a full
| language. Run a command, set an env var. There's some
| additional flags to set for optimizations. Otherwise minimal
| things to get you to an env in a container that you can do
| whatever you want.
|
| Also +1 to dagger+cue. I like that it gives me essentially for
| full power in buildkit's llb exposed as cue configs.
| forty wrote:
| My latest story with Dockerfile:
|
| my.tar contains files owned by user 0, I want them to be
| owned by nobody: USER nobody ADD
| --chown nobody:nobody my.tar /app
|
| Damn, --chown does not work when untaring and doesn't fail
| either but the files end up being owned by root. Fine, I'll
| just chmod USER nobody ADD my.tar /app
| RUN chmod -R nobody:nobody /app
|
| Now it fails (of course) because the files are owned by root
| and the command is ran as nobody.
|
| While it all makes sense, I'm not sure why it's logic to have
| the ADD command always run as root and the others following
| the USER directive (yeah and I know, it's all my bad for
| using ADD rather then COPY)
| zamalek wrote:
| > RUN chmod -R nobody:nobody /app
|
| Did you know: RUNning chmod increases the size of the layer
| by the size of the file being chmodded[1]. Just some more
| of that magical secret sauce.
|
| (With buildah you choose when to create a layer, so both
| the COPY and the chmod can be layered together)
|
| [1]: https://blog.vamc19.dev/posts/dockerfile-copy-chmod/
| forty wrote:
| Ah well, no I did not know, thanks :) in my case it's not
| too bad because I use kaniko's option to make a single
| layer of my changes, but good to know.
| zamalek wrote:
| > I'm not sure I get the "magic" part
|
| The problem I was trying to solve was container signing, as
| well as building AMD64+ARM64 multi-arch images without buildx
| having access to both contexts, i.e. on different VMs (a
| CircleCI limitation). When you use buildah _without bud_ you
| learn just how many implicit steps are involved in a
| Dockerfile, and how those implicit steps have been the
| problem all along.
|
| buildx works great when you're on the razor thin happy path.
| The abstraction is great for the majority who need it but,
| crucially, _Dockerfiles provide no way to escape the
| abstractions._
| IceWreck wrote:
| Can someone explain why people are creating so many CLI tools
| (with a very weird and custom syntax) to extract JSON using a
| very complicated one line command when you can make one off
| scripts to do the same using python/go/ruby/node/whatever ?
| QuikAccount wrote:
| I'd argue that it is less about practicality and more about
| learning a language by recreating a CLI tool.
| NAR8789 wrote:
| oneliners are generally nicer for fast data exploration than
| one-off scripts. With a oneliner I can get started instantly,
| and build incrementally. With a one-off script my first step is
| boilerplate input handling.
|
| Maybe a nice middle ground would be for more language runtimes
| to have an equivalent to perl's `-p` flag.
| https://perldoc.perl.org/perlrun#-p
|
| This flag is so convenient that I mostly use `perl -pe` instead
| of `sed` for regex search-and-replace.
| UncleEntity wrote:
| I wrapped boost::property_tree in a python module to read in a
| rss xml feed and store the items in a json file to track the TV
| shows I download all because the pure python version was kind
| of slow.
|
| The amount of time I spent on getting all that working is
| probably a lot less than waiting a few extra seconds on a
| python script I run every few days so...
| vemv wrote:
| I keep using "My jq killer" as I commented back in 2016:
|
| https://news.ycombinator.com/item?id=13091469
| sleepybrett wrote:
| Now my build/ci/cd container needs fucking ruby in it .. no
| thanks. I broke myself from my 'just use perl instead of
| awk/sed' for the very same reason. The smaller those containers
| are the faster my pipeline runs.
| pipeline_peak wrote:
| Why not just pipe into a python program that uses a mature JSON
| parser? These "*q" things are really boring.
| joshbaptiste wrote:
| because many folks just want a single binary and not a full
| language library
| pipeline_peak wrote:
| If you are using whateverq and writing a shell script,
| everyone else using it needs to have whateverq installed as
| well....
| sidpatil wrote:
| Unless you bundle the whateverq alongside the shell script.
| Which is a primary design advantage of single small
| statically-built binaries.
| pipeline_peak wrote:
| That's true, but isn't that kind of a hassle on both
| sides?
| adolph wrote:
| Just encode and append the binary to the script:
|
| https://unix.stackexchange.com/questions/450614/is-there-
| a-w...
| wpietri wrote:
| You could use the same logic to ask yourself why there should
| be any command-line tools at all, just python libraries.
|
| Python is my daily driver, but I still use JQ when I'm doing
| something quick. If you try some complex examples, I'd bet
| you'll quickly figure out why.
| sleepybrett wrote:
| Why do I want to bloat my pipeline container with a fucking
| python install including all the libraries? I'd rather pop a
| single tiny binary in there. Container bloat = slower
| pipelines.
| pipeline_peak wrote:
| If you're working in a Unix environment within a container
| isn't there a high likelihood there already is Python as
| well?
|
| I never said every single library, also Python scripts can
| run as standalone executables.
|
| If you truly can't have Python because it's too bloated, then
| JQ in shell scripts is a fair use case, that just sounds so
| rare.
| adolph wrote:
| True, high likelyhood there is _a_ Python but maybe not the
| Python your script needs. The situation is better nowadays
| but 2.7 persisted for a really long time.
| eatonphil wrote:
| Is go's json parser less mature than python's?
| pipeline_peak wrote:
| I was under the assumption that this thing is using it's own
| parser, I guess it doesn't, idc. The point I'm trying to make
| is that JQ depends on the user investing the time to know an
| esoteric domain language. This thing is fed by Go scripts
| that get interpreted, at that rate just use a powerful shell
| language like Python.
| tedunangst wrote:
| What if I'm more familiar with go and python is an esoteric
| language?
| pipeline_peak wrote:
| In the world of shell scripts, python isn't an esoteric
| language....
| mdaniel wrote:
| this tool may interest the same audience:
| https://github.com/hairyhenderson/gomplate
|
| I find it especially handy for poc-ing helm snippets, although
| regrettably they don't have the same "standard library" of
| functions
| renewiltord wrote:
| I use https://github.com/itchyny/gojq . I like `jq`'s functional
| style, but I don't like the way it breaks number precision (or
| used to).
|
| However, for deep `jq` selects I use `<file.json | gron | grep
| key -C n | norg` where `norg` is `gron --ungron`.
|
| I also prefer `gron` for ad-hoc pipelines and `jq` for longer-
| lasting ones.
___________________________________________________________________
(page generated 2022-04-27 23:02 UTC)