[HN Gopher] A Common Lisp jq replacement
___________________________________________________________________
A Common Lisp jq replacement
Author : tmtvl
Score : 123 points
Date : 2025-05-02 12:15 UTC (10 hours ago)
(HTM) web link (world-playground-deceit.net)
(TXT) w3m dump (world-playground-deceit.net)
| precompute wrote:
| Whoa, love the website!
| mhitza wrote:
| It's definitely rocking that Common Desktop Environment style
| https://en.m.wikipedia.org/wiki/Common_Desktop_Environment
|
| For those that like that style, on Linux both Xfce and KDE have
| themes that replicate it for their window decorations
| (recommending the desktop environment would be a bit too much)
| ramses0 wrote:
| He's got an "about..." section which is a treat and gives some
| details on it. It works surprisingly well and is a bit of a
| blast from the past! :-D
| diggan wrote:
| On a similar note: https://github.com/borkdude/jet
|
| Can convert between JSON<>EDN<>YAML<>Transit easily, plus
| includes a nifty little query language that is basically Clojure,
| so data transformations/extraction ends up really simple and
| concise.
|
| I've always liked jq for simple things, but since I never sat
| down to actually work through the syntax, harder things tend to
| be too complicated to figure out quickly. Usually end up using
| Jet instead as if you already know Clojure, you already know the
| query language Jet uses.
| foobarqux wrote:
| Also lqn/jqn
|
| https://github.com/inconvergent/lqn
| Jenk wrote:
| And jaq: https://github.com/01mf02/jaq
| pama wrote:
| Good performance improvement here: https://world-playground-
| deceit.net/blog/2025/03/speeding-up...
| stassats wrote:
| >(safety 0)
|
| Please, don't do that!
| sauercrowd wrote:
| Why's that? What does it do?
| aidenn0 wrote:
| The Lisp standard allows it to be implementation dependent,
| but in SBCL (which I believe the author of TFA is using),
| it disables all runtime type and bounds checking. This
| usually speeds things up negligibly[1] in exchange for
| allowing all kinds of undefined behavior.
|
| 1: If it speeds things up non-negligibly, there's almost
| always a way to get a similar speedup without setting
| safety to 0; e.g. if you check your types outside of your
| hot loops, the compiler is smart enough to omit type-checks
| inside the loop.
| jlarocco wrote:
| Honestly it's more nuanced than that, and probably not
| such a big deal.
|
| It's kind of like building in Debug mode in other
| languages. Internally and for testing, use (safety 3). If
| the code in question doesn't trigger any errors or
| warnings, then in most cases it's safe to turn (safety 0)
| and get the tiny performance boost.
|
| I wouldn't recommend (safety 0) globally, but it's
| probably fine locally in performance critical code that's
| been tested well, but I do agree it's probably not worth
| going to (safety 0) in most cases.
|
| The best solution is a compiler who's (speed 3)
| optimization level is smart enough to optimize out the
| unnecessary safety checks from (safety 3). I think SBCL
| can do that in some cases (the safety checks get
| optimized for speed, at least).
| aidenn0 wrote:
| > If the code in question doesn't trigger any errors or
| warnings, then in most cases it's safe to turn (safety 0)
| and get the tiny performance boost.
|
| This is trivially not true. Consider:
| (defun foo (x) (declare (safety 0)
| (type x (array fixnum (4))) [Lots of code that
| doesn't trigger any warnings])
|
| Then in a different source file doing e.g:
| (foo nil)
|
| Nothing good will come of that.
|
| > I wouldn't recommend (safety 0) globally, but it's
| probably fine locally in performance critical code that's
| been tested well, but I do agree it's probably not worth
| going to (safety 0) in most cases.
|
| > The best solution is a compiler who's (speed 3)
| optimization level is smart enough to optimize out the
| unnecessary safety checks from (safety 3). I think SBCL
| can do that in some cases (the safety checks get
| optimized for speed, at least).
|
| The only thing I can think of is that I communicated
| things poorly in my comment, because this is nearly
| exactly what I was saying in my comment.
| jlarocco wrote:
| Sure, that code fails with a memory fault with (safety
| 0). But I explicitly recommended testing and debugging
| with (safety 3) first, and in that case it gets a type-
| error and doesn't crash. Once the broken `(foo nil)` is
| fixed and it's time to ship a release build to the user,
| it should be safe to drop down to (safety 0).
|
| I think we both agree that 99.9% of the time it's not
| worth using (safety 0), though.
| BoingBoomTschak wrote:
| True, this was initially for the sake of benchmarking. Very
| bad idea, especially since it doesn't really benefit here.
| naniwaduni wrote:
| This is a bit of a nonsense benchmark--the author is basically
| benchmarking process startup time, and finding that if you only
| start one lisp process, it runs faster than if you start one jq
| process per input file, which runs faster than if you start one
| lisp process per input file. This becomes quite clear at the
| end when they compare multiple jq implementation (which are
| known to have nontrivially different parsing performance
| characteristics, which is the bulk of the "real work" for this
| task lies) and come up with execution times on the same order
| of magnitude.
|
| A more apples-to-apples comparison would be to use find {} + to
| pass multiple filenames to jq and output using input_filename.
| gray_-_wolf wrote:
| One huge advantage of JQ is that it often is installed. I have jq
| in our Jenkins image, but I do not have this tool. The syntax is
| bit arcane, but once you invest bit of time into learning it, it
| starts to make sense (to a degree). It is a reasonable language
| for a stream processing.
| ramses0 wrote:
| There's a few jq patterns I've adopted: echo
| "$SOME_JSON" | jq '.[]' --raw-output --compact-output | while
| read -r LINE ; do ... ; done
|
| ...lets you process stuff "record by record" pretty
| consistently. (and `( xxx ; yyy ; zzz ) | jq --slurp '.'` lets
| you do the reverse, "absorbing" multiple records into an array.
|
| Don't forget `--argjson` echo "{}" | jq
| --argjson FOO "$( cat test.json )" '{ bar: $FOO }'
|
| ...lets you "load" json for merging, processing, formatting,
| etc. The leading "{}" is moderately necessary because `jq`
| technically _processes_ json, not generates it.
|
| Finally, it's a huge cheat code for string formatting!!
| $ echo "{}" | jq \ --arg FOO "hello \$world" \
| --arg BAR "complicated \| chars" \ --arg ONE 1 \
| --arg TWO 2 \ '"aaa \( $FOO ) and \( $BAR ) and \(
| ($ONE | tonumber) + ($TWO | tonumber) ) bbb"' "aaa
| hello $world and complicated \\| chars and 3 bbb"
|
| ...optionally with `--raw-output` (un-json-quoted), and even
| supports some regex substitution in strings via `... |
| gsub(...)`.
|
| Yes, yes... it's overly complicated compared to you and your
| fancy "programming languages", but sometimes with shell stuff,
| the ability to _CAPTURE_ arbitrary command output (eg:
| `--argjson LS_OUTPUT="$( ls -lart ... )"`), but then also use
| JSON/jq to _safely_ marshal/deaden the data into JSON is really
| helpful!
| naniwaduni wrote:
| > The leading "{}" is moderately necessary because `jq`
| technically _processes_ json, not generates it.
|
| The --null-input/-n option is the "out-of-the-box" way to
| achieve this, and avoids a pipe (usually not a big deal, but
| leaves stdin free and sometimes saves a fork).
|
| This lets you rewrite your first "pattern":
| jq -cnr --argjson SOME_JSON "$SOME_JSON" '$SOME_JSON[]' |
| while read ...
|
| We also have a "useless use of cat": --slurpfile does that
| job better: jq -n --slurpfile FOO test.json
| '{bar: $FOO[]}'
|
| (assuming you are assured that test.json contains one json
| value; --argjson will immediately fail if this is not the
| case, but with --slurpjson you may need to check that $FOO is
| a 1-item array.)
|
| And of course, for exactly the single-file single-object
| case, you can just write: jq '{bar: .}'
| test.json
| great_wubwub wrote:
| gron does 90% of what I need for json processing, it's a
| great first step and often the only necessary step.
|
| https://github.com/tomnomnom/gron
| aidenn0 wrote:
| I fail to see how your string-formatting example is better
| than using bash's printf?
| cube2222 wrote:
| The sentiment resonates with me.
|
| Had similar thoughts a couple years ago, and wrote jql[0] as a jq
| alternative with a lispy syntax (custom, not Common Lisp), and
| I've been using it for command-line JSON processing ever since!
|
| [0]: https://github.com/cube2222/jql
| behnamoh wrote:
| How is $ echo "$json" | cljq '(? $ "root" * 1)'
|
| more intuitive than the good ol' jq $ echo
| "$json" | jq '.root | map(.[1])'
|
| Really, people should know by now that jq does point-free
| programming.
| naniwaduni wrote:
| Well you see, the author already knows common lisp, is familiar
| with their own ad hoc DSL by virtue of having just come up with
| it, and refuses to learn jq.
|
| Personally, I probably would've written '[.root[][1]]' for that
| problem myself though--not a huge fan of map/1.
| BoingBoomTschak wrote:
| Just noticed this was submitted here.
|
| 1) I dislike that .[1] can be both an expression evaluated as a
| query and a "lambda". Really messes with my mind.
|
| 2) In my eyes, it's more intuitive because it looks like
| globbing and everybody knows globbing (this is the reason I use
| `**` too).
|
| But yeah, this is a bit subjective. What isn't, though, is that
| I don't plan on adding much more than that; maybe merge,
| transform and an accessor using the same syntax. So if you know
| the host language, there's much less friction.
|
| I really see this like Avisynth vs Vapoursynth.
| pletnes wrote:
| Pyjq is the python package for jq, in case you want to use jq as
| a library. The article claims that this doesn't exist.
| cAtte_ wrote:
| no, what the author claims is that jq should've been designed
| as a regular python library, using python syntax, rather than
| as a bespoke DSL (that happens to have parser bindings for
| python)
| pletnes wrote:
| Well spotted!
| vindarel wrote:
| Similar, in CL too:
|
| * [lqn](https://github.com/inconvergent/lqn) - query language and
| terminal utility for querying and transforming Lisp, JSON and
| other text files.
|
| (by this person doing nice generative art:
| https://inconvergent.net/)
| rafram wrote:
| The `?` query operator is just a different, equally inscrutable
| DSL...
| aidenn0 wrote:
| I'm guessing there's supposed to be something other than an empty
| black box after "already an improvement, in my eyes:"? I'm just
| seeing a black box (on firefox).
|
| [edit]
|
| Removing "Unifont" from the font-family list fixes the problem,
| so I must have an issue with my unifont install?
| account-5 wrote:
| I learned the basics of jq and quite liked it, but since I
| discovered Nushell it has replaced nearly all my data processing
| I do at the cli. It really is good technology.
| 1oooqooq wrote:
| site looks like a bashcompletion thing, how does it replace jq?
| rafram wrote:
| Not sure where you're getting the idea that it's a Bash
| completion extension from. It's a new shell (see name) that
| natively supports complex nested data structures, numbers,
| numbers with units, and so on. Compare with the classic POSIX
| shell model where everything is line-based (defined loosely),
| numerical operations rely on hacks, and splitting command
| output lines relies on hardcoded column indices.
| bpshaver wrote:
| I have nushell installed and use it sometimes. Does it have
| built-in JSON parsing like jq?
|
| Edit: Well, I just found out about `cat some.json | from
| json` in nushell. Pretty cool! The nested tables are nice.
| account-5 wrote:
| No need for that even. It's just:
|
| open some.json
| account-5 wrote:
| It's definitely not bash completion. But on the jq note you
| can read JSON, and various other formats, directly into
| nushell's data model (a table) and just start querying it.
| forty wrote:
| > I seriously dislike jq's convoluted, impossible-to-remember ad
| hoc DSL that instantly joined heaps of misery like CMake and
| gnuplot in my heart.
|
| I like jq and gnuplot quite well. Makes me want to try CMake out
| ;)
| selkin wrote:
| I am old enough to remember when creating a new DSL for every
| task was in vogue. What used to be an issue with that approach,
| which I've seen hinted to in another comment here, is that I can
| only become proficient in so many programming languages.
| Achieving proficiency requires a lot of practice, so using that
| specific DSL needs to have a high value to justify the time
| investment in practicing.
|
| This issue is almost negated today: I find myself no longer
| writing jq queries, or regular expressions (both I am quite
| proficient in,) but having AI write those for me. This is exactly
| where the so-called "vibe coding" shines, and why I no longer
| care about tool specific DSLs.
| adamgordonbell wrote:
| If you go through and learn some basic examples, JQ is a lot more
| understandable than seeing golfed examples in the wild might led
| you to believe. I wrote a tutorial once.
|
| But it does also seem like a place where LLMs are handy. Why
| learn jq or regex or AWK, if you use them infrequently, when you
| can just ask an llm?
|
| Edit: tutorial: https://earthly.dev/blog/jq-select/
| guelo wrote:
| It feels like we're in danger of being stuck with the popular
| tools we have now for the foreseeable future. Why use a new
| tool when you can have the LLM figure it out using the old
| tools? Which means LLMs can't learn new tools because there's
| no human examples for them to learn from.
| photonthug wrote:
| Of course, it's different for tools that aren't open, but.
| Using best-in-class tools rather than making new ones is
| usually just a reasonable choice to avoid fragmentation of
| effort/interest that's ultimately pretty harmful to any
| software ecosystem.
|
| As an example.. any candidate for replacing jq needs to be
| either faster or easier. If it's only a faster
| implementation, why change the query language? If it's only a
| different query language but not faster, then why not
| transpile the new query language into one that works with the
| old engine? Doing _both_ at the same time without sacrificing
| completeness /expressiveness in the query language may
| warrant fragmentation of effort/interest, but that's a _very_
| high bar I would think..
___________________________________________________________________
(page generated 2025-05-02 23:00 UTC)