[HN Gopher] Jsonnet - The Data Templating Language
___________________________________________________________________
Jsonnet - The Data Templating Language
Author : tomas789
Score : 104 points
Date : 2023-03-27 11:52 UTC (11 hours ago)
(HTM) web link (jsonnet.org)
(TXT) w3m dump (jsonnet.org)
| [deleted]
| jonfw wrote:
| I love go templates for this sort of thing, but obviously only
| works if you're using golang
| mdaniel wrote:
| The discoverability of golang templates is terrible, IMHO,
| since it's missing a "dir(locals())" equivalent and every
| execution environment gets to make its own rules about what
| pipelines/functions are exposed
|
| Look at helm as an example:
| https://helm.sh/docs/chart_template_guide/function_list/ is
| some of them,
| https://helm.sh/docs/chart_template_guide/accessing_files/#p...
| are some others, but they also glued in _some version_ of
| https://masterminds.github.io/sprig/ So, short of (a) knowing
| that's the case (b) having 3+ bookmarks in your favorite
| browser to refer to those reference pages, how would anyone
| know what pipelines are available?
|
| Separately, I _dooooo nooooooot_ understand why every joker has
| to invent their own new thing when we have like 50 or so
| templating languages already. Golang may be an outlier in that
| competition due to the Google Promotion Packet Effect(tm) but
| how they came up with `{{ range }}{{ end }}` as sane syntax is
| some true facepalm, to say _nothing_ of the same landmine that
| ansible stepped on by not switching jinja2 's default
| characters: `{{` is not _yaml safe_
| jonfw wrote:
| Helm does a poor job of documenting their functions, and when
| you use helm, you're not really using go templates, you're
| using helm's wrapper around go templates.
|
| If you use go templates directly- it's very obvious what
| functions are available, there's the default ones from the Go
| template package, and there's whatever you pass explicitly.
| The go template package is very small and well documented.
|
| What I love about Go templates, is that I can pass any
| function I want, using good old Go code (meaning I can use
| SDKs and whatever else). This is really powerful- we're able
| to have functions that go out to APIs and return data and
| it's very easy to manage
| mdaniel wrote:
| > helm, you're not really using go templates, you're using
| helm's wrapper around go templates.
|
| Well, that's a tautology when having any contact with
| golang templates outside of quite literally just importing
| "text/template" and good luck, as your subsequent comment
| somehow says and yet misses the point. I draw one's
| attention to the sibling comment of _yet another_ golang
| templating gizmo that injects its own cutesy functions with
| completely random naming into the evaluation namespace
| eviks wrote:
| wha's the best of the 50 in your opinion?
| verdverm wrote:
| own-project plug: https://docs.hofstadter.io
| CSDude wrote:
| We've been using Jsonnet to generate Pods for our complex DAGs at
| Opsgenie, wrote a similar one using Tekton:
| https://mustafaakin.dev/posts/2020-04-26-using-jsonnet-to-ge...
|
| But if I was doing it again now, I would just use
| https://cuelang.org/ or https://dagger.io Jsonnet is really hard
| to debug.
| [deleted]
| sleepybrett wrote:
| Agreed.
|
| I dislike that it's json to transform json, it becomes a rats
| nest. I had similar experiences with xslt back in the day.
| spion wrote:
| Its not JSON though, its more of a stripped down purely
| functional language - functions are quite central.
| aliasxneo wrote:
| I can't seem to wrap my head around CUE. Most of my Jsonnet is
| creating a nice abstraction layer using functions. It seems CUE
| specifically does not like that approach. Maybe I'm just
| looking at it the wrong way.
| rch wrote:
| Similar experience. I ended up switching to hocon with some
| extensions.
| chewbacha wrote:
| Nothing I dread more than needing to edit the tangled web of
| jsonnet files that constitute our grafana dashboards. A jumble of
| spaghetti with hard to find documentation.
| parcec wrote:
| Aren't you supposed to edit grafana dashboards via the UI?
| TechBro8615 wrote:
| I'm not the person you're replying to, but in our company we
| edit them in the UI but check the results into source control
| (so that we can re-deploy the dashboards consistently). I
| imagine for more complicated dashboards you could get stuck
| editing them manually, and it's probably the kind of thing
| where once you do it one time, you're stuck doing it for the
| rest of time.
| sleepybrett wrote:
| There are a few very specific situations in which I might
| consider generating a dashboard without the ui, but even
| then i wouldn't use jsonnet.
| graypegg wrote:
| Honest question, why not just use... JavaScript? I get that this
| is scoped to only templating json, but the syntax is already
| diverging pretty heavily from actual JavaScript, while also
| trying to pretend to be similar? You're using operators and a
| self/this keyword and things, but I can only assume they obey
| different rules.
|
| Could just be config.js. (Or just generate JSON in your app
| programmatically, it's a pretty simple well-defined syntax and
| there's already a hash-to-json library for whatever platform
| you're using)
| mschaef wrote:
| > Honest question, why not just use... JavaScript? I get that
| this is scoped to only templating json, but the syntax is
| already diverging pretty heavily from actual JavaScript, while
| also trying to pretend to be similar?
|
| Jsonnet's stated aim is to be a superset of JSON rather than
| something similar to JavaScript.
|
| Having used Jsonnet for a while, it's nice in that it makes it
| relatively easy to take existing JSON and incrementally turn it
| into templates. You wind up with the ability to create some
| nice abstractions for elements of JSON. The close coupling
| between the syntax of the emitted output and the Jsonnet script
| itself makes it far easier to write a correct template than
| when using something like gotmpl to create Yaml in a Helm
| chart.
|
| Whether or not this is worth another language is a judgement
| call, but it's not that hard to learn or work with, so I've
| tended to find Jsonnet a nice tool to have around.
| lovehashbrowns wrote:
| Used it previously to template some Terraform module
| configurations. The way the flow would work would be YAML
| config -> Jsonnet + defaults -> Terraform + Spinnaker
|
| We wanted to avoid having to inject a more complex piece to the
| puzzle. It worked okay at first, but I knew from the get-go
| that wasn't going to hold for very long and eventually there
| was a Python app that took the YAML config and the
| Jsonnet+defaults and mashed them together before sending the
| info to TF + Spinnaker.
| GoblinSlayer wrote:
| >Or just generate JSON in your app programmatically
|
| Or make an application run on a normal config.
| bboreham wrote:
| One take on that idea: https://github.com/jkcfg/jK
| phailhaus wrote:
| Scoping to templating json lets the library enforce a really
| strict contract that you don't get with javascript. For
| example,
|
| * Accessing a missing field is totally fine in JavaScript and
| returns undefined; this is an error in Jsonnet.
|
| * JSonnet guarantees no side-effects, not true of JS.
|
| * When you run a jsonnet file, you have a consistent CLI that
| handles things like external vars, whereas you'd have to write
| your own wrapper to do this in JS.
|
| * JSonnet lets you export to multiple different config formats,
| whereas JS only support JSON really. Again, you'd have to write
| a wrapper to do this.
|
| * JavaScript would let you return any valid JS object, your
| wrapper would need to validate.
|
| Like yes, you could write your own JS library to mimic a lot of
| these behaviors, but the constraints are what make the tool
| useful. I do wish it had better typing though, so you could
| more easily inspect the structure of the resulting object.
| graypegg wrote:
| If the constraints are what makes it special, why is it just
| not very constrained? If you really wanted something totally
| focused on having multiple configs sharing some base config
| template, then you wouldn't be able to do anything other than
| overwrite specific keys entirely. (json deep merge) Jsonnet
| seems to let you concat strings, do some math using it's
| standard lib (?!), reference other parts of the object etc.
|
| I also don't think "object matches type description" is a
| niche that JS is lacking, there's tons of libraries out there
| to assert that.
|
| You're just trading footguns for different footguns IMO.
| phailhaus wrote:
| Yep, it's all about them tradeoffs. JS gives you absolute
| freedom, JSonnet doesn't. JS is easier to debug, JSonnet
| isn't because it's very declarative.
| spion wrote:
| 1. can be fixed with typescript
|
| 2. can be fixed with Deno -
| https://deno.land/manual@v1.31.3/basics/permissions
|
| 3, 4, 5. Why not have that wrapper to solve those? (AFAIK you
| do need the wrappers for jsonnet too)
| gjulianm wrote:
| I honestly... don't know. I have been using Jsonnet to
| programatically generate Grafana dashboards, because AFAIK the
| only official library to generate them is written in Jsonnet
| [1].
|
| In my experience, it works, but it doesn't really give me any
| distinct advantage and instead it gives me some headaches.
| Maybe for things that look almost like JSON it'd be helpful,
| but the moment you start dealing with more complex generations
| you start finding lack of typing, lack of IDE support, lack of
| easy debugging, etc, fairly problematic. For example, something
| I really dislike is that due to how the expressions are
| evaluated, the only way to add debug/trace statements is to use
| them to "transform" a value you're going to use, if you don't
| use the result of the trace in the final output, the trace does
| not appear.
|
| Also, I really dislike the error messages. Again, due to the
| lazy evaluation design, when you mess up something the error
| message might appear deep in some unrelated call stack and with
| a jarring lack of context. Debugging it is a real pain.
|
| 1: https://github.com/grafana/grafonnet-lib
| thinkmassive wrote:
| Grafana have been working on a replacement for grafonnet but
| unfortunately haven't shared much detail. It sounds like they
| completely rewrote the dashboard spec using CUE, but there
| will be converters available at release.
|
| In some of the discussions where the community is trying to
| figure out how to future-proof current dashboard definitions,
| Grafana Labs has also recommended this Python tool
| (unofficially I suppose) by Weaveworks, called grafanalib:
| https://github.com/weaveworks/grafanalib
| verdverm wrote:
| You may be thinking of the https://github.com/grafana/thema
| library, which is the extracted version from some other
| projects / internal tooling.
| thinkmassive wrote:
| Here's the discussion, "Roadmap: As-code," that I've been
| tracking:
| https://github.com/grafana/grafana/discussions/39593
|
| Thema is mentioned there, as "the presumed successor to
| grafonnet," but that hasn't officially been confirmed and
| it sounds like bigger changes might be underway.
| Discussions are also happening elsewhere, like their
| Slack, but that link is the most complete overview I've
| found.
| verdverm wrote:
| Thema is still under heavy development for sure. I chat
| with Sam about it on the CUE slack from time to time. I
| have a related effort I am working on for data model
| evolution, tracking, and transformation generations as
| part of https://github.com/hofstadter-io/hof
| mike_hearn wrote:
| Yes that's a common problem with config languages. They're torn
| between competing priorities:
|
| 1. Fast to parse with a small engine, good error messages, safe
| to evaluate.
|
| 2. Powerful, can express config with arbitrary logic.
|
| In Conveyor we try an alternative approach. The config is
| HOCON, which is a superset of JSON syntax designed for human
| readability/writability/convenience first and foremost, so it's
| got a very nice and clean feel to it. You can see an example
| here:
|
| https://github.com/hydraulic-software/github-desktop/blob/co...
|
| It can be parsed with a normal-sized config library and the
| errors you get are reasonable.
|
| But then what if you hit the limits of what it can express? We
| added support for "hashbang includes":
| include "#!script.js"
|
| You can embed arbitrary commands in the config which are
| executed when found unless the app is running in untrusted
| mode. The script is expected to produce more config on stdout,
| which is then included. This lets you encode only the minimal
| needed logic using a full programming language, whilst the rest
| stays declarative.
| wruza wrote:
| In case you want a fresh opinion, every time I stumble upon a
| config like this, I wish it was just a .json, .js, .py, heck
| even .pl or .php file rather than that.
|
| Why. It's unclear what's the syntactic structure of it at a
| glance. Yes, in stricter languages there's more of
| /[()[\\]{}"'`,;]/, but also I know for sure what's a literal,
| what's an identifier, what's a key, what's a number or a
| date. The whole structure and ordering is obvious.
|
| Same issue I have with nginx, terraform and other ad-hoc
| half-languages half-formats half-templates. The worst part is
| that you have to hack them once a year, but it never "sinks
| in" for good even if you read the docs.
| mike_hearn wrote:
| Thanks. HOCON is a superset of JSON so you can write JSON
| if you like. There's also a spec:
|
| https://conveyor.hydraulic.dev/7.2/configs/hocon-spec/
|
| You can also convert HOCON to JSON and back. I find it a
| lot easier to work with configs when you can get rid of
| superfluous syntax, can use comments, substitutions,
| include files etc. But you don't have to use those.
| rssoconnor wrote:
| jsonnet has lazy evaluation. This allows you to create mutual
| references between different parts of your configuration
| without having to manually topologically sort the evaluation
| order yourself.
| wruza wrote:
| Agreed. Tomorrow they'll realize that templating often requires
| string capitalization and other operations and will start
| reinventing Math, String.prototype and exception handling, if
| not yet.
|
| Nevermind, https://jsonnet.org/ref/stdlib.html
|
| It's just a new language with a new runtime, for some reason
| marketed as a json templater. Literally any language with
| "json" module is at least equivalent to it, but more familiar
| to developers.
| nikolay wrote:
| Jsonnet had a huge opportunity, but due to the extremely slow and
| heavily opinionated lead, this project literally exists only at
| Grafana now!
| ithkuil wrote:
| shameless plug: https://github.com/kubecfg/kubecfg
| tnisonoff wrote:
| Thanks for kubecfg! Happy user :)
| aliasxneo wrote:
| How does this differ from Tanka? At first glance, they seem
| very similar.
| q3k wrote:
| kubecfg is much less opinionated. It will just take a jsonnet
| file, no need to follow any set directory format or hierarchy
| of environments.
| rstrg wrote:
| Template languages are odd. Just use normal programming language
| to generate whatewer you want.
| IshKebab wrote:
| The issue with normal programming languages is that they are
| rarely properly sandboxed and deterministic. Jsonnet can't wipe
| your hard disk or steal your passwords.
|
| Something like Starlark is probably a good option.
| return_to_monke wrote:
| The nix language, albeit for a package manager, is really
| similar. You get json-like "attrsets" and can have variables and
| functions. Then of course you can export it to different formats
| as well.
|
| https://nixos.org
|
| There is also the Dhall and Nickel languages providing features
| alike
| rssoconnor wrote:
| My discussions with David Cunningham resulting in David
| producing jsonnet and me creating what is now known as overlays
| in Nix.
| gillh wrote:
| We use Jsonnet extensively in Aperture project for generating
| control policies.
|
| - Policy spec is expressed in protobuf format
|
| - GRPC gateway plugin is used to generate Swagger (OpenAPI v2)
| spec from proto files
|
| - Jsonnet bindings are generated from Swagger spec.
|
| - Blueprints are implemented using Jsonnet bindings. The users
| generate policies from blueprints by providing configuration in
| yaml (using aperturectl CLI) or via jsonnet mixin.
|
| Relevant links:
|
| - Aperture Blueprints (Jsonnet):
| https://github.com/fluxninja/aperture/tree/main/blueprints
|
| - Generated doc example:
| https://docs.fluxninja.com/development/reference/policies/bu...
|
| - CLI for generating policies from blueprints:
| https://docs.fluxninja.com/development/reference/aperturectl...
|
| - Policy spec:
| https://docs.fluxninja.com/development/reference/policies/sp...
| baq wrote:
| See also https://dhall-lang.org/
|
| > Dhall is a programmable configuration language that you can
| think of as: JSON + functions + types + imports
| jordemort wrote:
| There's also CUE: https://cuelang.org/
| IshKebab wrote:
| Cue is a little different I think. It doesn't attempt to make
| writing configs less tedious through the use of functions.
|
| It is more about schemas and reliably merging documents as
| far as I can tell. But weirdly given its focus on schemas I
| couldn't find any way for a document to link to a schema in-
| band!
| joshuamorton wrote:
| It makes writing configs less tedious using templates.
|
| Not really sure what you mean by "documents" though.
| IshKebab wrote:
| I mean multiple files.
| verdverm wrote:
| Some config formats like Yaml or Jsonl support multiple
| "documents" per file
| evancordell wrote:
| > But weirdly given its focus on schemas I couldn't find
| any way for a document to link to a schema in-band!
|
| In cue there's no real distinction between "schemas" and
| "documents". If you say: value: string
| value: "abc"
|
| then cue "unifies" the definitions for `value`, sees that
| "abc" is a string, and therefore `value` is valid.
| IshKebab wrote:
| Yes I know, but there's no in-band way to say "this file
| conforms to the schema in this other file" like you can
| with XML or JSON schemas.
|
| For Cue you would probably just want to #include a
| file... but as far as I know you can't do that.
|
| That makes it far less useful for IDEs, linters and so
| on.
|
| I don't think there's any reason they couldn't add that
| feature. Just a bit odd that they haven't already.
| vhodges wrote:
| And https://nickel-lang.org/ too.
| mdaniel wrote:
| relevant: https://github.com/tweag/nickel#comparison-with-
| other-config...
|
| FWIW, "YAML Typing = None" is for sure wrong, that's what
| the `!!map` annotations
| (https://yaml.org/spec/1.1/#tag/syntax) are for, and a very
| common RCE vector for dynamic languages since they can
| cause execution when the parser instantiates the _types_
| ackfoobar wrote:
| The first time I have heard of Jsonnet is that the Scala
| implementation is faster than Google's. I wonder if that is still
| the case.
| jorpal wrote:
| Yes the go version takes way longer materialize our configs
| compared at my work compared to sJsonnet (from Databricks). The
| first time you switch it adds (or removes? I forget) a bunch of
| newlines at the end of files which is annoying to handle in git
| but otherwise it's faster and the same.
| ackfoobar wrote:
| Good to hear. Scala is amazing when used pragmatically.
| samgranieri wrote:
| My company uses this, but not for kubernetes. It's a nifty
| product
| aliasxneo wrote:
| We use Jsonnet with Tanka (by Grafana) for managing our
| Kubernetes manifests. It does take a bit to pick it up properly.
| The language itself is straight-forward (coming from Nix it was
| easy), however, learning how to use ksonnet is the challenging
| part.
|
| I'm still tinkering with how to organize things. I don't think
| I've fully grasped the purpose of mixins yet.
| the-alchemist wrote:
| "Any sufficiently complicated data templating language contains
| an ad hoc, informally-specified, bug-ridden, slow implementation
| of half of a Lisp."
|
| Change my mind. :-P
| [deleted]
| bdsa wrote:
| The Prometheus Operator and Thanos projects only officially
| provide jsonnet libraries and example implementations, rather
| than Helm charts (which are provided by the community, but I
| detest Helm charts anyway) so I've picked up Jsonnet in the last
| few weeks to build our new monitoring stack and I'm really
| impressed.
|
| It's a massive learning curve but it has become really powerful
| and flexible.
|
| I started trying to use Tanka but found the mental model a bit
| strange and struggled passing parameters into environments, so I
| tried Kapitan instead and that has been much more productive.
| brancz wrote:
| Original creator of both of those here. I recommend just
| generating things out straight with the `jsonnet -m` command,
| and then doing `kubectl diff` on a PR and apply on merge.
| Scaled huge amount of infra this way and it's easily
| understandable and extensible.
|
| To clean up the diff a bit I recommend using:
| https://github.com/sh0rez/kubectl-neat-diff
|
| Hope it's helpful!
| [deleted]
| jjgreen wrote:
| Ruby's hashes have escaped!
| redleader55 wrote:
| Configuration as code, but in a quirky language - how about no?
| I'm sure someone uses this, someone created this, and so it might
| seem like it's useful, but we have enough clownishness out there
| with yaml, json combined with js, json with eval and so on.
|
| Create a framework in a normal language, but one which is very
| limited by design with regard to what is able to access -
| Skylark/Starlark is a good example[0], "limited Python" and use
| that. The rest works the same way, take an input in this
| language/framework, generate outputs.
|
| [0] - https://bazel.build/rules/language
| spion wrote:
| Why do you think jsonnet is quirky? It has pretty normal
| components overall: local bindings with lexical scope, imports,
| functions. Certainly more pedestrian / closer to most general
| purpose languages. The only strange bit may be thunks / lazy
| evaluation, but its definitely possible to opt out of relying
| on their behavior.
| dilyevsky wrote:
| Tried it[0], worked reasonably well. Be prepared for strong
| opposition from traditional "devops" folks (really just
| rebranded sysadmins) and boneheaded management "who don't mind
| yaml" and will drag everyone down.
|
| [0] - https://github.com/cruise-automation/isopod
| GlickWick wrote:
| My employer (who I don't speak for) used this for a long time.
| Im not normally particularly opinionated about tech choice, but
| I can very strongly say do not use jsonnet unless you enjoy
| adding an esoteric unsupported language to your job configs for
| no reason.
| spion wrote:
| What would you use instead?
| techhazard wrote:
| Not OP, but I can highly recommend just using the
| programming language you already use to make key-value map
| to and then converting that to JSON. That way you only have
| 1 syntax to worry about. In my case that's Python (using
| dictionaries and json.dumps)
|
| You could also try for a templating framework (like jinja2)
| but then you have 3 syntaxes colliding: the programming
| language you call the templating engine with (e.g. Python
| for jinja2) the templating language itself (e.g. jinja2)
| and then JSON as well.
| spion wrote:
| This works for some languages but not others (some typed
| languages are particularly ill-suited to the problem of
| writing CLI tools to dump heterogenous dictionaries
| and/or objects with many optional fields)
| dangwhy wrote:
| > Create a framework in a normal language
|
| i hate working with SBT. Makes maven pom.xml less cruel in
| comparision.
| bminor13 wrote:
| I can definitely sympathize here - in every context, just
| straight JSON/YAML configuration seems never expressive enough,
| but the tooling created in response always seems to come with
| sharp edges.
|
| Here are some of the things I appreciate about Jsonnet:
|
| - It evals to JSON, so even though the semantics of the
| language are confusing, it is reasonably easy to eval and
| iterate on some Jsonnet until it emits what one is expecting -
| and after that, it's easy to create some validation tests so
| that regressions don't occur.
|
| - It takes advantage of the fact that JSON is a lowest-common-
| denominator for many data serialization formats. YAML is
| technically a superset of JSON, so valid JSON is also valid
| YAML. Proto3 messages have a canonical JSON representation, so
| JSON can also adhere to protobuf schemas. This covers most
| "serialized data structure" use-cases I typically encounter
| (TOML and HCL are outliers, but many tools that accept those
| also accept equivalent JSON). This means that with a little bit
| of build-tool duct-taping, Jsonnet can be used to generate
| configurations for a wide variety of tooling.
|
| - Jsonnet is itself a superset of JSON - so those more willing
| to write verbose JSON than learn Jsonnet can still write JSON
| that someone else can import/use elsewhere. Using Jsonnet does
| not preclude falling back to JSON.
|
| - The tooling works well - installing the Jsonnet VSCode plugin
| brings in a code formatter that does an excellent job, and
| rules_jsonnet[0] provides good bazel integration, if that's
| your thing.
|
| I'm excited about Jsonnet because now as long as other tool
| authors decide to consume JSON, I can more easily abstract away
| their verbosity without writing a purpose-built tool (looking
| at you, Kubernetes) without resorting to text templating (
| _ahem_ Helm). Jsonnet might just be my "one JSON-generation
| language to rule them all"!
|
| ---
|
| Though if Starlark is your thing, do checkout out skycfg[1]
|
| [0] - https://github.com/bazelbuild/rules_jsonnet
|
| [1] - https://github.com/stripe/skycfg
| capableweb wrote:
| It always bothered me that the "tutorial"
| (https://jsonnet.org/learning/tutorial.html) is using mixing
| cocktails as examples, rather than actual networks/infrastructure
| which is a much more likely use cases for using jsonnet (or, I'm
| much less into mixing than my fellow developers/operators).
| TechBro8615 wrote:
| If you didn't want a cocktail before working with jsonnet, you
| will want one after.
| drewdrewdrewbie wrote:
| I'll take a Moscow Mule please.
| Leo_Verto wrote:
| The excellent Jsonnet training course [1] by u/Duologic
| somewhat fills this niche by having you play around with k8s
| resources and using external packages.
|
| [1]: https://jsonnet-libs.github.io/jsonnet-training-course/
| kazinator wrote:
| _self_ should refer to the immediately containing object, the
| outermost object should be called _top_ (or whatever). Or perhaps
| found relatively with _self.out.out.out_...
|
| Imagine having to take a nested configuration fragment that
| refers to its own pieces via the absolute self, and trying to
| plant it in some other configuration, at a different nesting
| level.
| q3k wrote:
| > Or perhaps found relatively with self.out.out.out
|
| Jsonnet's internal predecessor /counterpart/inspiration (Google
| BCL/GCL) has something like this and not including it in
| Jsonnet is a feature :).
___________________________________________________________________
(page generated 2023-03-27 23:01 UTC)