[HN Gopher] Jevko: a minimal general-purpose syntax
___________________________________________________________________
Jevko: a minimal general-purpose syntax
Author : memorable
Score : 46 points
Date : 2022-10-21 13:09 UTC (4 days ago)
(HTM) web link (djedr.github.io)
(TXT) w3m dump (djedr.github.io)
| tomjakubowski wrote:
| I'd be interested in a more detailed comparison to s-exprs. Jevko
| reads less noisily because it uses fewer brackets, which is nice,
| but the ambiguity around whitespace gives me pause.
|
| How is it simpler or lower-level than s-exprs?
| djedr wrote:
| One comparison is here:
| https://jevko.github.io/compactness.html
|
| To see how it's syntactically simpler than S-exps note that the
| grammar of Jevko can be condensed into one short line of ABNF:
| Jevko = *("[" Jevko "]" / "`" ("`" / "[" / "]") / %x0-5a / %x5c
| / %x5e-5f / %x61-10ffff)
|
| The grammar of S-exps on the other hand, I won't quote here,
| but I assure you it's much more complicated. How much depends
| on your flavor (Jevko is also simpler in this regard: there is
| only one flavor, clearly specified).
|
| There is no (intended) ambiguity around whitespace in Jevko:
| whitespace does not occur explicitly in the grammar. Whitespace
| characters are just characters. This is the defining feature of
| the syntax.
|
| For this reason Jevko is more low-level: if you want to treat
| whitespace in some special way, you have to do that yourself.
| Although for most use-cases this is very similar and simple,
| e.g. https://news.ycombinator.com/item?id=33334314
|
| But the point is that you can also leave it as-is, e.g.:
| https://github.com/jevko/queryjevko.js
|
| or do something else -- it's up to your format.
| simplotek wrote:
| > _To see how it 's syntactically simpler than S-exps (...)_
|
| Has anyone ever complained that S-expr were too complex?
| Adding white noise as a tradeoff doesn't seem like a win.
| kragen wrote:
| Hmm, wouldn't S-exps in the same ABNF notation be
| ex = ("(" *ex ")" / *%x2a-10ffff) *%x0-20
|
| We could write the data model corresponding to this grammar
| in OCaml as type ex = List of ex list |
| Atom of string
|
| If we add double-quoted strings with GW-BASIC/SQL-style
| quoting (and resolve the ambiguity greedily):
| ex = ("(" *ex ")" / *%x2a-10ffff / %x22 *(%x22 %x22 / %x0-21
| / %x23-10ffff) %x22 ) *%x0-20
|
| This corresponds to the data model type ex
| = List of ex list | String of string | Symbol of string
|
| This still seems both simpler and more expressive than the
| one-line grammar you give. Maybe I'm missing a subtlety of
| S-expressions here, and I haven't tried it, but I think this
| correctly parses your examples like (first-
| name"John"last-name"Smith"is-alive true age 27
| address(street-address "21 2nd Street"
| city"New York"state"NY"postal-code"10021-3100")
| phone-numbers((type "office"number"212 555-1234")
| (type "home"number"646 555-4567"))
| children()spouse())
|
| or ( :first-name "John"
| :last-name "Smith" ... )
|
| though of course the name-value pairing is lost there
| (because S-expressions lack it).
|
| (By the way, if you want to attribute your JSON example for
| copyright reasons, you need to attribute it to its author or
| authors, not to the Wikipedia, which is just the site they
| posted it on.)
|
| -- *** --
|
| Maybe more importantly, though, I think your abbreviated
| Jevko grammar is wrong in an important way, though it
| describes the same set of strings as the full grammar.
| According to your abbreviated Jevko grammar, the Jevko
| examples lack most of the structure of the XML, JSON, and
| S-expression versions. It implies that a Jevko is an ordered
| sequence of (Unicode!) characters and nested Jevkos
| (indicated by []). We could express this data model as
| type jevko = atom list and atom = Char of char | Nest
| of jevko
|
| If so, then abbreviating your S-expression example
| (first-name "John" last-name "Smith")
|
| the closest Jevko equivalent is not, as you claim
| first name [John] last name [Smith]
|
| but rather (supposing the hyphens were just an unfortunate
| concession to S-expression syntax rather than actually
| desired) [first name][John][last
| name][Smith]
|
| We had to sacrifice the formatting white space because
| there's nowhere that Jevko (as specified above!) ignores it.
|
| Using this grammar, the S-expression equivalent of the Jevko
| first name [John]
|
| is rather ("f" "i" "r" "s" "t" " " "n" "a"
| "m" "e" " " ("J" "o" "h" "n"))
|
| If we instead use the full Jevko grammar
| Jevko = *Subjevko Suffix Subjevko = Prefix "[" Jevko
| "]" Prefix = Text Suffix = Text Text
| = *Symbol Symbol = Digraph / Character
| Digraph = "`" ("`" / "[" / "]") Character = %x0-5a /
| %x5c / %x5e-5f / %x61-10ffff
|
| or, I think, equivalently in this context:
| Jevko = *Subjevko Text Subjevko = Text "[" Jevko "]"
| Text = *(Digraph / Character) Digraph = "`" ("`" /
| "[" / "]") Character = %x0-5a / %x5c / %x5e-5f /
| %x61-10ffff
|
| then we do preserve the name-value structure you seem to be
| going for, which your above one-line version loses. And this
| allows us to write first name[John]last
| name[Smith]
|
| as in your compactness examples.
|
| I think this is a more useful level of abstraction, and it's
| more or less the level used by, for example, queryjevko.js's
| jevkoToJs, although that erroneously uses () instead of [].
| (Also, contrary to your assertion above that this is an
| example of "leaving [Jevko's data model] as-is", it forgets
| the order of the name-value pairs as well as I guess all but
| one of any duplicate set of fields with the same name and
| also the possibility that there could be both fields and a
| body.)
|
| Essentially at this level of structure a Jevko is a (possibly
| empty) set of name-value pairs followed by a plaintext body
| ("Suffix"). This is exactly like an email message, except
| that the values are themselves Jevkos. In OCaml we could
| write: type jevko = Jevko of (string *
| jevko) list * string
|
| For your example
| include[author]fields[articles[[title][body]]people[[name]]]
|
| this gives the representation Jevko
| ([("include", Jevko ([], "author")); ("fields",
| Jevko ([("articles", Jevko
| ([("", Jevko ([], "title")); ("", Jevko ([], "body"))], ""));
| ("people", Jevko ([("", Jevko ([], "name"))], ""))],
| ""))], "")
|
| although I notice that queryjevko handles it very
| differently.
|
| -- *** --
|
| Unlike the data model implied by your one-line grammar, I
| think this is an _extremely useful_ data model. Email
| messages are the one and only structured data format that has
| remained compatible in active use and extension for over half
| a century; you can literally take internet email messages
| from 01972 and load them into a mail client today (most
| easily by putting them into a qmail-style maildir) and
| everything will just work. This is largely a result of the
| decentralized extensibility properties of name-value pairs:
| mail clients just ignore header names they don 't understand,
| and they don't require header names that weren't originally
| present. This is also the basis of the extensibility of
| HTTP/1.0 and HTTP/1.1.
|
| It's also very similar to the data model of popular
| "semistructured" or "free-form" databases of the 01990s like
| askSam and Filemaker. Unlike RFC-822 and HTTP/1, but like
| Jevko, those systems support recursively nested data. askSam
| even used almost the same firstname[John] syntax.
|
| This data model does have a semantic mismatch with things
| like the rose-tree representation you describe at
| https://xtao.org/blog/rose.html, since it associates the
| rose-tree label with the first branch and the empty-string
| label with subsequent branches.
|
| If your audience is people like me, I think it would probably
| be worthwhile for you to spend some time up front describing
| the intended semantics of a data model, as I've attempted
| above, rather than leaving people to infer it from the
| grammar. (Maybe OCaml is not a good way to explain it,
| though.) You might also want to specify that leading and
| trailing whitespace in prefixes is not significant, though it
| is in the suffix ("body"); this would enable people to format
| their name-value pairs readably without corrupting the data.
| As far as I can tell, this addendum wouldn't interfere with
| any of your existing uses for Jevko, though in some cases it
| would simplify their implementations.
|
| ______
|
| Runnable OCaml code: type jevko = Jevko of
| (string * jevko) list * string (* XXX doesn't
| escape `[ `] `` *) let rec dump (Jevko(hdrs, body)) =
| hdr(hdrs) ^ body and hdr = function [] -> "" | (k, v)
| :: t -> k ^ "[" ^ dump(v) ^ "]" ^ hdr(t) let
| dict kvs = Jevko(kvs, "") let text s = Jevko ([], s)
| let v = dict ["include", text "author";
| "fields", dict [ "articles", dict
| ["", text "title"; "", text "body"];
| "people", dict ["", text "name"] ]]
| ;; print_endline(dump v)
| djedr wrote:
| I love your comment. Thanks for taking the time to look so
| deeply into this!
|
| I'll respond to the main points and then expand on the
| details later.
|
| I'm not sure what the authoritative source on S-expressions
| is (or even if there is one, which to me is a problem and
| part of the value proposition here), so I'll take R^7RS[1]
| as a reference.
|
| If you look at the formal definition there (chapter 7),
| it's significantly more complex than both Jevko and what
| you beautifully constructed here.
|
| And indeed, you have just freestyled a simplifed version of
| S-expressions (impressive!), but this is not the real
| thing.
|
| If you would keep refactoring it with the constraints I had
| in mind for Jevko, you'd eventually end up with Jevko.
|
| > though of course the name-value pairing is lost there
| (because S-expressions lack it).
|
| Indeed, and that's another part of the value proposition of
| Jevko. The grammar is designed purposefully to take
| advantage of natural syntactic name-value (prefix-subjevko)
| pairing tendencies.
|
| Which brings me to the next point.
|
| **
|
| You are absolutely correct that the abbreviated grammar
| matches the same strings, but doesn't have the same
| structure.
|
| *The correct grammar is the one in the specification*.
|
| I have shown the condensed version of it to illustrate the
| point that Jevko is indeed extremely simple. The single
| line captures all the essential elements and, again,
| matches the same strings as the full grammar.
|
| This is unlike the similar condensed grammar for
| S-expressions which you sketched out here. If you would
| continue, it would get significantly more complex before it
| matches the same strings.
|
| The OCaml type definition you wrote down should do the job
| of capturing the structure, although I prefer to name the
| elements (which may be not-so-convenient, depending on the
| language, so it's fine).
|
| **
|
| Indeed, I also think that this is an extremely useful data
| model. Thanks for pointing out the similarity to e-mail
| messages and other references which I'd love to dig into
| (please send if you have any links or resources about these
| databases).
|
| Thanks for all the pointers, I'll take them into account.
|
| And thanks again for your time.
|
| [0] this seems kinda official, but there is no single
| standard here: https://www.s-expressions.org/standards
|
| [1] https://standards.scheme.org/official/r7rs.pdf
| bokumo wrote:
| A minimal general purpose syntax for what?
|
| I followed the link and read the page, but I'm still not sure
| what the point is!
| saurik wrote:
| If it was "for" something wouldn't it no longer be general
| purpose?...
| eterps wrote:
| _> A minimal general purpose syntax for what?_
|
| Apparently for the same things as with XML/JSON/TOML.
|
| _> I followed the link and read the page, but I'm still not
| sure what the point is!_
|
| According to the website:
|
| _> It has no data types, no semantics, no underlying model of
| cons cells or anything similar. It's as close to pure generic
| syntax as it gets._
|
| _> So at the lowest level Jevko is a minimal formal
| specification for flexible trees of text._
|
| For its simplicity it seems quite powerful to me. I don't see
| it replacing JSON/markdown or even TOML. But is seems trivial
| to implement on very low-level, older, less bloated, or less
| powerful systems.
| djedr wrote:
| This is correct.
|
| Every time I've written or otherwise dealt with JSON/XML/etc.
| I wished I was dealing with something simpler, so I created
| it. If I had Jevko as a full-fledged alternative to JSON or
| XML, supported by tools, etc., I'd pick Jevko in a heartbeat.
| I like minimalism. Some people like it too, so perhaps Jevko
| can serve them well. There are many good reasons.
|
| > But is seems trivial to implement on very low-level, older,
| less bloated, or less powerful systems.
|
| Indeed, this is THE feature and a realistic application. I
| hope!
|
| For a naive unrealistic vision that I sketched out some time
| ago see:
|
| https://github.com/jevko/writing/blob/main/2022-01-13-vision.
| ..
| Nullabillity wrote:
| I wish we could stop making up new languages in 2022 that use
| unenclosed strings...
| User23 wrote:
| There are no strings in this language.
| Nullabillity wrote:
| The first example on https://jevko.org/ contains plenty of
| strings.
| User23 wrote:
| No, it contains a tree of text objects, which aren't
| strings. Now the language that the Jevko parser is written
| in will probably store that text tree using strings, but
| that's a statement about the parser implementation and not
| the formally specified Jevko language, which, as I said,
| has no strings. For example, a Common Lisp hosted Jevko
| parser implementation might choose to represent Jevko text
| objects as symbols.
| eterps wrote:
| Found an example where it is used as markup:
| https://github.com/jevko/jevkodom.js/blob/master/test.js#L6
| eterps wrote:
| If I understand correctly adding 'sub tags' requires an
| additional level of square brackets on siblings. I.e. when
| adding a bold tag on 'world!': p [Hello
| world!]
|
| Changes it to: p [[Hello ] b[world!]]
| djedr wrote:
| Nice find and you understand correctly! ;)
|
| This is one way. I've experimented with many others. Some
| semi-documented here:
|
| https://github.com/jevko/markup-experiments
|
| Some ways don't need additional level of nesting, but it's a
| trade-off.
|
| In the end something like the way you linked is my favorite,
| because it's very simple.
|
| It also gives you explicit control over your text nodes.
|
| Attributes can also be added in many ways, but I think I
| prefer them simply looking like child nodes, except with `=`
| appended to attribute name: p [class=[sth]
| [content]]
|
| Technically you can mix attributes and elements this way, but
| it does no harm.
| eckza wrote:
| If you see this, and you like it: Elm's HTML DSL is similar,
| and really nice.
|
| Every HTML tag (except for `text`) is an implementation of
| the base function `node`, which is: node :
| String -> List Attr msg -> List Html msg -> Html msg
|
| So in the base library, the `p` tag is implemented as
| p : List Attr msg -> List Html msg -> Html msg p =
| node "p"
|
| Which means that in Elm, the above content would look like
| this: p [] [ text "Hello ", b [] [ text
| "World" ] ]
|
| i.e., every HTML tag can accept as parameters:
| - A list of attributes - A list of child tags
|
| I've found it to be very elegant to work with; and with a
| good auto-formatter, it's really easy to build complicated
| HTML without losing your place and dropping a closing tag.
| djedr wrote:
| Certainly!
|
| The advantage of a Jevko-based format over the above is
| that it has the simplicity and elegance parameters (IMO)
| cranked up to maximum! ;)
|
| Now, for the tool support...
| Trufa wrote:
| Interesting, I'm wondering where something like this could be
| useful.
| eterps wrote:
| There are a couple of examples here:
| https://github.com/jevko/examples
| djedr wrote:
| Author here, thanks for posting this.
|
| If anybody has any technical questions or comments, ask away!
| ianbicking wrote:
| Seems like there's a bunch of problems:
|
| 1. It relies on later stages to do things like convert to native
| values or remove whitespace. As such an intermediary can't really
| "understand" the document very well. E.g., if you are using this
| syntax to express key/value then you can only know the keys if
| you know the whitespace rules that will be used to interpret the
| keys.
|
| 2. It reminds me more of XML than S-Expressions. If you are
| familiar with the ElementTree [1] representation of XML then
| these parsed examples become more familiar, just a prefix instead
| of suffix, and no attribute children.
|
| 3. Connecting whitespace and XML, I suppose this explains why so
| many XML formats ended up only using attributes for text values,
| and used tags and children only for nesting. Otherwise
| interpretation requires understanding how to interpret these
| mixed documents with unclear whitespace rules. But Jevko doesn't
| have attributes.
|
| 4. If an intermediary can't understand it then why define a
| syntax at all?
|
| 5. I don't see any escaping rules so including literal []
| seems... impossible? You can essentially deserialize the parsed
| value to reintroduce them, but that's weird and awkward, and only
| works with balanced brackets anyway.
|
| 6. No way to embed binary data, or otherwise uninterpreted data.
| JSON has the same problem. Base64 encoding things is yet another
| way in which the data is not interpretable by intermediaries.
|
| I think XML shows that "just strings" isn't fatal, and lots of
| things get jammed into strings in JSON too, you'll never be able
| to reproduce every type in a general purpose serialization format
| (I guess XML with namespaces attempted, but also clearly failed).
| So I can see a place for something that defines a tree where all
| leaves are strings. But this doesn't seem like a very clean way
| to do that.
|
| [1]
| https://docs.python.org/3/library/xml.etree.elementtree.html...
| djedr wrote:
| > 1. It relies on later stages to do things like convert to
| native values or remove whitespace. As such an intermediary
| can't really "understand" the document very well. E.g., if you
| are using this syntax to express key/value then you can only
| know the keys if you know the whitespace rules that will be
| used to interpret the keys.
|
| From the point of view of Jevko, handling whitespace is a
| higher-level concern. Indeed, you typically will need
| additional rules to communicate effectively with it.
|
| This is where you should specify a format (which should be
| standardized) that you use, e.g.:
|
| * https://github.com/jevko/easyjevko.lua
|
| Note: this is just a simple library I wrote recently that does
| the most straightforward thing imaginable. I haven't yet wrote
| a spec for the format.
|
| > 2. It reminds me more of XML than S-Expressions. If you are
| familiar with the ElementTree [1] representation of XML then
| these parsed examples become more familiar, just a prefix
| instead of suffix, and no attribute children.
|
| Yes, Jevko has the features of both XML and S-expressions (or
| neither, depending on how you look). It's supposed to be
| uniquely suitable for both markup and encoding of data/code in
| a simple way.
|
| > 3. Connecting whitespace and XML, I suppose this explains why
| so many XML formats ended up only using attributes for text
| values, and used tags and children only for nesting. Otherwise
| interpretation requires understanding how to interpret these
| mixed documents with unclear whitespace rules. But Jevko
| doesn't have attributes.
|
| A markup format built on Jevko can have the notion of
| attributes and rules for whitespace, e.g. see
| https://news.ycombinator.com/item?id=33334774 -- a nice thing
| about this particular format is that text nodes are explicitly
| specified, so you know exactly where your significant
| whitespace goes.
|
| The nice thing about attributes made with Jevko over XML
| attributes is that you could naturally make them extensible
| (which is a pain in XML -- if you want to turn your
| unstructured string value stored in an attribute into a tree
| you have a problem).
|
| > 5. I don't see any escaping rules so including literal []
| seems... impossible? You can essentially deserialize the parsed
| value to reintroduce them, but that's weird and awkward, and
| only works with balanced brackets anyway.
|
| That's not correct. There are only 3 special symbols
| (delimiters) and they can all be escaped. It's all in the
| specification.
|
| > 6. No way to embed binary data, or otherwise uninterpreted
| data. JSON has the same problem. Base64 encoding things is yet
| another way in which the data is not interpretable by
| intermediaries.
|
| Indeed, Jevko is not a binary format. Although I've been
| experimenting with binary equivalents, e.g.:
|
| https://github.com/jevko/binary-experiments
|
| At some point I might go forward with one, but that's not the
| focus right now.
| int_19h wrote:
| In XML world, XDM (XPath and XSLT data model) is the de facto
| standard on how to interpret whitespace.
| zokier wrote:
| I don't understand the whitespace handling here, it seems very
| underdefined?
|
| Taking the example from github page (abbreviated):
| first name [John] last name [Smith] address [
| street address [21 2nd Street] ]
|
| why isn't it equivalent to this json? {
| "first name ": "John", "\nlast name ": "Smith",
| "\naddress ": { "\n street address ": "21 2nd
| Street" } }
|
| somehow the whitespace is instead implicitly magiced away, but
| that isn't defined in the grammar anywhere as far as I can see?
| zokier wrote:
| Simple answer is that apparently the examples are false,
| feeding the above string to the JS parser spits out the
| following object: { "subjevkos": [
| { "prefix": "first name ", "jevko":
| { "subjevkos": [], "suffix":
| "John" } }, {
| "prefix": "\nlast name ", "jevko": {
| "subjevkos": [], "suffix": "Smith"
| } }, { "prefix": "\nis
| alive ", "jevko": { "subjevkos":
| [], "suffix": "true" }
| }, { "prefix": "\nage ",
| "jevko": { "subjevkos": [],
| "suffix": "27" } }, {
| "prefix": "\naddress ", "jevko": {
| "subjevkos": [ {
| "prefix": "\n street address ", "jevko": {
| "subjevkos": [], "suffix": "21 2nd
| Street" } }
| ], "suffix": "\n" } }
| ], "suffix": "", "opener": "[",
| "closer": "]", "escaper": "`" }
|
| tbh seems like a pita to clean up afterwards
| djedr wrote:
| Thanks for taking the time to look at this. ;)
|
| What you are getting is the plain Jevko parse tree.
|
| The examples explicitly talk about a format built on top of
| that. See this answer:
| https://news.ycombinator.com/item?id=33334314
| djedr wrote:
| Good question!
|
| Jevko itself, defined by the spec, is not the same as a format
| based on Jevko.
|
| If you use a plain spec-compliant Jevko parser you for this you
| indeed should have all whitespace included. Although it will
| look more like this: { subjevkos: [
| { prefix: "first name ", jevko: { subjevkos: [Array], suffix:
| "John" } }, { prefix: "\nlast name ", jevko: {
| subjevkos: [Array], suffix: "Smith" } }, { prefix:
| "\nis alive ", jevko: { subjevkos: [Array], suffix: "true" } },
| { prefix: "\nage ", jevko: { subjevkos: [Array], suffix: "27" }
| }, { prefix: "\naddress ", jevko: { subjevkos:
| [Array], suffix: "\n" } } ], suffix: "" }
|
| Now on top of that you can build a format that specifies what
| happens to whitespace, etc. Usually you'd want it trimmed if
| the prefixes mean keys in a map.
|
| The simplest format/library that does just that is this:
|
| * https://github.com/jevko/easyjevko.lua
|
| * https://github.com/jevko/easyjevko.js
|
| That will give you the JSON (or equivalent) that you expect
| (with autotrimmed keys).
| zffr wrote:
| If the Jevko spec does not handle white space and leaves it
| to clients, then it will cause issues for data interchange.
| In order for 2 different clients to read the same Jevko file,
| they will need to align on how they handle white space.
| djedr wrote:
| This is a valid concern, but on a higher-level.
|
| Jevko leaves whitespace handling not to clients, but to
| formats built on Jevko which would be the thing the clients
| typically use (rather than plain Jevko). Such formats of
| course should be clearly specified, which is something I
| hope to get around to soon.
|
| I already have a good candidate format for specifying
| first, which has 2 implementations:
|
| * https://github.com/jevko/easyjevko.js
|
| * https://github.com/jevko/easyjevko.lua
|
| For more details see also:
|
| * https://news.ycombinator.com/item?id=33335181
|
| * https://github.com/jevko/specifications/issues/1
| zokier wrote:
| Without any whitespace rules I wouldn't really call this
| human-readable format. The canonical serialization of a tree
| is without any whitespace and unlike json/xml/sexpr you can
| not reformat the document for human readability because any
| added whitespace changes the content.
| [deleted]
| djedr wrote:
| Indeed, in most of the formats I built on Jevko this is how
| it works.
|
| To use the nomenclature from the spec, if there are no
| subjevkos (subtrees) in a jevko (tree), then it is
| interpreted as a string (which is made from its suffix).
| [deleted]
| User23 wrote:
| I wondered this too. Magic behavior is the opposite of what I'd
| want from Jevko. At that point I may as well use s-exps.
| chriswarbo wrote:
| How does this relate to the (earlier?) tree-annotation/TAO system
| (seemingly from the same author):
| https://djedr.github.io/mirror/tao/tao.html
| djedr wrote:
| Nice find!
|
| Jevko is the evolved version of that (I'm the author).
___________________________________________________________________
(page generated 2022-10-25 23:01 UTC)