[HN Gopher] JSON Canvas - An open file format for infinite canva...
___________________________________________________________________
JSON Canvas - An open file format for infinite canvas data
Author : nickmain
Score : 435 points
Date : 2024-03-11 17:22 UTC (5 hours ago)
(HTM) web link (jsoncanvas.org)
(TXT) w3m dump (jsoncanvas.org)
| ramses0 wrote:
| The spec basically fits on a page:
| https://jsoncanvas.org/spec/1.0/
|
| Summary: "node: { type: ..., x/y/color }; edge: { from/to: ...,
| color/label/... }"
|
| Refreshingly simple, especially paired with their "gif of usage":
| https://obsidian.md/canvas
| jakelazaroff wrote:
| I really like the idea of a format for interchange between
| infinite canvas apps, but the preset colors and list of node
| types makes this spec feel strangely opinionated. You could build
| something like Kinopio [1] but not much beyond that.
|
| It looks really promising though! I'm definitely interested in
| seeing this grow.
|
| [1] https://kinopio.club
| iainmerrick wrote:
| Thinking about it, I'm not sure _any_ interchange spec would
| leave useful room for innovation. This isn 't like EPS or PDF
| or something where the kind of output is well-defined
| (printable graphics + text) and the innovation is in the
| editing interface. The innovation in canvas apps is surely in
| the semantics of the nodes themselves; but if you add some new
| node type with special behavior, how can you usefully export
| that to other apps? You could make it a black box that can
| round-trip safely, but that doesn't seem very useful. Maybe if
| it's something like an HTML embed or iframe?
| heleninboodler wrote:
| My very first thought about this format was that someone is
| definitely going to pervert it to be used as a graphics
| format, and for that it's woefully inadequate. I decided to
| hold my tongue since it's not the creators' fault that
| someone will misuse it, but I can totally see people adding
| optional ad hoc fields to nodes to allow for example all
| sorts of fancy line styles.
| cmgriffing wrote:
| I think a baseline spec would be really useful.
|
| I like to think of it like the unist ecosystem for ASTs.
| Unist provides a baseline spec that compatible tools can use
| to comb an AST. Then, specific AST tools like hast for HTML
| or sast for CSS/SCSS can add their own metadata on top.
|
| I'm imagining an ecosystem of "adapters" that would help you
| translate some of the metadata across providers.
| gcr wrote:
| It would be fun to go the postscript route and have the nodes
| be specified by some bytecode on some VM that lays the
| objects out on the canvas while specifying their editable
| properties.
|
| Then, conforming implementations could render any document
| just by following the instructions, while editors that
| actually understand them can provide their own high-level
| control.
|
| The trick is keeping it editable, which postscript doesn't do
| well.
|
| Example: if the language is strong enough to, say, implement
| force-directed node layout, an editor that doesn't understand
| it could still add nodes and they would move around according
| to the document author's wishes whereas perhaps the original
| editor might have more powerful editing capabilities.
| Caddickbrown wrote:
| Just realised this is by the Obsidian guys! Good on them!
| mock-possum wrote:
| Love obsidian and it's cute that the site itself is presented as
| a 'canvas' interface element, but...
|
| not a great experience on mobile (iOS/safari) - pinch to zoom and
| drag to scroll are both a bit busted, the zoom ui controls can
| disappear altogether, and there's no ui indication that the box
| on the right, that contains the details of the spec, is actually
| a scrollable container (ie no scroll bar visible)
| subtra3t wrote:
| Really? I used the website on Chrome on my Samsung Galaxy M12
| (a fairly average smartphone) and the experience was fine.
| regus wrote:
| What is the upside of using an infinite canvas? I tried playing
| around with them in the past but I think my brain's mental model
| of how to parse data is incompatible with "clutter" (for lack of
| a better term) of an infinite canvas.
| aragonite wrote:
| Check out this video from one of my favoriate infinite canvas
| softwares, especially beginning around 1:25 (which I'm deep-
| linking to):
|
| https://youtu.be/GblI7GI0jQ4?t=85
|
| (Alas, the original desktop app (iMapping, in Java) as shown in
| the video is no longer being developed, and now they are only
| developing the web app (Infinitymaps) only run in the browser,
| which imo is not the best fit for infinite canvas apps which
| can be _very_ resource-intensive)
|
| Also see the app author's own all-purpose mega-map that he used
| to organize everything:
| https://youtu.be/bTQWL5wmdZY?si=6VrnPIErOzasisEe
| DrDeadCrash wrote:
| Imo it would need to show varying levels of information
| density depending on scale. Otherwise, as another commenter
| stated it's overwhelming....
| allenu wrote:
| I've always wanted to build an infinite canvas idea to try it
| out, but whenever I've used implementations of it (such as in
| the Muse app), it just feels wrong. It feels like a beautiful
| way to interact with essentially fractal information, but in
| practice it doesn't quite work to me.
|
| I agree with you that I don't think it maps well to the mental
| model of the brain. Seeing the youtube video link in the
| sibling comment, when the user is completely zoomed out and can
| see everything, I just feel overwhelmed looking at it all.
| Maybe it's the nature of the boxes having different scales that
| you can't compare them as easily to each other compared to just
| a regular canvas where things just place in two dimensions.
| Each time you zoom into a canvas, the transition causes a lost
| sense of place and space, akin to walking into a doorway to a
| room and wondering why you walked there to begin with.
| theogravity wrote:
| It would be nice to see actual examples. We use an infinite
| canvas (https://switchboard.app), but it's hard for me to map the
| objects we have on our virtual-desktop-like-canvas with the file
| format described here.
|
| The items on our infinite canvas are more akin to a desktop app,
| where most objects are application windows.
| m12k wrote:
| > It would be nice to see actual examples
|
| If you click "Toggle output" on the linked page, you can see
| the code for the page itself
| theogravity wrote:
| Thanks. The available types definitely do not map to what we
| do. It seems the format is more suited for displaying DAGs.
| heleninboodler wrote:
| Suggestion: add some metadata at the top level, including a
| bounding box that includes all nodes so that you can encapsulate
| a canvas and include it within another context before having
| created all the elements within it. It's redundant information
| and it needs to be kept in sync with the nodes and edges, but
| it's very useful for applications working with your data.
|
| Also, this is not very json-ish, but optimizing your serializer
| so your metadata is always written first is pretty handy for
| embedding, since it allows you to use a pull parser and do useful
| things before the entire doc is parsed. (e.g. picture a huge doc
| being embedded and starting out as just a box and having its
| elements filled in async. You can't do that well if you don't
| know the bounding box ahead of time)
| iainmerrick wrote:
| This looks a little pre-1.0, it's quite short on detail. For
| example:
|
| _file (required, string) is the path to the file within the
| system._
|
| What kind of path, within what system? It's not clear that the
| 'file' type couldn't just be another kind of 'link'. If various
| fields like 'background' were defined to be URLs, that would
| offload a lot of complexity onto existing web specs.
| gvx wrote:
| Yeah, other details are missing too. For example
| _backgroundStyle_ :
|
| > _cover_ fills the entire width and height of the node.
|
| Does that work like the CSS _background-size: cover;_ or
| _background-size: 100% 100%;_?
|
| > _ratio_ maintains the aspect ratio of the background image.
|
| Does that mean CSS _cover_? _contain_? Something else?
| wongarsu wrote:
| Colors can be specified in hex format, e.g. "#FFFFFF". Six
| preset colors exist, mapped to the following numbers:
| 1 red 2 orange 3 yellow 4 green
| 5 cyan 6 purple
|
| I'm sure everyone will infer the same color codes here.
|
| Maybe the file format isn't meant to reproduce the exact same
| look in different software, but merely communicate user
| intent. Your guess is as good as mine.
| gcr wrote:
| I'd almost prefer string literals for color with constants
| like "red", "blue", etc. that compatible implementations
| could theme as they see fit (eg terminal emulators).
| Perhaps that's the intent behind the numeric constants but
| string literals would convey that better IMO
| adamfeldman wrote:
| This was released (and originally implemented) by Obsidian:
| https://obsidian.md/blog/json-canvas (see also
| https://news.ycombinator.com/item?id=39670684)
| mstipetic wrote:
| I would highly recommend using this plugin with it -
| https://github.com/rpggio/obsidian-chat-stream
|
| It has basically changed the way I interact with LLMs and
| research and plan things
| kepano wrote:
| Whoa! Didn't expect this to bubble up to the top of HN. Some
| context about why we created JSON Canvas:
|
| https://obsidian.md/blog/json-canvas/
|
| We just released it today, so this is still a very nascent
| project. A little over a year ago we released Obsidian Canvas.
| The .canvas file format has felt stable enough to give it a name
| and resources that other apps can freely use. See the original
| Show HN:
|
| https://news.ycombinator.com/item?id=34066824
|
| The spec is conservative, and definitely does not support many
| features canvas apps will want to implement (yet).
|
| The purpose of giving JSON Canvas a name and site is to encourage
| an interoperable ecosystem to grow around this format. We're
| definitely looking for feedback of all kinds!
|
| It's great to see all the suggestions already shared in this
| thread because it starts to provide a roadmap for how this could
| become a more useful format for other apps.
| remram wrote:
| Who was involved in this open spec? Or was it built by Obsidian
| with a hope that it works for everyone else?
|
| Which existing formats were considered before building your
| own, e.g. SVG/Excalidraw/draw.io/...?
| eichin wrote:
| Yeah, didn't see any discussion in the other thread, do you
| have any notes on why this isn't SVG?
| npunt wrote:
| It makes sense if you see Obsidian as the starting point -
| it's a document store. While other canvas products may be
| more graphics-oriented, Obsidian's is about laying out
| documents and objects and providing simple relationships
| between them. For this purpose, JSON's probably a lot
| easier to work with than XML/SVG.
| naikrovek wrote:
| SVG is great and you are the first person I have ever seen
| who would (and did) imply that SVG might be or would be
| better than JSON for this kind of thing.
|
| SVG _really_ stretches the "human readable" thing to its
| limit, and I personally would not adopt an SVG-like format
| for anything that isn't SVG.
| altairprime wrote:
| For those having difficulty viewing the spec at the linked
| jsoncanvas.org site, the spec/1.0 pane on the right has the
| scrollbar hidden; there's more content if you scroll that pane
| down.
| hiccuphippo wrote:
| Click the spec link, it shows the same content in a full
| page.
| desmondl wrote:
| I agree with a lot of comments that it's minimal, but in my
| opinion that is a good thing. I'm a big fan of Obsidian, and of
| the things I like about it is the data source is all markdown
| files. Markdown is meant to be very lightweight and portable,
| and overcomplicating it will limit adoption and extensibility
| (imagine markdown vs pdf).
|
| JSON Canvas seems to follow in that spirit by being very
| lightweight, so a lot of implementation details (i.e. how are
| files rendered, what file formats are supported, etc), edit
| tags, etc) are left open to implementation.
|
| Markdown and JSON are meant to be non-opague file formats that
| prioritizes portability and human readability over other
| features. An application format like Sqlite has a lot of
| benefits over markdown, but it loses the benefits of text based
| formats like being compatible with git and is less portable.
|
| What I would like to see is a convention for extending the node
| and edge definitions, similar to frontmatter in markdown files-
| something that is not _required_ for basic rendering but is a
| nice-to-have for applications to consume) - that way
| portability between apps of varying complexity can be maximized
| while still allowing for more complex features that some apps
| might implement. Markdown has the benefit of supporting
| extensions (for example like tables in GFM) - apps that are not
| compatible can still render the unsupported markup. But there
| should be an explicit way to extend open JSON formats.
|
| Some feedback off the top of my head and from reading the
| comments:
|
| 1. *Specifying node ordering*. Obsidian seems to just send
| whatever is last touched to the top, but this makes a common
| feature in many apps (move up/down/back/front) more difficult
| to impement
|
| 2.*More explicit definition for line shape*. Adding a way to
| "bend" a line in a specific way. Useful for creating more
| complex diagrams.
|
| 3. *Relations between nodes*. Group nodes contain child nodes,
| but the spec doesn't specify how the child nodes are defined. I
| would expect it to have a `children` property to nest nodes.
| Obsidian seems to implicitly link nodes to groups based on
| whether their bounds intersect. This makes it difficult to
| implement some common features:
|
| a. nodes that exist outside of the bounds of its group, for
| example a node that "floats" just outside of the edge of the
| group's borders.
|
| b. nodes that are not part of a group even though it exists
| within the bounds of that group.
|
| There are many different ways for a canvas app to extend the
| spec to implement those features, but it seems like something
| that should be defined in the spec to maximize portability
|
| 4. *Extensibility.* Either explicitly support or provide a
| standard for defining more styles for nodes and edges, such as
| stroke width, stroke style, rotation, etc. It seems like
| "color" should be a part of this as well, rather than being an
| explicit property of a node.
|
| 5. *Embeds.* Supporting "embeds" as a node type. I even think
| the "file" node should be redefined as `embed` with a `uri`
| property to support different schemes (`file://`, `oembed://`,
| `https://`) and maybe a `mime-type` (`text/markdown`,
| `image/webp`). The file node's "subpath" property seems to be
| only relevant for markdown files, so I think that should be an
| extension rather than an explicitly defined.
|
| 6.*YAML* :) (Should just seemlessly convert from json, but yaml
| is more readable than json)
|
| Being able to design standards that evolve over time and making
| tough decisions about what what to make explicit and what to
| leave implicit is a skill I want to improve on as a developer
| this year. Does anyone have any resource recommendations or
| best practices to recommend for me to research?
| exceptione wrote:
| > 6. YAML
|
| Please don't, it has one of the most confusing syntax out
| there with lists and maps, and it won't do well for parsing.
| desmondl wrote:
| I haven't had any issues with yaml in markdown frontmatter
| or openapi specs. What kind of issues do you see with list
| and maps that make you against yaml? I agree that for
| computers and consistency json is preferred. I already use
| a linter for my markdown files so I would do the same with
| yaml to keep lists and maps consistent
| cstrahan wrote:
| YAML is kind of like C++:
|
| > You like C++ because you're only using 20% of it. And
| that's fine, everyone only uses 20% of C++, the problem
| is that everyone uses a different 20% :)
|
| https://eli.thegreenplace.net/2009/10/17/the-c-bashing-
| seaso...
|
| The YAML footguns are too numerous to reproduce here, so
| here are some sources:
|
| https://stackoverflow.com/questions/3790454/how-do-i-
| break-a...
|
| https://www.arp242.net/yaml-config.html
|
| https://noyaml.com/
|
| YAML isn't terrible if you only ever have to read what
| _you_ wrote. Now consider that there are 63 different
| ways to write multi-line strings in YAML -- how many of
| those have you committed to memory? Yeah... now throw
| 10-100 developers into the mix, each with their own
| favorite alternative syntaxes -- good luck making sense
| of your YAML.
| desmondl wrote:
| Yeah true, I'm starting to remember the headaches with
| yaml when I was using kubernetes or cloudformation....
| bbkane wrote:
| Point taken, but you can mitigate a lot of this with
| yamllint.
| exceptione wrote:
| In the past I had to craft yaml files. Sometimes I needed
| quotes for a string, sometimes I had to put in a dash in
| front of a key, or just not. You basically needed to have
| the whole schema in your head.
|
| There can only be so much nesting before you lose track
| of what item belong to which parent. Copying some yaml
| structures over to another level requires care, as the
| result might look correct, but the white space parser
| thinks otherwise.
|
| I have lost hours of debugging yaml files when a dash was
| missing somewhere or when I needed one more leading
| space. The parser accepts it happily, but half of the
| typical javascript programs will only detect things are
| wrong when it has already executed on half of your spec.
| The other half will just run with input that wasn't
| intended that way.
|
| I remember writing artillery.io test specs where all
| those problems pop up.
|
| _Now the good thing from JSON is JSON Schema_. The
| latest spec allows you to specify quite advanced
| validations. Yaml has no such thing.
|
| As to your remark: Yaml for front matter is defensible,
| as you dont have deeply nested structures. Though, as an
| obsidian user you want to make sure your front-matter is
| conforming to your own schema. That would require writing
| a json spec and then have your yaml internally converted
| to json before handing it over to the validator.
|
| A spec is worthless if you cannot validate against it.
| Json and xml have a good story there. I concede that yaml
| is more human-readable than json without an editor.
| Correctness is the holy grail though.
| chatmasta wrote:
| > Markdown and JSON are meant to be non-opague file formats
| that prioritizes portability and human readability over other
| features
|
| I don't think human readability is a critical feature of JSON
| at this point. If that's your priority, you can use YAML.
| Readable JSON is nice because for small files you can read or
| edit small sections of it, and it's easy to debug when
| manipulating it with machine code. But there are plenty of
| cases where a huge JSON file is still useful even if it's
| barely human readable.
|
| My heuristic has always been: use YAML if you expect humans
| to create the file (or maintain large chunks of it),
| otherwise use JSON. For example, Kubernetes config is YAML
| because humans create it from scratch, and it would suck to
| do that with JSON. Whereas package.json is JSON because
| machine code initializes it and humans only make minor edits
| to specific fields.
|
| In the case of this canvas format, I wouldn't expect humans
| to create the file from scratch, so use JSON over YAML. Then
| the question is, will humans even care about _reading_ the
| raw JSON? Probably not. So why not use something like SQLite
| or Protobuf? The most compelling reason would be that humans
| writing code to interface with the format can use parsing
| tools from their language 's standard library.
| desmondl wrote:
| Yep, I think the compelling reason of humans writing code
| is key here. SQLite would make it less accessible for
| people to write external tooling to integrate with an
| obsidian vault. There are lots of existing and open that
| support diffing/parsing/syncing/manipulating json, while
| with sqlite you have to not only know sql but support
| another application's database schema, which third party
| developers are less likely to do
| naikrovek wrote:
| > I don't think human readability is a critical feature of
| JSON at this point. If that's your priority, you can use
| YAML.
|
| Wow you have kinda lost the plot on a few things.
|
| JSON was designed to be human readable and writable. YAML
| was designed to be a human readable format for the
| automated interchange of data between automated systems.
| Human writability was neither a goal for YAML nor its
| intended use. Like everyone else on the frakking planet,
| you've misunderstood what YAML was intended and designed
| for. YAML was never intended for human-written
| configuration storage, which is what everyone used it for
| the instant after they became aware of it.
|
| YAML can bite you very hard if you misunderstand it. JSON
| is simply invalid if you misunderstand it when writing it.
|
| If you don't need human readability, use a binary format.
| Binary formats are _so freaking fast_ compared to literally
| any structured text format, past, present, or future. High
| speed and low latency matter and binary formats make both
| of those easier.
|
| If you need to inspect the binary data, write a viewer
| using the code you use to read it. It's a lot simpler than
| people believe it to be. I find Protobuf to be more of a
| hassle than writing the code myself, and protobuf is very
| easy to use, and I'm quite a moron. Binary stuff is not
| hard.
| louthy wrote:
| > I agree with a lot of comments that it's minimal, but in my
| opinion that is a good thing
|
| The purpose of a spec is to specify, and if you don't specify
| and leave things open to interpretation, then that completely
| defeats the purpose.
|
| Anybody who's worked with a poorly defined spec knows exactly
| how bad this can be. A good example would be the shambles
| that is the HL7 spec used in healthcare.
|
| A former colleague had a phrase for this: "once you've seen
| one HL7 message... you've seen one HL7 message". Which really
| highlights the issue of a standard that's open to
| interpretation.
|
| The issues raised (in the comments here) seem to hint at a
| lack of specificity. That is something that they should
| really look at improving.
|
| I think overall any group that tries to come up with a
| standard that can unify a field should be lauded and
| supported. But perhaps calling this a 0.1 release, and taking
| the feedback on board, would be the best way forward.
| lovasoa wrote:
| As a maintainer of an open source JavaScript infinite canvas
| application [1], I was very interested, and now I am a little
| disappointed. The set of supported objects on the canvas is quite
| minimal.
|
| [1] https://github.com/lovasoa/whitebophir
| apitman wrote:
| It's an early revision. Now sounds like the perfect time for
| you to share your experience and suggestions to improve it.
| Retr0id wrote:
| If ids are unique strings, then why are `nodes` and `edges`
| arrays, as opposed to id-string-keyed maps?
| Retr0id wrote:
| A couple of other nit-picks:
|
| - I'd like to see the `file` node have a mime type field.
|
| - Why limit links to URLs, as opposed to URIs?
|
| - You might want to put a version field in the top level!
| eropple wrote:
| This is always a tough spot in JSON, I think. You want the `id`
| in the node or edge object, but you also want uniqueness. I
| don't think there's a great way to get this _any_ be able to
| just `JSON.parse()` and go without further fixup.
| Retr0id wrote:
| What do you concretely gain from embedding the id within the
| object?
| Macha wrote:
| You have a reference to an object, maybe found by filtering
| the list based on co-ordinate intersection, and you need
| its ID to put somewhere else (e.g. in the "to" field of an
| edge). Sure, all your find/filter methods could return (ID,
| object) or just return an ID and make you do a second
| lookup to find the object, but at the kind of object counts
| we're talking about for this implementation, a small amount
| of extra memory to have the ID as a field in exchange for a
| nicer API is a reasonable tradeoff.
| desmondl wrote:
| An object in JSON is meant to be unordered [1], so arrays are
| better if you want order to be preserved.
|
| [1] https://www.json.org/json-en.html
| Retr0id wrote:
| Sure, but why do you want order to be preserved? The spec
| never says the order of nodes is pertinent. Is there
| undocumented z-index behavior?
| Retr0id wrote:
| Looking at their reference impl, it does seem that array
| index implies z-index, in which case fair enough, I suppose
| an array does make the most sense - they should probably
| document the z-indexing though!
| desmondl wrote:
| Sometimes you have nodes that overlap each other, so you
| want to control whether or not a node is in front of or
| behind another node.
|
| Though yes, they could have explicitly defined a z-index or
| defined a convention on how the ordering should work (first
| nodes top and last nodes bottom or vice versa?). It's
| interesting to think about the trade offs between
| explicitly defining these things vs. leaving the
| application to implicitly make the choice. JSON Canvas
| seems to be designed to use in tandem with markdown files
| in a note taking app, so it makes sense why they opted for
| the more implicit design to be similar to markdown
| tiborsaas wrote:
| I really like that you commit to keep this stable and open.
|
| Do you plan to make the TypeScript definition part of this new
| site?
|
| https://github.com/obsidianmd/obsidian-api/blob/master/canva...
|
| For me it's easier to read TS format.
| egeozcan wrote:
| I didn't know about this and made ChatGPT prepare me a
| typescript definition form the markdown:
| https://gist.github.com/egeozcan/27db06f6771dcf214f0f92bce8c...
| :) Perhaps I shouldn't be too lazy to google things.
| raggi wrote:
| I would love to see this and excalidraw converge feature & spec
| wise.
| desmondl wrote:
| Any reference for obsidian's specs? The closest thing I could
| find is this: https://docs.excalidraw.com/docs/codebase/json-
| schema but it seems to be really minimal and doesn't go into
| detail on what properties belong in elements
| dekhn wrote:
| Is this infinite canvas similar to QGraphicsView?
|
| I've been looking for a good equivalent to QGraphicsView and HTML
| Canvas is not it (HTML Canvas is a raster image, QGraphicsView is
| a size-independent fixed-sized, scrollable canvas with indexed
| objects).
| tonyarkles wrote:
| Would it be possible to bang SVG into what you're looking for?
| It's a little bit quirky but I've used it to draw some pretty
| wild diagrams with a small amount of JS for pan and zoom.
| CubsFan1060 wrote:
| Is there any control over the routing of the edges?
| ianbicking wrote:
| There's some questions about interoperability, which I imagine to
| be hard since any canvas app involves viewing and editing and
| probably has unique entities and approaches which other apps
| wouldn't be able to handle.
|
| BUT, I imagine this being more useful for creating non-frontend
| tools. For instance a server that returns subsets of nodes for a
| particular viewport. Or something that may index nodes, or
| produce search results. Or tools that simply generates canvases
| from other data as a one-way operation.
| treflop wrote:
| Since a whiteboard is much more a visual thing, I imagine the
| spec should spend a lot of space defining the visual elements
| like specifying the control points of the Beizer curves because
| where a line is drawn and what it overlaps matters a lot on a
| whiteboard.
|
| But to be a total downer, this spec looks like an extremely
| rudimentary graph file format, of which there are already like a
| hundred and all define more visual aspects than this spec.
| xamde wrote:
| From my Analysis, GraphML, although specified in XML, seems to
| be one of the most widely used exchange formats, especially
| with the yWorks extensions for yEd.
| asa400 wrote:
| I've worked a few jobs now where application data was stored in
| text files of various kinds (homegrown as well as well-defined
| formats, JSON included) and it pretty quickly becomes a mess when
| you start talking about modifying it over time, evolving its
| schema, validating it in the face of end-user edits, ensuring
| threads don't write to it concurrently, etc.
|
| This strikes me as _exactly_ the type of application data that
| would benefit from being represented in SQLite. Of course, JSON
| is a `JSON.parse` away, but now you 're building your
| own...everything else. Storage/retrieval, validation,
| querying/filtering/indexing, graph traversal, etc. It's all
| yours.
|
| There's so many benefits to building this kind of thing in
| SQLite. You get data validation, atomic transactions in memory
| and on disk, a high-level query interface, lazy loading (i.e.,
| only load nodes at most 2 edges away), triggers (when I delete
| this node, automatically delete edges that point to it), and a
| stable on-disk format, to say nothing of the fact that SQLite
| itself is just about the most stable software there is.
|
| By the way, no disrespect to JSON Canvas, it looks like good
| work, just trying to offer the perspective of someone who has
| done stuff like this in the past.
| simonw wrote:
| I'm usually the first person to suggest SQLite for just about
| anything, but in this particular case I do feel like JSON is a
| better default format.
|
| Interacting with SQLite from different programming languages is
| easier than most other formats, but you still need a SQLite
| binding. They're available for every language but that's still
| a not-completely-trivial dependency.
|
| I expect most tools that people build against JSON Canvas will
| run in a web browser. Adding SQLite as a dependency means you
| need SQLite running in WebAssembly - totally possible, and even
| officially supported these days (the SQLite team run their own
| WASM builds now) but still a sizable piece of extra complexity
| over just using JSON.parse(...)
|
| Also: SQLite files aren't very easy to diff, so they're not
| great for collaboration in version control. JSON is better for
| that.
|
| I'm 100% with you on the schema changes and versioning
| challenge. The best way to address this IMO would be for the
| spec to include a top-level "version": key which indicates the
| version of the spec that a file was created against.
|
| Handled carefully and introduced right at the start of the
| project this could ensure an ecosystem grows up around the
| standard such that older spec versions can always be opened by
| newer implementations, and any implementation can fail-fast if
| it is given a file that it doesn't yet know how to handle.
| asa400 wrote:
| Valid points and I generally agree! I think you're probably
| right that the SQLite dependency is possibly too much for
| some applications to pull in, though I will ask why we're so
| often willing to pull in more or less anything else
| regardless of weight (DOM manipulation, state management,
| animation, etc.) but a robust data layer is often a bridge
| too far. I wish I knew why this is, as it seems to be one of
| the larger cultural differences between folks who work mainly
| "on the backend" and "on the frontend".
|
| > Also: SQLite files aren't very easy to diff, so they're not
| great for collaboration in version control. JSON is better
| for that.
|
| Yeah, if we're talking about diffing the literal file itself,
| then that changes things. At that point, we're not just
| talking about a storage format, we're talking about
| interchange as well. In that case I'd ask - of course this is
| application specific, not general - what data are we putting
| on the wire? Where does that data live?
|
| For example, if the main state state lives in a your browser
| instance, and you ship updates (i.e., "CREATE", "EDIT",
| "DELETE" or some such) back and forth between collaborators,
| then diffing the state of whatever you have is fairly easy,
| SQLite or JSON or whatever else. But if we're shipping the
| actual file itself over the wire or attempting to version
| control it, then you're absolutely right and diffing the
| SQLite file is inferior.
|
| There are some interesting tradeoffs in this space. This is a
| fun discussion!
| desmondl wrote:
| I think you're talking about the trade offs between
| supporting features like "DOM manipulation, state
| management, animation, etc." and "shipping updates" out of
| the box, versus only storing the data as simple files and
| leaving everything else to the implementation.
|
| Sqlite as an application file format is great [1], but for
| a knowledge base / note taking app the benefits are not
| worth the tradeoffs in my opinion.
|
| Sqlite is more performant more performant and provides lots
| of built-in features. However, most note taking users do
| not have enough notes or files to benefit from that
| performance. Sqlite will also lock the user into the
| application, whereas a "pack of files" can be used in the
| shell as a text editor. Using markdown files + a open json
| format has the benefit of being supported by multiple
| applications (e.g. sometimes i open my obsidian vault in
| vscode), while a sqlite database would need a proprietary
| schema coupled with a single application
|
| I prefer an open file format that isn't tied to a vendor. A
| "data bridge" might handle syncing and diffing more
| efficiently than plain files, but it is still tied to the
| vendor. For example, I prefer not to pay for Obsidian Sync,
| and I'm able to use a git plugin and storing my files on
| nextcloud to sync between my devices. This leverages
| existing tech without having to implement features from the
| ground up
|
| [1] https://www.sqlite.org/appfileformat.html
| asa400 wrote:
| > I think you're talking about the trade offs between
| supporting features like "DOM manipulation, state
| management, animation, etc." and "shipping updates" out
| of the box, versus only storing the data as simple files
| and leaving everything else to the implementation.
|
| I'm not sure I understand. Can you clarify?
|
| > Sqlite is more performant more performant and provides
| lots of built-in features. However, most note taking
| users do not have enough notes or files to benefit from
| that performance.
|
| For a graph with lots (thousands+) of nodes/edges, SQLite
| is probably capable of being more performant than a JSON
| file, depending on whatever specific kind of performance
| we're measuring. That said, to me, the most interesting
| thing that SQLite gives for applications when compared to
| flat files is data integrity, via transactions and
| schemas/constraints. Performance is nice but not even
| close to the most interesting thing about SQLite in most
| applications. Performance has never been the reason I've
| chosen SQLite over flat files for my applications.
|
| > Sqlite will also lock the user into the application,
| whereas a "pack of files" can be used in the shell as a
| text editor. Using markdown files + a open json format
| has the benefit of being supported by multiple
| applications (e.g. sometimes i open my obsidian vault in
| vscode), while a sqlite database would need a proprietary
| schema coupled with a single application
|
| In a sense you're right about this! I'll grant it's
| easier to open a JSON file in vscode and edit it if you
| already know vscode and JSON. That said, SQLite is in the
| public domain with a well-defined, stable format and
| there are countless free and open source database
| editors/viewers out there.
|
| SQLite is self-describing, also. You open `sqlite3` and
| type `.schema` and it shows you the database schema. You
| enter a query and get some results. It's all right there.
| So, while a database might have a schema that was
| designed for a particular application, that doesn't mean
| you as the end user can't tinker with it, and the number
| of people who know SQL is rather large.
| chrisweekly wrote:
| Crucial context: its provenance is embedded infinite canvas in
| Obsidian (amazing markdown-based notes app++), which supports
| JS but has no external datastore per se. SQLite is fantastic,
| but inappropriate for this use case.
| asa400 wrote:
| Oh I read the post and a few of the other linked posts so I'm
| aware. If we're storing a file, we're already storing a file
| of some kind, so we're just talking about what kind of file
| it is. I'm just talking about a different set of tradeoffs
| one can make if that file is something other than JSON.
| LordDragonfang wrote:
| >If we're storing a file, we're already storing a file of
| some kind
|
| It's not just a file "of some kind", it's a _text_ file.
| That 's one of Obsidian's key selling points - that a vault
| is simply a collection of _text_ files. The "text" part is
| important to both Obsidian's philosophy and the majority of
| its users.
| asa400 wrote:
| Sure! And in a lot of ways this is about values more than
| technology.
| toddmorey wrote:
| Agree that SQLite is a great local format for this sort of
| thing, but seems to get tricky when trying to sync across
| clients without conflicts. I've seen most CDRT schemes working
| with json documents rather than SQLite, but curious if there's
| solid conflict-free syncing out there for SQLite files being
| used inside applications?
| apitman wrote:
| SQLite is awesome, but it's still an order of magnitude more
| complex than JSON. If it wants to be the one file format to
| rule them all, we're going to need high quality and heavily
| used implementations in most languages.
|
| Adding C sqlite to a golang project adds a significant hit to
| build times and cross-compilation/static linking complexity[0].
| When I looked into the native Go implementations of sqlite I
| came away with the feeling it wasn't worth the tradeoffs
| compared to using the C version, but now I still have to deal
| with the issues above.
|
| I haven't looked deeply into how sqlite works, but my instincts
| tell me the reason we don't have high quality implementations
| in every language is because it's actually too complex to treat
| as a protocol.
|
| I would love to see something fill the void between plain text
| and sqlite.
|
| [0]: https://www.arp242.net/static-go.html
| exceptione wrote:
| SQLite has no real types. Use SQLite if data quality is of no
| concern. The applicability domain of SQLite is far smaller than
| people think.
|
| If you want to improve on JSON, you would have to go into an
| other direction. Maybe something like postgis would be helpful
| for extremely large canvasses.
|
| JSON Schema is pretty powerful by the way. Checkout the
| documentation. SQLite is absolutely no match there. What Sqlite
| could bring is speed, but I dont see how in the contxt of
| canvas it would be of any help here.
| asa400 wrote:
| SQLite's types are quite loose by default, no doubt.
|
| There is a somewhat recent STRICT mode that strengthens them:
| https://www.sqlite.org/stricttables.html
| samatman wrote:
| Exactly so. SQLite also has a rich collection of CHECK
| constraints which can raise errors if data is not to your
| liking in some fashion, this includes validating JSON. Not
| a JSON schema, admittedly, although (just like for
| Postgres) this is available as an extension.
| https://github.com/asg017/sqlite-jsonschema
|
| Saying that SQLite doesn't have "real types" is simply
| false. If one doesn't want to learn how to use a tool,
| blaming it for that failure is poor form.
| exceptione wrote:
| The author of SQLite is quite open about it. The lack of
| typing has been part of the design from the beginning.
| Sqlite has hardly any types: INT
| INTEGER REAL TEXT BLOB
| ANY
|
| Of course one can program all kind of check constraints,
| like one can program all kinds of value validations in
| javascript.
|
| Unfortunately, that is not the same as typing. Sqlite
| lacks typing because, as the sqlite author explains in
| the docs, flexibility is the goal. He continues with "But
| other developers are aghast", and so strict tables where
| born, but you can clearly see this cannot overcome real
| concerns. Try to look for the DATETIME datetype in that
| list.
|
| Deep bow to sqlite, its design goal was to be the ini
| file replacement and it has outperformed itself on that
| one.
|
| Thanks for the extension link. Although constraints are
| not reuable type definitions, they would still be helpful
| in this context. Pity that json doesn't have a type for
| dates, one has to rely on string formats: https://json-
| schema.org/understanding-json-schema/reference/...
| simonw wrote:
| It would be neat if the README or spec included links to some
| real-world examples of files - would be easier to start playing
| with building simple tools on top of this if there was already an
| example file to start experimenting with, without me having to
| learn Obsidian first.
|
| UPDATE: Figured out how to create one:
|
| 1. Install and then open Obsidian
|
| 2. Click the "Create new canvas" icon - third down of the icons
| on the left
|
| 3. Add some stuff to the canvas - I double clicked to create a
| few boxes, put some text in them and then dragged lines between
| them
|
| 4. In the ... menu on the top right click "Reveal in Finder"
|
| You can then open the file it reveals in a text editor to see the
| JSON Canvas format.
| jakelazaroff wrote:
| If you hit "toggle output" on the bottom right, it shows a JSON
| Canvas representation of the page's content.
| simonw wrote:
| Is that an icon? I can't seem to find it (Obsidian 1.4.16 on
| macOS)
| jakelazaroff wrote:
| Sorry -- I mean on the bottom right of
| https://jsoncanvas.org!
| politelemon wrote:
| > The JSON Canvas format was created in hopes of providing
| longevity, readability, interoperability, and extensibility to
| data created with infinite canvas apps.
|
| If I'm reading between the lines, this is only supported by
| Obsidian (as it's by Obsidian)? Considering the complexities and
| 'malleability' of infinite canvas tools, it would have been
| prudent to have involved or approached some of the major players
| in this space, like Excalidraw, Draw.io, Microsoft, Figma. Or at
| least started at version 0.1 and once it gained a wider
| consensus, release 1.0.
| wongarsu wrote:
| Based on the title I would have expected this to be about
| describing images on an infinite canvas, where different parts
| have different zoom levels.
|
| This is instead about whiteboard-style graphs. Which is useful,
| but I find the branding "An open file format for infinite canvas
| data" to be confusing. Nothing in it implies whiteboards or
| graphs to me. The fact that the canvas is infinite doesn't even
| have an obvious influence on the file format, apart from the
| absence of a canvas size property.
| airstrike wrote:
| This is neat, I'll definitely try it out. I do have a related
| question: what are people using to implement high performance
| canvases on React these days?
| catapart wrote:
| Oh sweet! I was in the middle of building my own version of a
| node-graph component (ref:
| https://github.com/catapart/Magnit.NodeGraph), but the
| `canvas.js` implementation from this JsonCanvas repo is exactly
| what I was trying to build. I really, really like how elegantly
| uncomplex it is. Could not have been more exactly what I was
| looking for. So thanks for this!
|
| As far as the spec, I don't really like the idea of forcing well-
| known types for the nodes. A generic spec should allow for
| entirely generic nodes that can represent themselves to consuming
| functions with a 'type' property as a key, as well as arbitrary
| data types linked to arbitrary nodes. For instance: one of my use
| cases is an 'addition' node, which would take two number values
| and produce a number value. This node would also use an entrance
| execution pin as well as an exit execution pin.
|
| If the spec were to include a 'pin' data type and capture the
| type keys and labels for pins, those pins could be stored as a
| list on the node. Then, the type property could just tell the
| executing context how to route the node data and the pin
| properties would bring type safety to the functional inputs
| passed to the mapped function.
|
| Anyway, I assume all of that is out of scope for initial
| offerings, but that's my two cents on a generalized node spec.
| Regardless, thanks again for the sweet, simplistic node graph
| implementation!
| Macha wrote:
| One of the big reasons I use Obsidian is the data portability -
| while it provides some nice enhancements, if Obsidian went evil
| tomorrow and I had to use my notes in VS Code tomorrow, then even
| reduced to just "a folder full of markdown", I'd get 90% of the
| value without relying on any convoluted importers/exporters.
|
| I've stayed away from their Canvas feature largely because it
| is.. not that. Not because the Obsidian developers have kept it
| locked down in some crazy proprietary format, but like a JSON
| file representing the canvas is pretty useless without something
| to interpret it and these days Obsidian is still the only
| implementation.
|
| So I kind of hope this takes off. Having a second source
| available would make me feel a lot more comfortable trying out
| the canvas feature.
| chatmasta wrote:
| I've stayed away from their Canvas feature because the
| Excalidraw plugin for Obsidian is unbelievably good. I love it.
| skadamat wrote:
| Curious to know which projects & companies were involved in
| creating this. Ideally an open standard has a bunch of folks
| involved & informed of the design process so we get a design that
| lasts!
|
| Braintree payments did this well when they were still a startup.
| They had to collaborate _with_ their direct competitors to create
| a standard for the entire POS induustry.
|
| What prevents us from doing this here as well?
| gcr wrote:
| This is a great idea. For now though, the spec is under-specified
| and ambiguously terse. A few points that could be clarified:
|
| - How do coordinates work? Does +Y point up (OpenGL) or down
| (web)? Is the origin meaningful? What are the units - how does
| this interact with font sizes? High-DPI displays?
|
| - What's the difference between a file node and a URL node
| linking to file://./? Are files supposed to be transcluded? What
| filetypes are allowed? The home page seems to have an image --
| was this done using a file node or a markdown node with an <img
| /> element?
|
| - What HTML tags are allowable in markdown? Is JavaScript
| allowed?
|
| - Why does the group node allow a background image? If both image
| and color are specified, which takes precedence? How are children
| of the group specified?
|
| A couple feature requests for extensibility or interoperability
| with Excalidraw and TLDRaw and friends: drawings / scribbles,
| predefined shapes like circle or rectangles, ability to specify
| fill style, edge width, transparency, ability to embed images,
| more detailed placement for connector start/end points, etc.
| curtisblaine wrote:
| What I would expect from an infinite canvas app would be a
| streamable data structure, that can be rendere on demand on a
| "virtual" canvas - let's say you have a window from 0,0 to
| 100,100 - you can query the data structure to give you only the
| nodes in the viewport + the nodes connected to them.
| mywacaday wrote:
| FYI only the spec canvas is black on chrome mobile with dark mode
| enabled.
| apitman wrote:
| Obsidian is one of the few closed-source applications I would
| consider relying on, due to their commitment to building around
| simple, open file formats.
|
| Sure, they could screw me over and start charging absurd amounts
| of money for their app, but high quality open source alternatives
| would pop up immediately.
|
| Meanwhile, as long as they _don 't_ screw me over, it's unlikely
| an open source alternative is going to be able to catch up to a
| profitable business that keeps their users happy.
|
| It's an interesting approach, focused on incentive alignment,
| which is the best way to ensure quality long term.
| desmondl wrote:
| I completely agree. Even if they completely tank I can open my
| obsidian directory in a text editor or command line and still
| use it. I would still have access to features that are common
| in other apps like full text search or plain file sync.
| Attachments are just files in the filesystem that can be opened
| in any image viewer. Basically if i can't use obsidian anymore
| i can still use my notebook and take notes without implementing
| or finding new software
| chatmasta wrote:
| I love Obsidian. I wish they would consider open sourcing the
| application. It doesn't even seem in conflict with their
| monetization plans, because they're already distributing the
| app for free, and making money with things like "Obsidian
| publish." They've got enough critical mass and a sufficiently
| thriving ecosystem of community plugins that they could only
| stand to benefit from open sourcing the core app.
|
| See Mattermost for an example of a similarly positioned product
| that is fully open source.
| ivanjermakov wrote:
| Actual spec: https://jsoncanvas.org/spec/1.0/
| RyanHamilton wrote:
| I would like to take this opportunity to implore everyone to not
| invent a new format. Avoid it at all costs. First try to follow
| an existing standard or say that you will support a subset. When
| you invent a new standard, you start small and say oh users just
| need to know this and look it's so simple. Then you find all the
| edge cases, add all the features and voila you have created a
| half baked implementation of an existing standard. Only if you
| can explicitly say many reasons why you haven't used standard X
| or Y create something new.
|
| To take this specific example, some of it feels very similar to
| HTML. label, links, sections, groups, anchors, background fill
| options...I would have been tempted to define as a subset of HTML
| that is supported. Then if I wanted JSON, say how JSON maps to
| HTML. voila suddenly everything is standardly named and
| creatable. This means backgroundStyle is replaced by background-
| size = cover or contain. It means that those six preset colors
| are replaced by all HTML standard colors. Voila no one needs to
| learn different concepts or definitions. Try that existing
| standard, try 2 more and only if they don't work invent a new
| one. Please. I say this as someone that inherited standards
| invented by teams that I then had to try and train hundreds of
| users on. Funnily enough the previous people left when the coding
| was done, without teaching the users. THey probably left to
| implement version 2 elsewhere. :)
| darkteflon wrote:
| How interesting. I was just this weekend looking to implement a
| DAG/multitree-based tasks app as a side project, and my first
| thought was to do it as an obsidian plugin with D3 for rendering.
| I ended up canning that idea because it felt like a bridge too
| far to cram nodes and links into a human-readable format like
| markdown. I went with SQLite and a TUI instead. Seeing this spec
| emerge makes me think that Obsidian might be a good place for
| that kind of plugin after all. It's becoming something of an
| emacs, Obsidian is!
| darkest_ruby wrote:
| Is that what nodered uses for its diagrams?
| cyanydeez wrote:
| need, but on mobile I want to double tap to zoom and fit content
| dorian-graph wrote:
| I was hoping this would include infinite _nesting_!
| account-5 wrote:
| In a similar vein I prefer Treesheets to this sort of thing. I
| like grids better.
___________________________________________________________________
(page generated 2024-03-11 23:00 UTC)