[HN Gopher] Automerge: A library of data structures for building...
___________________________________________________________________
Automerge: A library of data structures for building collaborative
applications
Author : surprisetalk
Score : 245 points
Date : 2024-07-16 14:03 UTC (4 days ago)
(HTM) web link (automerge.org)
(TXT) w3m dump (automerge.org)
| mkl wrote:
| In practice most projects seem to use Yjs rather than Automerge.
| Is there an up-to-date comparison of the two? Has anyone here
| chosen Automerge over Yjs?
| CharlieDigital wrote:
| There's also Microsoft's Fluid Framework and Azure Fluid Relay
| which is powering their O365/SharePoint product.
| danielvaughn wrote:
| In a recent (abandoned) project, I used Reflect:
| https://reflect.net/
|
| It was by _far_ the most developer-friendly experience I 've
| had trying to implement collaborative editing. The one thing it
| didn't have that YJS did was built-in undo/redo.
| localfirst wrote:
| what about PartyKit?
|
| okay looks like partykit hides some extra metered costs with
| a call us button
|
| reflect's pricing seems a lot easier to understand!
|
| just trying to think how this will work with cloudflare +
| fastify
| danielvaughn wrote:
| Interesting, hadn't heard of PartyKit. FWIW Reflect seems
| to be pivoting sometime over the next 6 months, and they're
| going to open source their code, along with instructions on
| how to self-host. So if you're looking for long-term cost
| reduction, that might not be a bad choice.
| clarkrinker wrote:
| yJS has the webrtc adapter and appears to still win out in the
| edit benchmarks. I've used yJS in two projects: once just for
| presence chat and syncing menus for a coaching site and once in
| a overlay graphics app for a livestream
|
| Biggest problem with yJS for me has been the ergonomics when I
| use it with React. There's a third party project called synced
| store that I used for the stream overlay but it has some
| strange behavior.
|
| With first party support with React in automerge I think it's
| worth a shot for my rewrite
| 6gvONxR4sf7o wrote:
| The big yjs problem for me is the documentation. A few of the
| most important sections are just "todo" placeholders, like
| the "how to write a provider" section.
|
| It's great at first, but woefully underdocumented if you want
| to use it in a way that doesn't have off the shelf support,
| and the code is tough to parse (for me at least). Same with
| subdocuments.
|
| I wanted to use it with Lexical a while back, and yjs plugin
| was too tightly coupled too a single data model and was too
| complicated to DIY.
| josephg wrote:
| I'm quite familiar with both, having spent some time building a
| crdt library of my own. The authors of both projects are lovely
| humans. There are quite a lot of small differences that might
| matter to some people:
|
| - Yjs is mostly made by a single author (Kevin Jahns). It does
| not store the full document history, but it does support
| arbitrarily many checkpoints which you can rewind a document
| to. Yjs is written in JavaScript. There's a rust rewrite (Yrs)
| but it's significantly slower than the JavaScript version for
| some reason. (5-10x slower last I checked).
|
| - Automerge was started by Martin Kleppmann, Cambridge
| professor and author of Designing Data Intensive Applications.
| They have some funding now and as I understand it there are
| people working on it full time. To me it feels a bit more
| researchy - for example the team has been working on Byzantine
| fault tolerance features, rich text and other interesting but
| novel stuff. These days it's written in rust, with wasm builds
| for the web. Automerge stores the entire history of a document,
| so unlike Yjs, deleted items are stored forever - with the
| costs and benefits that brings. Automerge is also significantly
| slower and less memory efficient than Yjs for large text
| documents. (It takes ~10 seconds & 200mb of ram to load a 100
| page document in my tests.) I'm assured the team is working on
| optimisations; which is good because I would very much like to
| see more attention in that area.
|
| They're both good projects, but honestly both could use a lot
| of love. I'd love to have a "SQLite of local first software". I
| think we're close, but not quite there yet.
|
| (There are some much faster test based CRDTs around if that's
| your jam. Aside from my own work, Cola is also a very
| impressive and clean - and orders of magnitude faster than Yjs
| and automerge.)
| conaclos wrote:
| > I'd love to have a "SQLite of local first software"
|
| We have recently published a new research paper on
| replicating SQLite [1] in a local-first manner. We think it
| goes a step closer to that goal.
|
| [1] https://inria.hal.science/hal-04580135/document
| steida wrote:
| It looks very similar to Evolu
| (https://github.com/evoluhq/evolu)
| westurner wrote:
| cr-sqlite https://github.com/vlcn-io/cr-sqlite :
|
| > _Convergent, Replicated SQLite. Multi-writer and CRDT
| support for SQLite_
|
| From "SQLedge: Replicate Postgres to SQLite on the Edge"
| (2023)
| https://news.ycombinator.com/item?id=37063238#37067980 :
|
| >> _In technical terms: cr-sqlite adds multi-master
| replication and partition tolerance to SQLite via
| conflict free replicated data types (CRDTs) and /or
| causally ordered event logs_
| ndr wrote:
| How do you deal with persistence with the various solutions?
| What do you actually have to serialize to a db?
| firtoz wrote:
| I use yjs myself and you can choose to serialise anything
| you like, most solutions allow saving snapshots of the
| document state. You can also store any incoming changes too
| for more fine grained undos and redos etc. AFAIK the state
| in typical solutions is a binary representation.
| zogrodea wrote:
| I remember you describing Ropey's author as a "lovely human"
| too, and want to say that "it takes one to know one/real
| recognises real". :)
| tevon wrote:
| Sooo we're building "SQLite for local-first development",
| it's here! Uses CRDTs, can be a partially replicated db, peer
| to peer networking and discovery.
|
| Bruinen.co
|
| Shoot me a note if you want an early build! Or if interested
| in building with us :)
|
| tevon [at] bruinen.co
| tazjin wrote:
| Using a closed-source DB is a hard sell.
| __MatrixMan__ wrote:
| Agreed, especially for a local first app.
|
| Making the app work without an internet connection is
| step one. Making it reparable without an internet
| connection is step two.
|
| Step two is blocked if you can't keep the code for all of
| the app's dependencies near enough at hand such that its
| still accessible after the network partitions.
| Horusiath wrote:
| > (5-10x slower last I checked)
|
| This was a thing around 2 years ago. Nowadays speeds is the
| same or in favor of Rust, depending on the benchmark in
| question.
| josephg wrote:
| It was still much slower ~6 months ago when I benchmarked
| it. I'll rerun my benchmarks and confirm one way or
| another.
| antidnan wrote:
| I'm curious to hear your thoughts on loro
| stpn wrote:
| I've been working on a personal finance tracker that uses
| automerge as the primary backing store.
|
| One trick we've been pulling is tailing the automerge contents
| into a sqlite db in-browser for more complex querying.
|
| (some notes on how/why here: https://tender.run/blog/tender-
| and-crdts)
| mweidner wrote:
| Many projects use Yjs for its collaborative rich-text editing
| (e.g. Linear: https://x.com/artman/status/1733419888654291102).
| Yjs makes this easy by providing "bindings" to various rich-
| text editor GUIs, which sync between the editor's internal data
| structures and Yjs - something that involves a lot of detail
| work. Automerge's rich-text support is more recent (~last
| year), and so far they only have one editor binding
| (ProseMirror), so Yjs is naturally more popular here.
|
| For non-text collaboration, there is a more crowded "market",
| because it is an easier problem to solve - at least when your
| app has a central server. Tools range from hosted platforms
| like Firebase RTDB to DIY solutions like Figma's
| (https://www.figma.com/blog/how-figmas-multiplayer-
| technology...). Meanwhile, Automerge's target niche is
| _decentralized_ collaborative apps, which are rarer.
| zaphod420 wrote:
| I just recently found out about automerge from this youtube
| channel. https://www.youtube.com/@localfirstfm
|
| I love the idea of "local first software"
| https://www.inkandswitch.com/local-first/
| mkl wrote:
| Big discussion of local-first a few weeks ago:
| https://news.ycombinator.com/item?id=40786425
|
| A number of past discussions too:
| https://hn.algolia.com/?q=local+first
| surfingdino wrote:
| Interesting. IndexedDB + Automerge remind me of CouchDB.
| kennethwolters wrote:
| I am a big fan of local-first design principles. For most
| enthusiasts, it's the performance benefits. However, for me, it's
| the potential it has for privacy in web clients.
|
| If you have a web client that only does on-client data storage,
| you're not dependent on centralized server-side storage for
| persistence (which is the main privacy-loss hazard).
|
| The problem is that the app host still has all the technological
| freedom to not honor their privacy agreement, and there don't
| seem to be backstops to that behavior in browsers.
| bearjaws wrote:
| I just built a local-first app called "cluttr".
|
| Uses OPFS + WASM to run SQLite and host files. I've had a
| pretty healthy response to it from HN so I plan to make a
| desktop app + add PeerJS for file sharing.
| WolfOliver wrote:
| Any insights/research on what kind of users care / don't care
| about local first software?
| bionhoward wrote:
| security is a big reason to go local first in general (not
| necessarily with crdts but they are a strong option), less data
| over the wire to central servers, you can avoid even needing to
| know about user data if it stays on their device(s)
|
| collaboration software benefits a lot from this approach
| because people can just use it and know it won't get out of
| sync
|
| Also, anything where you might _need_ data offline at a
| specific moment. Like if the internet goes down but you still
| need the thing to show to the guy at the place, you'll be glad
| if they used crdts because you'll automatically have a copy
|
| Also, cheapskates who don't want to pay cloud bills
| WolfOliver wrote:
| > and know it won't get out of sync
|
| but will you ever be able to trust the automatic merge?
|
| Imagine a complex document with a few collaborators which
| work in parallel offline and make significant changes (that
| is the use case of automerge?).
|
| You will have to proofread the entire document after every
| non trivial automatic merge, or how good does this work in
| practice? Would it be easier to just wait for a wifi
| connection and do the changes in real time?
| WolfOliver wrote:
| now you can compare it to pull requests in software
| development, but in software development we have compilers,
| linters and tests which point out merge bugs.
| WolfOliver wrote:
| > Also, cheapskates who don't want to pay cloud bills
|
| You will have to store it in the cloud eventually, otherwise
| it is just local software, not local-first
| nanomonkey wrote:
| Not quite true, you can utilize peer to peer (p2p) storage
| options, where only the individuals that are allowed to
| edit and view the document/media/file do storage. Really
| you only need one peer that has an IP address to do NAT
| traversals, that peer need not do any storage, but just
| passes transactions along to the end users.
|
| With IPV6 and Torr onion routing NAT traversals is almost
| becoming a non issue.
| ______ wrote:
| Construction industry is a good one: often you are working in a
| remote area or have cut power (so no WiFi) and having access to
| your data is key. That's why there is so much printing still
| going on- paper works offline.
| firtoz wrote:
| Anyone using their laptops or phones on public or private
| transport, or when you have sporadic or unreliable internet.
| __MatrixMan__ wrote:
| People who are experiencing a natural or political disaster
| which partitions the internet.
| nik-graf wrote:
| Last week I published Automerge Jumpstart https://www.automerge-
| jumpstart.com/
|
| A boilerplate including user authentication & authorization
|
| Tech: Automerge, tRPC, Prisma and deployment on fly.io and
| vercel.com Bonus: includes explanation videos on the website
| amelius wrote:
| Why don't we have programming languages that use this as their
| fundamental design principle?
|
| It would be great if programs were collaborative out of the box.
| ecto wrote:
| Because you haven't created it yet.
| hpen wrote:
| Why a language and not a framework? I mean some langs even
| implement half their functionality in std libraries rather than
| part of the lang (as a preference actually).
|
| I see their being overhead and what not that would make it very
| domain specific and less appealing for anything that doesn't
| need collaboration.
| amelius wrote:
| > Why a language and not a framework?
|
| People said the same about web programming and yet we have
| e.g. Svelte.
|
| Some things are just not so well expressed in a framework.
| Especially things that manipulate state in special ways.
| firethief wrote:
| I can't picture how that would work. While collaborative
| features require that some objects are shared and synchronized,
| efficiency and programmer sanity rely on the fact that some
| objects are not. If synchronization is opt-in, how would a
| language integrate it any more fundamentally than a library
| can?
|
| Also, CRDT's don't provide synchronization for free. They
| ensure that all concurrent modifications will be merged
| _somehow_. If the data being synchronized has any structure, it
| requires careful CRDT-aware data model design to ensure the
| merging is semantically reasonable (or that, in the worst case,
| incompatible changes produce a _detectably_ broken state).
| photonthug wrote:
| Think about things like browser profiles or password managers
| that are kind of predicated on multi device and how none of
| that stuff is what you'd call zero setup. Does every app that
| wants to do this stuff need a key base or drop box backend?
| Is apps joining an ad hoc k8s network mesh or relying on a
| preexisting central redis or postgres or something really
| necessary for something like sharing data structures?
|
| There's definitely some room for interesting work here and
| language level support could be cool.
|
| Elixir interpreter clustering and otp is maybe the closest
| existing thing, which is awesome but only for existing erlang
| fans.
| CyberDildonics wrote:
| Why would you even want to start an entire new language with no
| tools, no debugging, no libraries, no ides and no optimizations
| just so you could make some data structures that could be made
| with C++ classes?
| dietr1ch wrote:
| It'd be great, but I think that we are not ready for this yet.
| Think on how long we needed to make the idea of having good
| tooling, cross-compiling support, testing or debugging support
| a requirement.
|
| A library or even built-in language support for distributed
| data structures will take a decade or two to get to the point
| of proving a set of features to be truly helpful and
| good/necessary to have as a library or maybe even as a built-in
| feature.
|
| BTW, we don't even have quickcheck yet, since the generation
| and proof reduction quality isn't great across implementations
| :/
| montroser wrote:
| See also Pigeon[1], which is inspired by automerge but takes a
| different approach:
|
| > While Automerge optimizes for working offline and merging
| changes periodically, Pigeon is optimized for online real-time
| collaboration.
|
| [1]: https://github.com/frameable/pigeon
| hpen wrote:
| What type of clock is used by this library?
| hpen wrote:
| Found it
| https://github.com/automerge/automerge/blob/b8f88cacc6f46f92...
|
| Vector clock. Why not a Hybrid Logical Clock?
| ttecho wrote:
| Wondering how this can be used in the digital audio tech space.
| Collaborative tools in DAWs would be amazing!
| elijahbenizzy wrote:
| I love CRDTs, but it really seems like the type of thing where
| the idea of them is the real impact (many systems are built with
| structures that enable CRDT-like operations without actually
| using CRDTs), and the implementations never quite hit product
| market fit.
___________________________________________________________________
(page generated 2024-07-20 23:13 UTC)