[HN Gopher] Show HN: Fuzzynote - terminal-based, CRDT-backed, co...
___________________________________________________________________
Show HN: Fuzzynote - terminal-based, CRDT-backed, collaborative
note-taking
Author : sambigeara
Score : 74 points
Date : 2021-06-23 16:46 UTC (6 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| sambigeara wrote:
| Author here. Happy to answer questions if there are any!
| benrbray wrote:
| Nice project! How was your experience using CRDTs? I often see
| the criticism thrown around that the history might become
| enormous even with regular usage. I'm curious what you have to
| say!
| sambigeara wrote:
| Thanks! Really good question, particularly in this context.
| It's a problem I'm actively working on.
|
| For context: the underlying CRDT is basically a WAL housing
| all the events originating from user input. The app itself
| operates "local-first", so as events occur, it
| collects/flushes partial WALs which are separately aggregated
| and merged into full ones, and it's these which are shipped
| around to the various "remotes". These will (by definition)
| grow indefinitely.
|
| In order to try and alleviate some of this, I've written a
| "compaction" algorithm which retrospectively traverses the
| WAL and attempts to condense some of the events from many
| into one. For example, if there are 10 update events on the
| same item, omit the previous 9 and store only the most
| recent. It's pretty basic but it's had quite a significant
| impact thus far. Beyond that, I'm going to have to keep
| thinking and iterating!
|
| The things you get for "free" when building on top of CRDTs
| are just brilliant, though. For example - I originally built
| the mergeable WALs purely because it was more convenient for
| me when syncing notes between my home and work laptops. Turns
| out that that gave me the collaborative working stuff for
| free! The real time collab is pretty much a websocket which
| accepts arbitrary encoded strings and echoes them back to
| different users. The app itself decides whether or not it's
| useful and merges it accordingly. I thought that was pretty
| cool.
| ampdepolymerase wrote:
| Does it support character level collaboration (like Google
| docs) or is this just generic entry/line level sync? IIRC CRDTs
| do not work well for Google-docs type text editing because they
| cannot capture intent well.
| sambigeara wrote:
| Actually, I'm intrigued by what you mean by this:
|
| > IIRC CRDTs do not work well for Google-docs type text
| editing because they cannot capture intent well.
|
| Could you elaborate? Appreciate the discussion
| dnautics wrote:
| Check out martin kleppmans YouTube video on YouTube: crdts:
| the hard parts
| ampdepolymerase wrote:
| They don't capture semantic intent that well compared to
| Operational Transforms when conflict arises.
|
| https://www.tiny.cloud/blog/real-time-collaboration-ot-vs-
| cr...
|
| Some teams also encountered engineering problems using
| them.
|
| https://news.ycombinator.com/item?id=24186883
|
| Done properly however, they can be magical.
|
| http://archagon.net/blog/2018/03/24/data-laced-with-
| history/...
| rkallos wrote:
| Definitely a tricky problem for CRDTs. A possible
| solution is outlined in the OpSets paper;
| https://arxiv.org/abs/1805.04263
|
| I hope that there are existing implementations of OpSets
| that perform well. I find the solution laid out in that
| paper to be really elegant and beautiful.
| sambigeara wrote:
| Ah, I understand. Fascinating stuff - thanks for sharing.
| Food for thought when considering how to evolve the app
| (e.g. moving beyond just plain text).
| sambigeara wrote:
| Currently it's at an entry/line level like you suggest,
| although theoretically I could flip (for lack of a better
| term) the algorithm and apply it laterally on a per character
| basis.
|
| The data structure is a WAL storing all mutation events, each
| of which has a reference to another uniquely identifiable
| item (by some UUID:timestamp combo). Rather than treating
| each line as a uniquely identifiable object, I could treat
| each character as such.
|
| Downside of the current implementation is that two people
| operating offline on the same line will see unpredictable
| results when they're able to sync again (basically it will
| just honour the most recent update event). The per-char
| change above would remedy that, I think!
| Johnyma22 wrote:
| You might also like https://github.com/guyzmo/vim-etherpad vim
| Etherpad plugin :)
| sambigeara wrote:
| Ooo! As an avid vim'er, I rate this highly - thanks for
| sharing!
| Evidlo wrote:
| So it looks like the clients talk to each other through AWS.
|
| Would it be possible to use GDrive/Syncthing/Dropbox as the IPC
| mechanism for CRDT (e.g. store changes in a bunch of small files
| in a hidden folder which gets synced to other clients)?
|
| I understand syncing would probably not be fast enough for a live
| cursor, but the benefit is you wouldn't have to worry about
| networking or accounts.
|
| Just an idea I've had in my head for a while.
| sambigeara wrote:
| Providing Go has a decent enough client/interface into the
| tools you listed above, absolutely!!
|
| The original "remote" I implemented was a single S3 bucket (no
| websockets, just a static file location), and if you set the
| synchronisation intervals to something in the realms of
| 10ths-1s of a second, you had near enough real time
| sync/collaboration!
|
| I created a (really poorly named) abstraction called a WalFile
| which is basically an interface to implement these remote
| targets. Theoretically, you could build one for any service out
| there. If there was demand for one, I'd totally give it a go!
|
| Sods law - I temporarily disabled support for the direct S3
| remote yesterday because it's slightly broken. But it's a
| minimal amount of work away from having it functioning again.
| Evidlo wrote:
| Why would you need to interface with these tools at all?
|
| In my mind it would be completely agnostic of the
| synchronization service and rely solely on file read/writes
| on the shared filesystem.
| sambigeara wrote:
| Sorry, I think I misunderstood.
|
| The sync mechanism itself is completely agnostic. The
| interface I'm describing is more in regards to accessing
| it. Only requirement is that you can access a file and run
| a bunch of basic operations (list, get, put etc)
| al_james wrote:
| Looks powerful. You mention the CRDT is a WAL, won't it grow
| indefinitely?
| sambigeara wrote:
| Thanks! Again, very good question. There's an answer I wrote in
| response to benrbray which explains this!
___________________________________________________________________
(page generated 2021-06-23 23:00 UTC)