[HN Gopher] Show HN: A version control system based on rsync
___________________________________________________________________
Show HN: A version control system based on rsync
Hi everyone! I'm trying to create a version control system that
solves some of the problems that Git and other version control
software has when working in a team. Let me know if you have any
feedback!
Author : zdgeier
Score : 205 points
Date : 2023-01-19 12:07 UTC (10 hours ago)
(HTM) web link (jamsync.dev)
(TXT) w3m dump (jamsync.dev)
| forgotpwd16 wrote:
| Question, due to a recent thread on an rsync-like CDC-based tool,
| for this problem how rsync compares to CDC?
| zdgeier wrote:
| Thanks for bringing this up! I somehow missed this. Yes I plan
| on implementing a CDC-based approach in the future but for now
| I'm just using a naive fixed chunk size.
| remram wrote:
| An option for mounting the files with fuse and/or doing a sparse
| checkout would probably be good. Otherwise your working copy may
| use a lot of disk space with redundant data.
| zdgeier wrote:
| Thanks! Yeah planning on adding the ability to mount specific
| directories so people don't have to pull down large assets they
| don't need, for example in gamedev or machine learning
| applications.
| hardwaregeek wrote:
| There seem to be a few projects that try to be a live,
| collaborative git. Perhaps it's just me, but I don't necessarily
| think I'd want this for my main system. Maybe a short term
| collaboration/pair programming session, but not in general. I
| like how git is not immediate. I like that I can take time before
| sharing my code to clean it up. The issues that a tool like this
| purports to solve are not issues to me. Live collaboration is a
| great idea, but maybe not as a version control system. But hey if
| other people get use out of it, why not!
| andix wrote:
| What I always wanted to have is a tool that syncs my
| uncommitted changes of git repos between all my devices. So
| that I can switch from the desktop to the laptop without
| committing, pushing and pulling. Right where I stopped. Also
| sync not yet pushed commits, I may want to rebase/edit before
| pushing.
| pmontra wrote:
| By uncommitted changes do you mean edited files? Then
| something like syncthing or anything functionally similar to
| Dropbox.
| klabb3 wrote:
| 100x yes. I agree with parent that pushing live changes is
| way to intrusive, but a synced working/staging directory
| could be incredibly helpful.
|
| I don't see any fundamental reason why git or, even 3p
| tooling for git, couldn't support this use case using
| something like private, ephemeral, auto-synced branches.
| amelius wrote:
| This may also be the reason why Google Wave failed.
| Jtsummers wrote:
| Google Wave primarily failed because:
|
| 1. Performance was never more than tolerable
|
| 2. Invite only access
|
| 3. Invites didn't go out immediately, just added people to a
| pool of potential future invites
|
| A product that required the network effect of a large
| _connected_ userbase to succeed only had random individuals
| with access. Literally no one I invited and would have used
| it with got in, and I was in one of the first couple batches
| of public users to get access. A spectacular self-own on
| Google 's part with that rollout.
| zdgeier wrote:
| Appreciate the feedback! Will keep this use case in mind, this
| could definitely work as a layer on top of Git when people need
| to pair program or preview code.
| quintussss wrote:
| Would be nice if you could talk about the problems you had with
| git and how jamsync solves them.
| zdgeier wrote:
| Hi! I decided to link to the About page since this content was
| better suited for HN. You can read more about the problems
| Jamsync plans to solve on the homepage at jamsync.dev/ . Thanks
| for the suggestion!
| kokanee wrote:
| I think a lot of folks in this thread are forgetting the first
| day they spent using git. It's kind of like asking why someone
| would create a text editor in a world where vim exists, since
| vim can already do everything you need. Sure, but it takes
| months to become proficient with and years to master.
|
| Years ago I worked at a company that had a team of front end
| developers that used Dropbox for version control instead of
| git. Conflicts were prevented by the default "this file has
| changed on disk, are you sure you want to save?" warning in
| their IDEs. It worked incredibly well for that team, and they
| never suffered a single gitastrophe.
| billconan wrote:
| reminds me of https://xethub.com/assets/docs/xet-specifics/how-
| xet-dedupli...
| frogperson wrote:
| I don't think I would want this for code, but it sounds really
| interesting for a shared drive. I would love to version control
| the dir where my team dumps all of its docs and spreadsheets and
| everything else.
| zdgeier wrote:
| Thanks for the feedback! What about a version control system
| would be valuable for storing your documents? I'm curious why
| you couldn't use something like Google Drive/Dropbox.
| jbverschoor wrote:
| Security, local copies, and limited restoration options
| richardwhiuk wrote:
| I'm curious whether there are common operations which cause
| extremely expensive resyncs - it's unclear, depending on the
| hash.
|
| e.g. does adding a space at the start of a file causing all of
| the blocks to be misaligned?
| zdgeier wrote:
| The rsync paper
| https://www.andrew.cmu.edu/course/15-749/READINGS/required/c...
| is better than me at explaining this, but the rolling hash is
| the key to solving this misalignment problem that you describe.
| Essentially, using the rolling hash enables you to detect when
| a block as been misaligned (bytes added at the beginning of the
| file for example). There's no way to cause expensive resyncs in
| this case since it rolls over every byte in the file and reuses
| blocks on any alignment, even with data inserted in between.
| account-5 wrote:
| How does this compare with fossil? I use fossil for personal
| projects.
| zdgeier wrote:
| There are some similarities but Fossil still has a
| push/pull/commit flow, although it does provide syncing
| features to automatically pull commits. Also, I don't think it
| has great support for large files.
|
| Ultimately, the priorities of Fossil are going to be different
| than Jamsync, since Jamsync is not decentralized. Being
| centralized means that we'll be able to do some different
| things, like live file editing, but with some downsides related
| to replication and distribution.
| sigmonsays wrote:
| how does this even work with merges or conflicts?
|
| How do you review changes easily?
|
| It seems like a lot is actually missing. Rolling hashes over
| blocks is not exactly revision control.
| zdgeier wrote:
| Yeah this project is still pretty early in development. The
| main algorithm described on the site is implemented but there's
| a lot missing that developers would expect from a system like
| this. Trying to make a version control system and hosting
| platform in my free time while working a full-time job is not
| exactly an easy task haha. I plan on adding more features in
| the next few months.
| jerpint wrote:
| One thing that could be useful for this is writing code locally
| and testing it on a remote server. I assume the simple act of
| editing reflects the change on the other machine? No need to
| commit, push, then pull to do a one-liner experimental change
| pbalau wrote:
| This is what vscode is doing with the remote-xxx family of
| plugins.
| zdgeier wrote:
| Thanks for checking this out! Yep that's where I'm planning on
| going with this. I think this could even be extended to
| modifying and deploying code without having the
| push/commit/pull process that we have now.
| rickydroll wrote:
| I would encourage you to explore this more thoroughly. I
| frequently find myself working with three machines at the
| same time. My Windows box because of needing to use speech
| recognition in conjunction with copilot and VS code. My Linux
| laptop for local endpoint, and a VM endpoint somewhere (the
| old man says standing in the yard shaking his fist at the
| cloud)
|
| Far too often I find myself making all my changes on my
| Windows box, or copying them to one destination, the other,
| or both. Then bug fixing on the local machine as part of my
| test cycle. Then I have a few moments of confusion when I try
| to remember what I changed where and how to pull them all
| back over to my Windows box.
|
| So yeah, something like your proposed tool would be quite
| useful.
| zdgeier wrote:
| Yeah definitely found myself having some of the same pain
| points when working on multiple machines. I'll keep this in
| mind!
| Fire-Dragon-DoL wrote:
| Isn't the goal of pushing to prevent that?
| nvartolomei wrote:
| I'm using this today for a similar use case
| https://www.cis.upenn.edu/~bcpierce/unison/
| passthejoe wrote:
| I'm a big Unison fan, but it can be finicky. You need the
| same version of Unison on both sides of the sync, and that's
| sometimes hard to do.
|
| However, it solves the problem of when there are changes on
| both sides and you want to reconcile them.
| pmontra wrote:
| I did use Unison but I lost a lot of time merging changes.
| Then I switched to syncthing that also work on my Android
| devices. Syncthing has it's share of pain points, most
| notably that out of sync / override changes button that
| many people don't understand. It's a very bad piece of UX
| and it makes people wonder if they are about to lose some
| data on one or both sides of sync. They should do without
| it at least when sync is not bidirectional.
| replwoacause wrote:
| This looks great. Keep going!
| zdgeier wrote:
| Thanks!
| Dave3of5 wrote:
| If two users are editing the same file this looks like you'll see
| my edits in your file in near real-time. This is being used to do
| way with merging / merge conflicts. A few observations:
|
| * I think this makes two devs changing the file basically
| unusable as you're changing the file and potentially the same
| line simultaneous which could cause me to overwrite your changes
| without realising
|
| * It still presumable doesn't get away from the fact I can edit
| the file with the jam command not running then run it after in
| which it'll just overwrite my local changes onto the remote
| without merging
|
| * What does it do if my editor has changes in the change buffer
| that aren't saved and an update comes in on the same line, then I
| save. Will my now out of date change buffer overwrite the server
| version ?
|
| I don't think you can do away with merging and merge conflicts
| it's a vital part of source control so that two devs can work on
| the same file simultaneously.
|
| How often does the thing sync, none of the demos really show
| this. The demo's showed the thing running very often i.e. every
| few seconds. Does that mean if I'm making large changes to a file
| it automatically upload every time I save the file ? I definitely
| do not want that as typically when editing code as I edit I will
| save something that is in a partially broken / incomplete state.
| It also precludes the main CI/CD workflows which will normally
| trigger from a dev uploading something they think will work.
|
| Where is the actual source, the source page links me to here:
| https://jamsync.dev/download
|
| This just gives a link to a .zip which is presumably out of date
| now. I presume you are trying to self host this but since you
| haven't ironed out the details it would be better to use an open
| source code tool like github /gitlab ...etc for the time being.
|
| You are allowing me to register / login to an "account" on
| something with basically no terms of service or privacy policy.
| Not only are you not GDPR compliant but more importantly at the
| moment you are not limiting your liability! The fact you've stuck
| this up without limiting your liability you are now partially
| liable for whatever stuff people are putting up on here baffles
| me. Seriously for your own good please get some terms of service
| with at least limitation of liability.
|
| You'll have to also sort the problem of copyright as I wouldn't
| push code to a service that attempts to claim any copyright of
| that which I assume you're not doing but I would need that in
| writing.
|
| Given the frequency of commits that this solution will introduce
| you'll need to offer way more than 5GB of hosting.
|
| How will that version slider work when I have 500k commits. Even
| squashing branches, the repo I'm working on at the moment has 10k
| commits. This is a team of about 30-40 devs and each day is about
| 15ish PRs. With Jamsync syncing every save I could easily see
| this going up to millions of commits.
| zdgeier wrote:
| Hi! This is the first time I'm posting publicly about this in-
| development project so thanks for the feedback.
|
| Causing a conflict will cause a .jamdiff file to be written out
| on the next sync with the remote changes and I'm planning on
| adding branches in the next update which will make how this
| works more clear. There's still a concept of parallel editing,
| since features cannot be developed/tested simultaneously
| without breaking. Also, anytime there's a conflict, we can just
| make a new branch and ask the developer to merge or keep
| working on the new branch.
|
| The client will watch your file system for changes and hold a
| gRPC stream open for remote changes. If you don't want to sync
| your changes, you don't have to leave the command running.
| CI/CD support will come later.
|
| I'll be compliant with regulations soon, but I'm not really
| expecting people to use this yet. I mostly wanted to release
| the source and see what people thought of the project. I am
| using GDPR compliant Plausible analytics if that makes you feel
| better!
|
| Thanks for the suggestion on the source, the zip file is up-to-
| date and is part of my build process. I might host on Github in
| the future but really wanted to make this the source of truth.
| My goal of open-sourcing is not to get contributors, but to
| give back to people who want to view the code and self host.
|
| As far as scaling issues go, I'm planning on using an approach
| similar to https://madebyevan.com/algos/log-spaced-snapshots/
| but I'm still figuring out the best way to handle commits.
| Dave3of5 wrote:
| > Hi! This is the first time I'm posting publicly about this
| in-development project so thanks for the feedback.
|
| You know I can see you post history right ? 5 months ago you
| posted about this:
|
| https://news.ycombinator.com/item?id=32546472
|
| Please don't lie to me...
|
| > Also, anytime there's a conflict, we can just make a new
| branch and ask the developer to merge or keep working on the
| new branch.
|
| Ok so this tool doesn't solve the problem of merge conflicts.
|
| > The client will watch your file system for changes and hold
| a gRPC stream open for remote changes. If you don't want to
| sync your changes, you don't have to leave the command
| running. CI/CD support will come later.
|
| Eek then the thing isn't consistent if two of us are using
| this tool and I keep it open all the time I'll generate a lot
| of commits. If you only run the command sporadically you'll
| generate significantly less.
|
| > I'll be compliant with regulations soon, but I'm not really
| expecting people to use this yet. I mostly wanted to release
| the source and see what people thought of the project. I am
| using GDPR compliant Plausible analytics if that makes you
| feel better!
|
| I feel nothing only pain :-). I note that you have missed out
| my comments on limiting your liability. Since you seem to be
| deliberately doing that I'll give you some advice. There are
| some heinous people online who will use this service in it's
| current form to share some vile shit with each other. At the
| moment you are liable for that. IANAL.
|
| > Thanks for the suggestion on the source, the zip file is
| up-to-date and is part of my build process. I might host on
| Github in the future but really wanted to make this the
| source of truth. My goal of open-sourcing is not to get
| contributors, but to give back to people who want to view the
| code and self host.
|
| Erm you know I can see the source. You have the code stored
| in a private repo in github.
|
| To be blunt I don't trust you anymore.
| zdgeier wrote:
| Appreciate the feedback! I used the same name for that
| project but this is actually a completely different project
| rewritten from scratch and open-source. I've been working
| on the general idea for a better VCS for some time so I've
| restarted around 6 times so far to get something that
| works. Not sure it will be too productive to continue the
| conversation here but feel free to reach out to me if you
| would like.
| kypro wrote:
| I have nothing much to add about the tool, but just want to say
| this is some beautiful documentation.
| zdgeier wrote:
| Thanks :) I unfortunately can't take the credit for this
| beautiful theme. Credit goes to https://radoslawkoziel.pl/ who
| made https://github.com/panr/hugo-theme-terminal which I based
| this off of.
| aliqot wrote:
| > We're currently not taking contributions but feel free to email
| us for feature requests.
|
| why
| samwillis wrote:
| Open source != open contribution or project
|
| The most well known project following this is SQLite.
|
| There are many reasons to do this:
|
| - ensuring that the copyright is fully owned
|
| - ensuring security by reducing the likelihood of someone
| submitting secretly vulnerable code
|
| - not wanting the overhead of running an Open Source _Project_
|
| - wanting ultimate control on the architecture of the tool
|
| - early stage of a rapidly changing codebase
|
| There is a danger that "gate keeping" the concept of open
| contribution == open source, reduces the motivation of people
| to open source their code.
|
| All open source, no matter the contribution policy, should be
| applauded.
| zdgeier wrote:
| Still in the early stages of development with code changing
| rapidly. Not sure if it would be the best idea to get other
| people working on things at this stage, but I'm open to
| suggestions!
| riceart wrote:
| My suggestion: keep doing what you're comfortable with.
| Having source available is better than not, full stop - and
| should be commended. Don't succumb to peer pressure. If you
| want to make a "community" project do it on your own terms.
| KerrAvon wrote:
| I don't understand the use case. Very large file support aside,
| are you saying you've observed people looking at git and say "I
| wish I could use this like I use a chat server?"
| zdgeier wrote:
| Haha appreciate you taking a look. One use case is to prevent
| merge conflicts, especially when working on a fast-moving
| project. Merge conflicts generally occur when you make changes
| on some old code. Jamsync constantly "rebases" your changes on
| top of new changes so that your code will always be up to date.
| Obviously some merge conflicts will still occur but much less
| so in this approach.
| zdgeier wrote:
| More details about the problems I'm trying to solve are on
| the homepage if you're curious.
| kccqzy wrote:
| How is that different from running `git pull` or the
| equivalent faster, maybe on a timer?
| zabzonk wrote:
| so is this kind of a better replacement for subversion? thinking
| about large file support. i'm not aware of major problems that
| git and/or hg have when working with a team.
| zdgeier wrote:
| Yes, that's one way to think about it. You could probably say
| it's closer to Google Drive than a VCS right now but my idea is
| to make a VCS that's more collaborative than the options
| currently available. Some people will have no issues with the
| current VCS that we have now but many developers that I've
| talked to have expressed frustration with the general user
| experience of VCSs and when merge conflicts occur.
| harryvederci wrote:
| Nice, I hope you won't get an avalanche of comments here by
| grumpy old devs that are afraid this will somehow mean they'll
| have to learn something new.
|
| I think it's great to look into possibilities of doing things in
| a better way, even if the majority of people think the current
| way is the only correct way.
| thrownawaydad wrote:
| Grumpy old dev here. I make my money cleaning up messes caused
| by ill-considered attempts to use something new. Thank you. :-)
| [deleted]
| tambourine_man wrote:
| I think I qualify as an old developer.
|
| I love learning new exciting things that I know I'll use for
| the rest of my live. C, Vim, Bash, HTML, CSS, JavaScript will
| all outlive everyone here. It's a pretty safe bet. I hate the
| amount of useless knowledge I accumulated for obsolete
| unexciting things.
|
| I feel like Git is not the final answer to version control, it
| had a lot of great ideas, but it's not even good enough for
| some things and too complicated for most others. My hope is
| that it becomes like SVN, mostly legacy stuff, and that we can
| build something new with the lessons learned from Git. I wish
| Jamsync luck, it looks interesting and I freaking love rsync.
| johngalt_ wrote:
| What do think are the shortcomings of Git?
| carlhjerpe wrote:
| That it's build on snapshots rather than patches, see darcs
| or pijul.
|
| I'm not using either, because of the git momentum. Only
| having to know one is nicer than using one for work and one
| personally.
| tambourine_man wrote:
| In a nutshell, terrible end user UX and naming. Can't deal
| with largish files well.
| mbork_pl wrote:
| I guess Git suffers from a similar problem as (La)TeX: it's a
| truly great prototype which sadly wasn't thrown away to build
| something better.
|
| Don't get me wrong, I like both TeX and Git _very_ much. (I
| even co-authored a LaTeX textbook.) I also have a lot of
| respect for DEK and LT. But they were trailblazers
| (especially DEK), and so they did a lot of things not knowing
| their true impact - and sometimes this means these decisions
| were far from optimal. (The case of TeX is even worse because
| the machines of the time were very limited compared to what
| we have today.)
| tambourine_man wrote:
| That's a very interesting take. Both were also developed
| mostly by one person to do one job. Knuth for his "The Art
| of Computer Programming" masterpiece and Linus for the
| kernel. There may be similarities that arise from such
| histories and constrains.
| zdgeier wrote:
| Thanks! Haha yeah my plan is to certainly not try to replace
| any existing tools but provide options for working faster,
| especially for new developers.
| ActionHank wrote:
| I don't think of myself as old, but certainly grumpy and this
| comment rubbed me the wrong way.
|
| In my time as a dev I've worked with CVS, SVN, Mercurial, and
| then git. I can confidently say that no dev I've worked with
| ever kicked up a fuss when they switched, because each
| iteration brought improvements.
|
| I would however say that modern devs carry with them modern
| baggage. There are far too many bootcamps churning out devs who
| say "this is git, everyone uses it, heres the minimum you need
| to know". These are the devs who will struggle with this sort
| of change, it would literally change a magical system they
| don't fully understand for reasons they may not fully
| understand.
| ohitsdom wrote:
| "no dev I've worked with ever kicked up a fuss when they
| switched"
|
| Wow. I don't think your experience is the common one. I've
| been on teams switching to git, and there was always much
| fussing. Even if the benefits of switching were clear and it
| was worth it, doesn't mean there won't be pain along the way.
| no_wizard wrote:
| I would have fussed going from Mercurial to git. Mercurial is
| arguably superior, much like Fossil, but unfortunately
| mindshare / marketshare is everything.
|
| The layering model of merging in both those systems is
| superior to the standard git merge (rebase is close though).
| they also made cherry picking a trivial operation compared to
| git
| eikenberry wrote:
| I willingly switched from Mercuial to Git due to, at the
| time, lack of in-repo branching and lack of history
| editing. IMO Mercurial shot themselves in the foot with
| some of their early decisions.
| icelancer wrote:
| Yeah, I was gonna say exactly this. When I switched from
| SVN to hg that was a bit of a shock, but I got over it and
| saw that DVCS was superior.
|
| Going from hg to git has been fairly annoying and... I'm
| not really convinced it's any better. Or maybe even at
| parity.
| klyrs wrote:
| I've seen whole teams kick and scream about migrating, even
| when their preferred system was taking minutes to do basic
| operations because the repo was too large.
| mjburgess wrote:
| Both POV here can be reconciled with an experience-dependent
| exploitation-exploration tradeoff.
|
| For _everyone_ ,
|
| Eg., suppose t is the reward for exploitation, and r for
| exploration, and xp is your experience.
|
| Then as xp -> inf, t -> inf and r ->0.
|
| As xp -> 0, the reverse happens: t -> 0 and r -> inf.
| brendoncarroll wrote:
| This is a very similar to one of my projects "Got".
|
| https://github.com/gotvc/got
|
| The algorithms it uses are superior to rsync and git in a few
| ways. It comes short on features, especially for software
| development compared to Git. The motivation is more for personal
| file storage.
|
| I notice you're using Go and AGPL licensed, so you could borrow
| any of Got's libraries without issue. (Got is GPL licensed.)
| Definitely reach out in a GitHub issue.
| zdgeier wrote:
| Ah awesome! I'll check this out
| zdgeier wrote:
| Took a look and this is really cool. Will definitely keep
| this is mind. One major difference and area I'm focused on in
| this project is hosting. I would argue that there are
| currently much better VCS options for projects than Git (like
| yours and Fossil) but the reason these haven't taken off is
| that Github and Gitlab offer unmatched hosting and
| collaboration tools for these projects. Would be curious to
| know your thoughts/approach to this!
| brendoncarroll wrote:
| My approach to hosting with Got has been to make it easy
| and secure for users to host from any machine.
|
| INET256 solves that problem nicely. If you have access to
| an INET256 network, then all you have to do is swap
| addresses and two Got instances can communicate.
|
| https://github.com/inet256/inet256
|
| Also, end-to-end encryption is table stakes. Any data that
| leaves the user needs to be encrypted in transit, and if it
| hangs around away from the user, at rest.
| beagle3 wrote:
| FYI: "got" is also the name of the OpenBSD re-implementation of
| git (supposedly, an acronym, "game of trees"):
| https://gameoftrees.org/
| red-iron-pine wrote:
| in the game of trees you either live, or branch
| kamranjon wrote:
| "The algorithms it uses are superior to rsync and git in a few
| ways." - Would you mind explaining this?
| brendoncarroll wrote:
| Sure, Git stores data in a trie. Each file is one blob
| identified by hash, and directories (called trees in Git) are
| blobs where each line is a directory entry with a name and
| the hash of a file or another tree. This means that modifying
| an object /down/a/long/path/like/this.txt has to create
| copies of all the trees on the way up. The technical term for
| this is "write amplification", and in Git it is affected by
| path length among other things.
|
| Got stores data in a probabilistic tree (GotKV[0]). The
| number of nodes before you get to data will scale
| logarithmically with the size of the entire filesystem, not
| the depth of a specific object.
|
| Then there is the issue of large files. A file in Git is
| always 1 blob. Syncing a large blob is not easy because if
| you are interrupted and have to restart, you have lost all
| your progress. You can't verify the hash of a blob until you
| have the whole thing. Got has a maximum blob size, so you'll
| only be buffering <2MB at a time before you can verify that
| the blob is correct. If a transfer is interrupted, the most
| you'll have to repeat is one blobs worth, plus any tree nodes
| above that blob.
|
| Compared to rsync, Got uses variable size chunks and a faster
| content defined chunking algorithm, recently featured here on
| HN[1]. I haven't thought about if variable vs fixed chunks is
| better for file transfer, but for version control, the higher
| chance of convergence is important. It means you have better
| deduplication.
|
| [0] https://github.com/gotvc/got/tree/master/pkg/gotkv
|
| [1] https://news.ycombinator.com/item?id=34303497
| toyg wrote:
| This is the awesome content I still come to HN for. Thanks
| for the post and good luck with Got.
| Zamicol wrote:
| I've not heard the term "probabilistic tree" and I've
| having difficulty pulling up references. I suspect it's
| implemented by subpackage ptree[0]. Could you explain what
| makes probabilistic trees different from hash tables or
| other similar data structures?
|
| [0]
| https://github.com/gotvc/got/tree/master/pkg/gotkv/ptree
| vnorilo wrote:
| I was thinking of treaps;
|
| https://en.m.wikipedia.org/wiki/Treap
| hinkley wrote:
| Some, but not all, treaps have a node weight that is
| updated in a probabilistic fashion. The act of balancing
| the tree is still deterministic, but the weights of each
| node are randomized.
|
| I keep trying to find a use for treaps, but haven't had a
| project that needed it. In particular, the value of a
| balanced tree is in consistent cost of lookups for
| arbitrary elements. But if you are mixing entries that
| are accessed often with those that are not, having an 8:1
| access time ratio between the two would be a _feature_
| not a bug.
| brendoncarroll wrote:
| Yep, your link is indeed to the probabilistic tree used
| in GotKV.
|
| Here "probabilistic" just refers to a way of balancing a
| tree. Rather than having a set of rules to keep the tree
| balanced, like with a btree or red-black tree, balancing
| decisions are made pseudorandomly. The result is that the
| tree is very likely to be balanced, and is unlikely to be
| unbalanced.
|
| In the case of GotKV's tree: the entries are stored
| together in a stream, and for each entry a hash is
| computed. If that hash is lower than a certain value then
| the entry is considered a split point, and a tree node is
| created. So now we have a stream of entries, divided
| probabilistically into sections. Each section is a tree
| node. Now take references to those nodes and turn them
| into entries, and repeat the process, so you have fewer
| nodes. That continues until you have one node, which is
| the root. This technique is very similar to content
| defined chunking, and some probabilistic trees are
| implemented using content defined chunking on their
| record format, rather than a pseudorandom value
| calculated per entry, as in GotKV.
|
| For those unfamiliar with probabilistic data structures,
| I highly recommend trying to understand skip-lists first.
| At least why they are balanced.
|
| https://en.wikipedia.org/wiki/Skip_list
|
| As an aside, one of the neat things about GotKV is that
| the keys are delta-encoded. Adding or removing a prefix
| from every key in the tree is a constant time operation.
| This might be obvious to some of the database folks out
| there, but it's a fun mind-blower if you haven't
| encountered the technique before.
| Zamicol wrote:
| So it is a synonym for a hash tree? https://en.wikipedia.
| org/wiki/Hash_tree_(persistent_data_str...
| dperalta wrote:
| Thank you for such a detailed response!
| [deleted]
| ipid wrote:
| I wonder whether this system works for zip-compressed files, such
| as large .docx, .npz, etc. These are common large files, but it
| seems that rsync cannot handle compressed files.
|
| Would Jamsync solve this problem?
| zdgeier wrote:
| Not currently, but it would definitely be interesting to have
| an unzip done automatically based on the file type. Then you
| would be able to track the contents efficiently, rather than
| having large binary blobs with large diffs. I haven't heard of
| other version control systems out there doing this.
| tuukkah wrote:
| It's not yet clear to me whether Jamsync has the concept of
| commits and tracks the history of locally made changes. (Rsync
| obviously hasn't and doesn't.)
|
| Does Jamsync have 3-way merging (e.g. of local and remote changes
| on top of a shared ancestor)?
| zdgeier wrote:
| I'm still figuring out semantics, but currently a "commit" is
| made every time you make a change to a file which is synced to
| the remote. Every time you run `jam`, you push a commit. Or if
| you leave `jam` running, commits will be made each time you
| save. 3-way merging will be possible in the future, but will be
| a little different than most VCS since there is no history
| stored locally.
| jeremyjh wrote:
| So how do you communicate with other developers the intention
| of a commit or set of commits, like you would with commit
| messages in git? How do you rewrite/cleanup history?
| zdgeier wrote:
| This is still in-development but my plan is to essentially
| have "branches" which are merged and create a single
| "commit" on the main branch. You'll have the option to add
| information to a branch and a merge message if you would
| like, but it won't be required. I don't have any approaches
| on rewriting/cleaning up history right now since I think
| managing code history is pretty low-value for most teams.
| In my experience, it's far better to add another commit
| than it is to try to change history. Would be curious to
| know your thoughts though!
| jacobsenscott wrote:
| This sounds great. Your VCS should fade into the
| background, rather than be front and center like git.
| Nobody should be breaking their flow, staging,
| committing, pushing as an ersatz backup, and then going
| back to coding. Teams shouldn't be bikeshedding on merge
| vs rebase, many commits vs one commit, etc. They should
| just be coding.
|
| However I can see where a sort of "checkpoint" feature
| before performing a "hold my beer" change makes sense. A
| commit in some sense. `jam checkpoint` ... code ... `jam
| revert`
|
| You should just be coding, and when the thing your are
| coding is done just do a single "squash merge (in git
| speak)" to the main branch.
| samwillis wrote:
| What's the particular use case for this tool?
|
| Would it be better for projects with large static/binary assets,
| such as game dev or CAD?
| zdgeier wrote:
| Yes, versioning large files is certainly one use case that
| Jamsync solves (and was the original problem that caused me to
| develop this). More use cases are on the homepage, but I think
| that preventing merge conflicts could be interesting. Typically
| a source of merge conflicts is not pulling/rebasing frequently
| enough which Jamsync plans to solve by pulling in changes as
| they happen on the current branch and rebasing branches as
| changes happen on the "main" branch. This means that branches
| are constantly being merged and reduces the risk that you base
| your changes off of old code.
| johnwalkr wrote:
| It's a bit more complicated for 3D CAD (I'm talking about the
| mechanical engineering type here) than just rebasing
| frequently. I'm going to write a long comment because I hope
| someone will solve my problem..
|
| I'm looking for a good tool for 3D CAD version control
| because the available ones are all expensive and focus on
| features for managers, not engineers (like approval
| workflows). I was excited by Jamsync, but I don't think this
| will work well for a few reasons. First, CAD software
| generally modifies every file you open, unless it is read-
| only. This is mainly because even rotating a model causes a
| file change. Second, it's really common for multiple
| engineers to work in the same assembly, even if they are
| working in different parts. Third, merging of the CAD files,
| which are binary, isn't usually possible even when using the
| same vendor for both CAD and version control. So all you can
| do is select a version to keep, you can't merge. It's
| annoying because often one person's real work is overwritten
| just because someone else had a stale copy of an assembly
| file, and saved over the real work with something trivial
| like a view rotation. Only there's no easy way even compare
| two versions so the most common scenario is redoing the work.
| CAD won't generally continuously load any changes like in the
| example on your website, because engineers have assemblies
| open, for the entire day, with all or most files in a
| project, and these are loaded into memory once and locked on
| the disk.
|
| All this means that one feature is 100% required: (a)
| explicit checkout and checkin of the files you want to work
| on, only allowed by one user at a time. It's almost a
| requirement (b) to do this with a plugin in the CAD software
| itself, because hierarchies within CAD assembly files are
| used to select, open, etc files, and because CAD files are
| often just names with a consecutive numbering scheme. It's
| quite painful to have to use a different interface to locate
| and checkout/checkin files.
|
| If you can offer these two things you may have a decent
| market for small companies using CAD. Self-hosting (c) is a
| requirement for many industries. A file viewer (d) is nice to
| have but not at all required, and every other software
| attempts to add JIRA-type features (e) which frankly just
| becomes another place other than the correct place to
| document things.
|
| The only software I've seen which really hit the mark was
| grabcad workbench, which is unfortunately being
| discontinued[1]. It did just a, b, and d and it was free too!
|
| The offerings from actual CAD vendors are way too heavy, need
| dedicated roles to maintain them, and cost more than CAD
| licenses. Kenesto[2] does a, b, d and and Bild[3] does a, d
| only. Kenesto looks good but I've heard complaints that it
| feels barely maintained. Bild looked promising but it's slow
| and not only is it lacking b, a plugin for CAD, you have to
| use its webview application which offers no search, preview
| or sorting options so you have spend your time scrolling
| through a list of files. And of course, both offer way too
| many unwanted features. Everyone selling version control
| software for CAD doesn't seem to realize that we don't want
| to them to compete with Siemens on features. We want a few
| features done well, and already have tools or workflows for
| tickets, approvals, commenting, etc.
|
| [1] https://blog.grabcad.com/blog/2022/08/09/a-fond-farewell-
| to-...
|
| [2] https://www.kenesto.com/buynow
|
| [3] https://www.getbild.com/plans
___________________________________________________________________
(page generated 2023-01-19 23:00 UTC)