[HN Gopher] Snowtrack - a new version control software for graph...
___________________________________________________________________
Snowtrack - a new version control software for graphic designers
and 3D artists
Author : sebastian_io
Score : 137 points
Date : 2021-06-08 09:06 UTC (13 hours ago)
(HTM) web link (snowtrack.io)
(TXT) w3m dump (snowtrack.io)
| twic wrote:
| Your repo has a section "Why not Git/Git-LFS, libgit2, or SVN?".
| Can i suggest you add a comparison to Mercurial's largefiles
| extension:
|
| https://www.mercurial-scm.org/wiki/LargefilesExtension
|
| Since, last time i looked into it, which was a while ago, that
| was the best option for this.
| sebastian_io wrote:
| Great idea! I've to admit my knowledge about Mercurial is very
| rusty. Could you help out on that? May I invite you to share
| some insights in the discussion board of the project?
| https://github.com/Snowtrack/SnowFS/discussions
| midnightclubbed wrote:
| I would also be curious how it compares to Perforce which is
| widely used for version control of binary art assets (pretty
| much ubiquitous in the games industry).
| redman25 wrote:
| On Mac, it would be nice if you could work with files completely
| inside Snowtrack instead of having to jump to Finder and back.
| Also, dragging a file onto Snowtrack seems like it should add the
| file to the current project. Otherwise, I love the idea for a
| simple and straightforward VCS tool.
| sebastian_io wrote:
| Thanks for your feedback! Better D&D handling was already
| requested a few times and will certainly be added in the next
| versions
| blacksmith_tb wrote:
| Hmm, version requirements seem odd? "Apple macOS 14.0 (or
| higher)" we're only on 10.15.x or 11, with 12 on the
| horizon...[1]
|
| 1:
| https://en.wikipedia.org/wiki/MacOS_version_history#Version_...
| sebastian_io wrote:
| I'm from the future. ;-) But I will fix it, thanks for the info
| nicolasrusso wrote:
| My question is, if you're running this with blender, say, does it
| simply save the same file over and over again internally or in a
| hidden way somewhere? If that's the case, that's a lot of waste
| of space for me.
| Grumbledour wrote:
| This looks really interesting. But Typescript and "Git uses a
| restrictive license" give me pause. Maybe it's a cultural thing.
| I'll be hesitantly optimistic I guess.
| sebastian_io wrote:
| In this case it's a comparison between MIT and GPL. GPL
| software is more restrictive than licenses like MIT/Apache/..
| and therefore not as much as present in the VFX or CG industry
| and that's key here. But you are absolutely right! The wording
| is potentially misleading and should be improved. I'll fix it
| southerntofu wrote:
| Depends on how you look at it. GPL is more permissive in that
| it allows you to contribute to and benefit from evolutions of
| the software. While MIT doesn't permit you to reuse code that
| was expanded on by huge corporations.
|
| Of course, you can frame the debate the other way around, i'm
| just making the point that "permissive/restrictive" means
| very little in this context and what matters really as a
| question is "do you wish huge corporations to benefit from
| your free-software work and expand on it to develop their own
| business, without EVER contributing anything back"? If yes,
| choose MIT and you'll probably regret it like so many open-
| source maintainers. Otherwise, choose (a)GPL and you probably
| won't regret it (at least i've never met someone so far who
| regretted choosing copyleft for their works, but it may exist
| of course).
| sebastian_io wrote:
| Just for clarification, there are two projects involved
| here. SnowFS is the underlying open-source VCS, and
| Snowtrack is the commercial UI build around it.
|
| But for your statement about licenses, there is actually
| not much I can say to counter your arguments. I've followed
| the fight between Amazon and Elastic so you have very valid
| points. For the moment I don't think SnowFS will face the
| same issues given its current scope and target
| audience/market
| phkahler wrote:
| >> Just for clarification, there are two projects
| involved here. SnowFS is the underlying open-source VCS,
| and Snowtrack is the commercial UI build around it.
|
| This discussion sounds like they were afraid to build
| Snowtrack on top of git.
|
| I'm not a fan of the names of git commands and some of
| the other details about it, but functionally it works
| well and is the industry standard. To avoid it because of
| the GPL doesn't seem right. There are many commercial
| tools built of git.
| ajford wrote:
| Git doesn't work well for binary files, in particular
| large binary files.
|
| This is the reason so few VFX, graphics, and game devs
| use Git for anything that needs to version large binary
| files.
|
| Git LFS can help, but it's not always the right solution.
| sebastian_io wrote:
| You're right. Git is widely used but I wouldn't call it
| an industry standard (yet), especially not not for
| repositories that exclusively contain binary data. And
| the GPL works well with commercial products. On top there
| is libgit2 which is a fantastic alternative as well. But
| Git doesn't fulfill the technical requirements for
| Snowtrack
| twic wrote:
| "Permissive" at least is an established term of art,
| meaning a license which does not impose many conditions on
| redistribution:
|
| https://fossa.com/blog/all-about-permissive-licenses/
|
| Even the GNU project uses it that way:
|
| https://www.gnu.org/licenses/license-list.en.html
|
| You're right that "restrictive" is an unusual term in this
| context. The antonym of "permissive" is usually "copyleft".
| sfvisser wrote:
| Makes me happy to see system tools being written in typescript!
| sebastian_io wrote:
| I absolutely agree. When I started evaluating the project I
| was positively surprised about the performance of Node for
| this field. It was a perfect fit so I continued with it
| sshumaker wrote:
| I built tech like this at a startup in 2010. It was aimed at
| musicians but basically the same idea - transparent version
| control of all of their DAW projects and easy collaboration
| (sending of huge projects, hard a decade ago). In this case it
| actually indexed all of the drives (musicians use attached media
| a lot) and maintained an online searchable archive with previews
| as well. We'd listen for file system notifications and there was
| enough logic to understand project directory structure, flac-
| compressing the audio (while preserving metadata so the it's
| could be perfectly reconstructed).
|
| This was only a means to an end though - it's clear deeper
| integration with these products works better for a number of
| reasons, and a much better collaboration workflow isn't just
| sending entire directories around. Look at Figma for an example
| of that - they reinvented the entire product space around collab.
|
| Eventually the company (https://gobbler.com/) pivoted the
| business model away from this space into plugins.
|
| I'm not convinced the opaque file approach can provide / capture
| enough value to be a sustainable business over the next 5-10
| years.
| stereosteve wrote:
| Yeah the company I work for (https://www.copia.io/) is doing
| similar things for Programmable Logic Controllers, which
| historically have binary project files.
|
| Also a means to an end given the current sad reality. But most
| vendors are slowly moving towards better VCS support.
| sebastian_io wrote:
| There is a lot going on in the space right now and there will
| be certainly a big shift in the next years. But as long as
| Blender, Cinema 4D, Photoshop, Pixelmator, Affinity are not
| running in the cloud I rather feel like DCC software packages
| and tablets will introduce a change first
| stereosteve wrote:
| From a quick read of the SnowFS source code, it looks like it
| splits large files into 100Mb blocks and builds up a zip of
| blocks over time. A version of a file is an ordered list of
| hashes for the blocks in that version.
|
| I like the simplicity of this! But is it at all problematic if
| something changes early in the file and all the subsequent blocks
| boundaries shift causing many new blocks to be created?
|
| rsync uses a sliding window to handle this situation. The
| implementation would be more complicated, but have you considered
| using librsync internally?
| sebastian_io wrote:
| I am currently working on the compression, as it is not
| complete yet. The 100 MB is indeed excessive but the window is
| dynamic and can differ from file to file since it is written to
| a `*.hblock` file which is stored next to the object in the
| object database
| https://github.com/Snowtrack/SnowFS/blob/03e5f839326e666c891...
|
| Let me explain where the 100 MB window comes from as its not
| only related to the upcoming compression implementation. Some
| graphic applications touch the timestamps of their files for no
| reason, making it harder to detect if a file changed. But some
| file formats always change their 'header' or 'footer'. Means,
| comparing the hash of the first or last 100 MB of a file that
| is 8GB in size gives a great performance boost to detect if a
| file got modified.
| high_byte wrote:
| cool. although I think with 4mb window it would be more
| efficient. 100mb seems excessive, then I assume you wouldn't
| need a sliding window. (if it works well enough for 100mb)
| stereosteve wrote:
| the problem happens with any fixed window spacing regardless
| of the block size.
|
| If you create a block every Xmb... inserting a single byte at
| the beginning of the file will change every subsequent block.
| 411111111111111 wrote:
| You're technically speaking wrong, but I'm sure the author
| doesn't want to reimplement block storage devices... So the
| spirit of the message is probably correct
| stereosteve wrote:
| Oh I'm not talking about disks... this is based on how
| SnowFS (the library for this project) splits up big files
| into chunks:
|
| https://github.com/Snowtrack/SnowFS/blob/main/src/common.
| ts#...
|
| The intent is a simple form of delta encoding, the hope
| is that many chunks will be common between two versions.
| sebastian_io wrote:
| I should clarify this. The 100 MB window in SnowFS is
| currently unrelated to compression as it is only used to
| compare if a block changed. Each block gets a hash. This
| is a fallback used for some file formats where the mtime
| timestamp cannot be trusted. Some files have a change in
| the first block e.g. 100 MB and that is faster to compare
| than an entire 8GB file. But this window size is dynamic
| and can be changed and used for compression in the future
| stereosteve wrote:
| Ahh this is my bad. For some reason I assumed the blocks
| were part of the storage scheme, but I see they only are
| used to compute hash, and that the whole file is added to
| zip. Sorry for the misunderstanding!
| [deleted]
| [deleted]
| digikata wrote:
| There's a large set of different algorithms with a sliding
| window. Another interesting one is the Rabin fingerprint. This
| kind of chunking is often used in storage file systems w/
| deduplication and snapshot features.
|
| https://en.wikipedia.org/wiki/Rabin_fingerprint
| RunawayGalaxy wrote:
| What would a workflow look like with Figma?
| chadlavi wrote:
| figma already has pretty robust version tracking built in,
| though it's not necessarily too discoverable
| sebastian_io wrote:
| Snowtrack focuses on local file versioning with tools that are
| locally installed (e.g. Photoshop, Blender, Cinema 4D,
| Pixelmator, ...)
| southerntofu wrote:
| That's really cool! What's the business model here? A lot of
| artists, especially in free-software/free-culture communities
| could use tooling like that.
|
| If you're willing to make it a proper donation-run project with a
| commitment to free-software, i'm sensing a great future like
| GIMP/Blender has :) Good luck on it, can't wait to try it out
| once it's packaged on Debian
| sebastian_io wrote:
| Thank you! The project consists out of two projects. SnowFS is
| the underlying core and is available as an open-source project
| (MIT). Snowtrack is the UI and the commercial product build
| around it and will be available as a Freemium product
| iFire wrote:
| Wow, reviewing. I am in the games industry.
| midnightclubbed wrote:
| Can definitely see the usefulness of this for artists, especially
| if there is integration into the various art programs.
|
| Unless I am mis-understanding this is only a solution for
| organizing local files? Ideally there would be a distributed
| version control component or integration with an existing version
| control system. I can see this lulling users into a false sense
| of security that their local data is backed-up and safe.
| sebastian_io wrote:
| Thanks for your input! Snowtrack has no collaboration features
| built in yet. The public beta of Snowtrack is available since
| today. The project has a lot of fields to discover and will
| evolve over time. There are certainly countless of areas
| untouched yet
| danpalmer wrote:
| This site fails to load in Firefox, with an error about there
| being no overlapping TLS ciphers available. Using the up to date
| public stable release of Firefox, no other TLS issues elsewhere,
| I wonder what has gone wrong.
| sebastian_io wrote:
| Thanks for the feedback! I will look into this
| sebastian_io wrote:
| I checked this on Windows and macOS with the latest version
| of Firefox and I have no issues opening the page
| makeworld wrote:
| Works fine in Firefox on Linux for me.
| darkstarsys wrote:
| This looks really interesting. I read through the docs on the
| site though, and it doesn't say anything about diffs and merges.
| Those are fundamental VCS operations so I hope it supports them.
| I recognize merging is complex for 2d/3d files but if it claims
| to be a VCS then I'd think it would have to support it, otherwise
| what use are branches?
| [deleted]
| pvitz wrote:
| How do you define a diff or merge for such files? Having a
| history available and branches (e.g. alternative versions of
| the same image) that can't be merged back seems to me to be
| still a version-control system.
| darkstarsys wrote:
| It depends on how deep the file parser is. For 2d, the zero-
| level feature would just be pixelwise image diff of the
| rendered result. For photoshop or inkscape, you could diff
| layers, show added/deleted/changed adjustments, etc.
|
| For 3d, to take Blender as an example, a zero-level diff
| feature would be to export (e.g. as glTF) and show which
| scene objects differ (meshes, textures, lights, etc.) More
| detailed parsing would give more details, e.g. image-diffing
| modified textures.
|
| As I noted, merging is nontrivial. But it seems important --
| if I save a branch, go back to the trunk and do more work,
| how do I add in the changes I made on the branch? Or even
| know what those changes were? Good diffs would help of
| course.
|
| Merging would certainly have to be product-specific. For some
| open-source tools one could imagine hooking into their
| undo/redo logic, or parsing the product files into a
| temporary line-based format, but from outside it seems much
| more challenging than line-based merge in git (which is
| already imperfect for many applications).
| pvitz wrote:
| I think this is one of those things that superficially
| sound not that complicated but when you look closer, it is
| a gargantuan complex task. If I think only about Photoshop
| and PSD, this seems terrible to implement and maintain.
|
| Also, I guess that the need is not really there, because
| otherwise, Adobe or some other company would have
| implemented this in the past 30 years.
| sebastian_io wrote:
| Thank you for your feedback. You brought up very good points.
| Snowtrack is available as a public beta since today and will
| evolve from here through user-requests and feedback. Snowtrack
| supports a simple before/after compare but this feature will be
| heavily extended with the next versions.
|
| About merge operations: The majority of solo artists and
| designers don't work in many branches. Most projects just go
| forward and branches are only useful if a rollback is needed.
| That being said, Snowtrack does offer a simplified "cherry-
| pick". But merge operations as you might know them from
| Git/Perforce are not implemented yet as it takes time to see
| how users adapt the software.
|
| Keep in mind, most artists and designers are new to VCS or have
| never heard of it before and therefore it is crucial to find
| the right balance between too little and too many features. Any
| second that keeps the artist away from creating their artwork
| is wasted and therefore and the goal is to not interfere with
| their workflow or to add additional "management tasks"
| maddyboo wrote:
| This looks great! Are you planning to support Linux?
| sebastian_io wrote:
| Thank you! I've already prepared the foundation for a Linux
| build, and it's planned in the near future but I don't have a
| build available yet
| detritus wrote:
| How is this an improvement over a myriad of confusing and
| recursively-named nested folders with repeated copies of existing
| content in ever-growing names like "final_final-
| FINAL-05-CLIENTEXPORT"?
| sebastian_io wrote:
| You have a point there :-D
| smaddox wrote:
| Why don't graphic artist just use hard links and ZFS dedup.
|
| /s
| QuadrupleA wrote:
| Some comments here about delta compression on image assets and
| compressed formats - this page from the Fossil SCM has a really
| interesting discussion of the issues involved: https://fossil-
| scm.org/home/doc/trunk/www/image-format-vs-re...
| sebastian_io wrote:
| This is incredibly helpful as I was looking for something like
| this for weeks. Thanks a lot for sharing! This looks super
| useful
| stereosteve wrote:
| Yeah this is cool analysis!
|
| Looking at Fossil's delta code it uses 16 byte blocks and a
| sliding window when finding deltas:
|
| https://fossil-scm.org/home/file?ci=trunk&name=src/delta.c&l...
| FrankyFire wrote:
| Love the idea. I was thinking about creating something similar
| (without any knowledge on how to achieve that, I'm not a good
| programmer) for music creation and especially collaboration.
|
| When writing a song with someone else, this can become really
| handy, as we sometimes don't know if some idea might work.
|
| I would totally be a customer (if its not bound to a monthly
| payment). But as of now, it's lacking linux support (why do I
| have to be that special unicorn all the time?).
|
| By the way: BandLab has something like this integrated already,
| but I want to use my own DAW.
| sebastian_io wrote:
| I received a lot of feedback by people with music background. I
| recently talked to someone from the music production industry
| and it was interesting to hear that they have the same workflow
| and problems as the graphics industry. Snowtrack could be used
| with any DAW but the missing thumbnails are a downer at the
| moment
| bellyfullofbac wrote:
| I thought it was just a pretty UI on top of git, the bit about
| "It's good for large binary files!" is hidden under the link to
| the SnowFS github project...
|
| Since the HN headline says "3D artists" I was expecting to see an
| app that has a 3D viewer that can show 2 versions of a 3d scene
| on top^W^W inside each other and you can see toggle to see the
| differences...
| wodenokoto wrote:
| > top^W^W
|
| Is this an abbreviation or a reference to a command or what
| does it mean?
| bellyfullofbac wrote:
| > 3d scene on top^W^W inside each other
|
| Ctrl-W is delete previous word on some terminals, so I wrote
| "3d scene on top [of] each other" and decided saying "3d
| scene inside each other" made more sense.
| PeterHacker123 wrote:
| that looks neat, does it store the files locally?
| sebastian_io wrote:
| Correct
| jayd16 wrote:
| Does this only work at the project snapshot level? My artists
| prefer P4 over git with the major reason being the ability to
| work on single files at a time (ie sparse check out).
|
| Auto-commit is neat.
| sebastian_io wrote:
| Unfortunately, Snowtrack has no collaboration workflow yet. The
| underlying file storage system is prepared to support
| collaboration in the future, but it's too early for this
| rsstack wrote:
| https://web.archive.org/web/20210608090928/https://snowtrack...
|
| "Snowtrack is based on our open-source community project":
| https://github.com/snowtrack/snowfs
___________________________________________________________________
(page generated 2021-06-08 23:02 UTC)