[HN Gopher] SnowFS - a fast, scalable version control file stora...
       ___________________________________________________________________
        
       SnowFS - a fast, scalable version control file storage for graphic
       files
        
       Author : karateka
       Score  : 54 points
       Date   : 2021-02-21 07:16 UTC (15 hours ago)
        
 (HTM) web link (github.com)
 (TXT) w3m dump (github.com)
        
       | pjmlp wrote:
       | JavaScript[0] isn't what I would call a fast storage repository,
       | but I guess it works out for prototyping.
       | 
       | On the context of porting to C and C++, or make it execute
       | faster, I can see two options with minor rewrites.
       | 
       | Use AssemblyScript and generate native code via WebAssembly AOT
       | compilers.
       | 
       | Try to adapt TypeScript to C++ compiler from Microsoft's MakeCode
       | project.
       | 
       | Implement your own C++ code generator.
       | 
       | It would be much easier than keep using multiple code bases in
       | parallel, plus any memory corruption issues would be most likely
       | bugs on the code generator.
       | 
       | [0] - Yes I know the source code is Typescript.
        
         | sebastian_io wrote:
         | Indeed, TS/JS is great for its quick turnaround times for
         | prototyping. But the I/O performance is executed by the
         | underlying C/C++ layer. For the rest TS/JS is fast enough. But
         | a full C++ backport is still on the horizon
        
       | amelius wrote:
       | But what if I have software _and_ images (e.g. as in game
       | software)? Which version control system do I use?
       | 
       | Honestly, I think the effort was better spent on an improved
       | version of Git.
        
         | sebastian_io wrote:
         | Programmers, game makers and 2D/3D artists are very different
         | target audiences with very different needs. To name one
         | example, the commit hash integrity which is a foundation of Git
         | is a must-have for software projects, but might be not useful
         | in environments like VFX or CG productions
        
           | amelius wrote:
           | > Programmers, game makers and 2D/3D artists are very
           | different target audiences with very different needs.
           | 
           | Yes, so if you want to make them use the same version control
           | system (not a strange requirement, see Google monorepo), then
           | the system should be all-encompassing, and not limited to one
           | type of user.
           | 
           | And it may turn out that features meant for one type of user
           | are also useful for other types of user. For example,
           | programmers may need to store large image files as part of a
           | deep-learning image classification system.
           | 
           | I'm usually not a fan of packing as many features as possible
           | into a single tool, but in case of version control it makes
           | sense. (A pluggable/modularized approach should of course be
           | used if possible).
        
             | sebastian_io wrote:
             | You are absolutely right! A tool that can cover everyones
             | needs is a great goal, and a pluggable or modularized
             | approach might be even the best. SnowFS is not a one-way
             | street, there are certain features that are currently
             | investigated to be feature-proposed for libgit2. Means,
             | this project is a great way to explore the possibilities
             | without any historical or technical barriers because it is
             | so lightweight. Best proof are the benchmarks
        
           | mbreese wrote:
           | Also, the project is very clear that it is for individuals
           | with no collaboration tools built in. While git does have
           | utility for a solo developer, having a separate system for a
           | solo artist (that isn't working with assets for developers),
           | starts make more sense.
        
         | lhoff wrote:
         | Maybe a mixed approach similar to what dvc [0] is doing. Fully
         | integrated with git but tracked and stored elsewhere.
         | 
         | [0] https://dvc.org/
        
         | iaml wrote:
         | AFAIK big studios use perforce.
        
       | jarym wrote:
       | I need to look into this more but unless you already know C/C++
       | really well then Rust may be a better choice.
        
         | sebastian_io wrote:
         | Same here, the aimed goal is a C++ backport, bc I am a C++ dev.
         | But Rust is on my radar and happy to learn
        
       | kevlar1818 wrote:
       | Very interesting. I'd like to learn more about how it works. How
       | does this compare to DVC[1], for instance?
       | 
       | I'll throw in a shameless plug for my tool in this area, Dud[2].
       | Dud is to DVC what Flask is to Django.
       | 
       | Are the mentioned benchmarks published somewhere?
       | 
       | [1]: https://dvc.org [2]: https://github.com/kevin-hanselman/dud
        
         | sebastian_io wrote:
         | SnowFS is still at the beginnings, so it will shape over time.
         | For the benchmarks, these are the results on a MacBook-2019
         | with APFS for a 4GB file:
         | 
         | [git add texture.psd: 20164ms] [snow add texture.psd: 4596ms]
         | [git rm texture.psd: 575ms] [snow rm texture.psd: 111ms] [git
         | checkout HEAD~1: 9739ms] [snow checkout HEAD~1: 1ms]
         | 
         | You might get slightly slower speeds on NTFS for 'add' and
         | 'checkout' but still very performant
        
       ___________________________________________________________________
       (page generated 2021-02-21 23:01 UTC)