[HN Gopher] JuiceFS is a distributed POSIX file system built on ...
___________________________________________________________________
JuiceFS is a distributed POSIX file system built on top of Redis
and S3
Author : tosh
Score : 77 points
Date : 2026-01-15 18:45 UTC (4 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| Plasmoid wrote:
| I was actually looking at using this to replace our mongo disks
| so we could easily cold store our data
| Eikon wrote:
| ZeroFS [0] outperforms JuiceFS on common small file workloads [1]
| while only requiring S3 and no 3rd party database.
|
| [0] https://github.com/Barre/ZeroFS
|
| [1] https://www.zerofs.net/zerofs-vs-juicefs
| huntaub wrote:
| Respect to your work on ZeroFS, but I find it kind of off-
| putting for you to come in and immediately put down JuiceFS,
| especially with benchmark results that don't make a ton of
| sense, and are likely making apples-to-oranges comparisons with
| how JuiceFS works or mount options.
|
| For example, it doesn't really make sense that "92% of data
| modification operations" would fail on JuiceFS, which makes me
| question a lot of the methodology in these tests.
| Eikon wrote:
| > but I find it kind of off-putting for you to come in and
| immediately put down JuiceFS, especially with benchmark
| results that don't make a ton of sense, and are likely making
| apples-to-oranges comparisons with how JuiceFS works or mount
| options.
|
| The benchmark suite is trivial and opensource [1].
|
| Is performing benchmarks "putting down" these days?
|
| If you believe that the benchmarks are unfair to juicefs for
| a reason or for another, please put up a PR with a better
| methodology or corrected numbers. I'd happily merge it.
|
| EDIT: From your profile, it seems like you are running a VC
| backed competitor, would be fair to mention that...
|
| [1] https://github.com/Barre/ZeroFS/tree/main/bench
| huntaub wrote:
| Yes, I'm working in the space too. I think it's fine to do
| benchmarks, I don't think it's necessary to immediately
| post them any time a competitor comes up on HN.
|
| I don't want to see the cloud storage sector turn as bitter
| as the cloud database sector.
|
| I've previously looked through the benchmarking code, and I
| still have some serious concerns about the way that you're
| presenting things on your page.
| zaphirplane wrote:
| > presenting things
|
| I don't have a dog in this race, have to say thou the
| vagueness of the hand waving in multiple comments is
| losing you credibility
| wgjordan wrote:
| > The benchmark suite is trivial and opensource.
|
| The actual code being benchmarked is trivial and open-
| source, but I don't see the actual JuiceFS setup anywhere
| in the ZeroFS repository. This means the self-published
| results don't seem to be reproducible by anyone looking to
| externally validate the stated claims in more detail. Given
| the very large performance differences, I have a hard time
| believing it's an actual apples-to-apples production-
| quality setup. It seems much more likely that some simple
| tuning is needed to make them more comparable, in which
| case the takeaway may be that JuiceFS may have more fiddly
| configuration without well-rounded defaults, not that it's
| actually hundreds of times slower when properly tuned for
| the workload.
|
| (That said, I'd love to be wrong and confidently discover
| that ZeroFS is indeed that much faster!)
| selfhoster1312 wrote:
| I have very limited experiences with object storage, but my
| humble benchmarks with juicefs + minio/garage [1] showed very
| bad performance (i.e. total collapse within a few hours) when
| running lots of small operations (torrents).
|
| I wouldn't be surprised if there's a lot of tuning that can
| be achieved, but after days of reading docs and experimenting
| with different settings i just assumed JuiceFS was a very bad
| fit for archives shared through Bittorrent. I hope to be
| proven wrong, but in the meantime i'm very glad zerofs was
| mentioned as an alternative for small files/operations. I'll
| try to find the time to benchmark it too.
|
| [1] https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021
| eYrKEC2 wrote:
| I'm always curious about the of the option space. I
| appreciate folks talking about the alternative s. What's
| yours?
| huntaub wrote:
| Our product is Archil [1], and we are building our service
| on top of a durable, distributed SSD storage layer. As a
| result, we have the ability to: (a) store and use data in
| S3 in its native format [not a block based format like the
| other solutions in this thread], (b) durably commit writes
| to our storage layer with lower latency than products which
| operate as installable OSS libraries and communicate with
| S3 directly, and (c) handle multiple writers from different
| instances like NFS.
|
| Our team spent years working on NFS+Lustre products at
| Amazon (EFS and FSx for Lustre), so we understand the
| performance problems that these storage products have
| traditionally had.
|
| We've built a custom protocol that allows our users to
| achieve high-performance for small file operations (git --
| perfect for coding agents) and highly-parallel HPC
| workloads (model training, inference).
|
| Obviously, there are tons of storage products because
| everyone makes different tradeoffs around durability, file
| size optimizations, etc. We're excited to have an approach
| that we think can flex around these properties dynamically,
| while providing best-in-class performance when compared to
| "true" storage systems like VAST, Weka, and Pure.
|
| [1] https://archil.com
| corv wrote:
| Looks like the underdog beats it handily and easier deployment
| to boot. What's the catch?
| aeblyve wrote:
| ZeroFS is a single-writer architecture and therefore has
| overall bandwidth limited by the box it's running on.
|
| JuiceFS scales out horizontally as each individual client
| writes/reads directly to/from S3, as long as the metadata
| engine keeps up it has essentially unlimited bandwidth across
| many compute nodes.
|
| But as the benchmark shows, it is fiddly especially for
| workloads with many small files and is pretty wasteful in
| terms of S3 operations, which for the largest workloads has
| meaningful cost.
|
| I think both have their place at the moment. But the space of
| "advanced S3-backed filesystems" is... advancing these days.
| wgjordan wrote:
| For a proper comparison, also significant to note that JuiceFS
| is Apache-2.0 licensed while ZeroFS is dual AGPL-3.0/commercial
| licensed, significantly limiting the latter's ability to be
| easily adopted outside of open source projects.
| anonymousDan wrote:
| Why would this matter if you're just using the database?
| Eikon wrote:
| It doesn't, you are free to use ZeroFS for commercial and
| closed source products.
| wgjordan wrote:
| This clarification is helpful, thanks! The README
| currently implies a slightly different take, perhaps it
| could be made more clear that it's suitable for use
| unmodified in closed source products:
|
| > The AGPL license is suitable for open source projects,
| while commercial licenses are available for organizations
| requiring different terms.
|
| I was a bit unclear on where the AGPL's network-
| interaction clause draws its boundaries- so the
| commercial license would only be needed for closed-source
| modifications/forks, or if statically linking ZeroFS
| crate into a larger proprietary Rust program, is that
| roughly it?
| Eikon wrote:
| > so the commercial license would only be needed for
| closed-source modifications/forks
|
| Indeed.
| wgjordan wrote:
| Also worth noting (as a sibling comment pointed out) that
| despite these assurances the untested legal risks of
| AGPL-licensed code may still cause difficulties for
| larger, risk-averse companies. Google notably has a
| blanket policy [1] banning all AGPL code entirely as "the
| risks outweigh the benefits", so large organizations are
| probably another area where the commercial license comes
| into play.
|
| [1] https://opensource.google/documentation/reference/usi
| ng/agpl...
| ChocolateGod wrote:
| Let's remember that JuiceFS can be setup very easily to not
| have a single point of failure (by replicating the metadata
| engine), meanwhile ZeroFS seems to have exactly that.
|
| If I was a company I know which one I'd prefer.
| dpacmittal wrote:
| The magnitude of performance difference alone immediately makes
| me skeptical of your benchmarking methodology.
| selfhoster1312 wrote:
| I'm not an expert in any way, but i personally benchmarked
| [1] juiceFS performance totalling collapsing under very small
| files/operations (torrenting). It's good to be skeptical, but
| it might just be that the bar is very low for this specific
| usecase (IIRC juiceFS was configured and optimized for block
| sizes of several MBs).
|
| https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1021
| maxmcd wrote:
| does having to maintain the slatedb as a consistent singleton
| (even with write fencing) make this as operationally tricky as
| a third party db?
| Eikon wrote:
| It's not great UX on that angle. I am currently working on
| coordination (through s3, not node to node communication), so
| that you can just spawn instances without thinking about it.
| victorbjorklund wrote:
| Can SQLite run on it?
| Dylan16807 wrote:
| > ZeroFS supports running multiple instances on the same
| storage backend: one read-write instance and multiple read-only
| instances.
|
| Well that's a big limiting factor that needs to be at the front
| in any distributed filesystem comparison.
|
| Though I'm confused, the page says things like "ZeroFS makes S3
| behave like a regular block device", but in that case how do
| read-only instances mount it without constantly getting their
| state corrupted out from under them? Is that implicitly talking
| about the NBD access, and the other access modes have logic to
| handle that?
|
| Edit: What I want to see is a ZeroFS versus s3backer
| comparison.
|
| Edit 2: changed the question at the end
| wgjordan wrote:
| Related, "The Design & Implementation of Sprites" [1] (also
| currently on the front page) mentioned JuiceFS in its stack:
|
| > The Sprite storage stack is organized around the JuiceFS model
| (in fact, we currently use a very hacked-up JuiceFS, with a
| rewritten SQLite metadata backend). It works by splitting storage
| into data ("chunks") and metadata (a map of where the "chunks"
| are). Data chunks live on object stores; metadata lives in fast
| local storage. In our case, that metadata store is kept durable
| with Litestream. Nothing depends on local storage.
|
| [1] https://news.ycombinator.com/item?id=46634450
| IshKebab wrote:
| Interesting. Would this be suitable as a replacement for NFS? In
| my experience literally everyone in the silicon design industry
| uses NFS on their compute grid and it sucks in numerous ways:
|
| * poor locking support (this sounds like it works better)
|
| * it's slow
|
| * no manual fence support; a bad but common way of distributing
| workloads is e.g. to compile a test on one machine (on an NFS
| mount), and then use SLURM or SGE to run the test on other
| machines. You use NFS to let the other machines access the
| data... and this works... except that you either have to disable
| write caches or have horrible hacks to make the output of the
| first machine visible to the others. What you _really_ want is a
| manual fence: "make all changes to this directory visible on the
| server"
|
| * The bloody .nfs000000 files. I think this might be fixed by
| NFSv4 but it seems like nobody actually uses that. (Not helped by
| the fact that CentOS 7 is considered "modern" to EDA people.)
| huntaub wrote:
| > * The bloody .nfs000000 files. I think this might be fixed by
| NFSv4 but it seems like nobody actually uses that. (Not helped
| by the fact that CentOS 7 is considered "modern" to EDA
| people.)
|
| Unfortunately, NFSv4 also has the silly rename semantics...
| jabl wrote:
| AFAIU the NFSv4 protocol in principle allows implementing
| unlinking an open file without silly rename, but the Linux
| client still does the silly rename dance.
| mrkurt wrote:
| FUSE is full of gotchas. I wouldn't replace NFS with JuiceFS
| for arbitrary workloads. Getting the full FUSE set implemented
| is not easy -- you can't use sqlite on JuiceFS, for example.
|
| The meta store is a bottleneck too. For a shared mount, you've
| got a bunch of clients sharing a metadata store that lives in
| the cloud somewhere. They do a lot of aggressive metadata
| caching. It's still surprisingly slow at times.
| huntaub wrote:
| > FUSE is full of gotchas
|
| I want to go ahead and nominate this for the understatement
| of the year. I expect that 2026 is going to be filled with
| people finding this out the hard way as they pivot towards
| FUSE for agents.
| dpe82 wrote:
| Mind helping us all out ahead of time by expanding on what
| kind of gotchas FUSE is full of?
| huntaub wrote:
| It depends on what level of FUSE you're working with.
|
| If you're running a FUSE adapter provided by a third
| party (Mountpoint, GCS FUSE), odds are that you aren't
| going to get great performance because it's going to have
| to run across a network super far away to work with your
| data. To improve performance, these adapters need to be
| sure to set fiddly settings (like using Kernel-side
| writeback caching) to avoid the penalty of hitting the
| disk for operations like write.
|
| If you're trying to write a FUSE adapter, it's up to you
| to implement as much of the POSIX spec that you need for
| the programs that you want to run. The requirements per-
| program are often surprising. Want to run "git clone",
| then you need to support the ability to unlink a file
| from the file system and keep its data around. Want to
| run "vim", you need the ability to do renames and hard
| links. All of this work needs to happen in-memory in
| order to get the performance that applications expect
| from their file system, which often isn't how these
| things are built.
|
| Regarding agents in particular, I'm hopeful that someone
| (which is quite possibly us), builds a FUSE-as-a-service
| primitive that's simple enough to use that the vast
| majority of developers don't have to worry about these
| things.
| IshKebab wrote:
| > you need to support the ability to unlink a file from
| the file system and keep its data around. Want to run
| "vim", you need the ability to do renames and hard links
|
| Those seem like pretty basic POSIX filesystem features to
| be fair. Awkward, sure... there's also awkwardness like
| symlinks, file locking, sticky bits and so on. But these
| are just things you have to implement. Are there gotchas
| that are inherent to FUSE itself rather than FUSE
| implementations?
| huntaub wrote:
| These are basic POSIX features, but I think the high-
| level point that Kurt is trying to make is that building
| a FUSE file system signs you up for a nearly unlimited
| amount of compatibility work (if you want to support most
| applications) whereas their approach (just do a loopback
| ext4 fs into a large file) avoids a lot of those
| problems.
|
| My expectations are that in 2026 we will see more and
| more developers attempt to build custom FUSE file systems
| and then run into the long tail of compatibility pain.
| IshKebab wrote:
| > just do a loopback ext4 fs into a large file
|
| How does that work with multiple clients though?
| huntaub wrote:
| tl;dr it doesn't. I'm not sure what they're planning in
| this capacity (I haven't checked out sprites myself), but
| I would guess that it's going to be a function of
| "snapshots" as a mechanism to give multiple clients
| ephemeral write access to the same disk.
| jabl wrote:
| > poor locking support (this sounds like it works better)
|
| File locking on Unix is in general a clusterf*ck. (There was a
| thread a few days ago at
| https://news.ycombinator.com/item?id=46542247 )
|
| > no manual fence support; a bad but common way of distributing
| workloads is e.g. to compile a test on one machine (on an NFS
| mount), and then use SLURM or SGE to run the test on other
| machines. You use NFS to let the other machines access the
| data... and this works... except that you either have to
| disable write caches or have horrible hacks to make the output
| of the first machine visible to the others. What you really
| want is a manual fence: "make all changes to this directory
| visible on the server"
|
| In general, file systems make for poor IPC implementations. But
| if you need to do it with NFS, the key is to understand the
| close-to-open consistency model NFS uses, see section 10.3.1 in
| https://www.rfc-editor.org/rfc/rfc7530#section-10.3 . Of
| course, you'll also want some mechanism for the writer to
| notify the reader that it's finished, be it with file locks, or
| some other entirely different protocol to send signals over the
| network.
| IshKebab wrote:
| > In general, file systems make for poor IPC implementations.
|
| I agree but also they do have advantages such as simplicity,
| not needing to explicitly declare which files are needed,
| lazy data transfer, etc.
|
| > you'll also want some mechanism for the writer to notify
| the reader that it's finished, be it with file locks, or some
| other entirely different protocol to send signals over the
| network.
|
| The writer is always finished before the reader starts in
| these scenarios. The issue is reads on one machine aren't
| guaranteed to be ordered after writes on a different machine
| due to write caching.
|
| It's exactly the same problem as trying to do multithreaded
| code. Thread A writes a value, thread B reads it. But even if
| they happen sequentially in real time thread B can still read
| an old value unless you have an explicit fence.
| xorcist wrote:
| > NFSv4 but it seems like nobody actually uses that
|
| Hurry up and you might be able to adopt it before its 30th
| birthday!
| willbeddow wrote:
| Juice is cool, but tradeoffs around which metadata store you
| choose end up being very important. It also writes files in it's
| own uninterpretable format to object storage, so if you lose the
| metadata store, you lose your data.
|
| When we tried it at Krea we ended up moving on because we
| couldn't get sufficient performance to train on, and having to
| choose which datacenter to deploy our metadata store on
| essentially forced us to only use it one location at a time.
| tptacek wrote:
| I'm betting this is on the front page today (as opposed to any
| other day; Juice is very neat and doesn't need us to hype it)
| because of our Sprites post, which goes into some detail about
| how we use Juice (for the time being; I'm not sure if we'll
| keep it this way).
|
| The TL;DR relevant to your comment is: we tore out a lot of the
| metadata stuff, and our metadata storage is SQLite +
| Litestream.io, which gives us fast local read/write, enough
| systemwide atomicity (all atomicity in our setting runs
| asymptotically against "someone could just cut the power at any
| moment"), and preserves "durably stored to object storage".
| AdamJacobMuller wrote:
| > It also writes files in it's own uninterpretable format to
| object storage, so if you lose the metadata store, you lose
| your data.
|
| That's so confusing to me I had to read it five times. Are you
| saying you lose the metadata, or that the underlying data is
| actually mangled or gone, or merely that you lose the metadata?
|
| One of the greatest features of something like this to me would
| be the ability to durable even beyond JuiceFS access to my data
| in a bad situation. Even if JuiceFS totally messes up, my data
| is still in S3 (and with versioning etc even if juicefs mangles
| or deletes my data, still). So odd to design this kind of
| software and lose this property.
| mrkurt wrote:
| It backs its metadata up to S3. You do need metadata to map
| inodes / slices / chunks to s3 objects, though.
|
| Tigris has a one-to-one FUSE that does what you want:
| https://github.com/tigrisdata/tigrisfs
| cbarrick wrote:
| As I understand it, if the metadata is lost then the whole
| filesystem is lost.
|
| I think this is a common failure mode in filesystems. For
| example, in ZFS, if you store your metadata on a separate
| device and that device is destroyed, the whole pool is
| useless.
| jeffbee wrote:
| It is not clear that pjdfstest establishes full POSIX semantic
| compliance. After a short search of the repo I did not see
| anything that exercises multiple unrelated processes atomically
| writing with O_APPEND, for example. And the fact that their
| graphic shows applications interfacing with JuiceFS over NFS and
| SMB casts further doubt, since both of those lack many POSIX
| semantic properties.
|
| Over the decades I have written test harnesses for many
| distributed filesystems and the only one that seemed to actually
| offer POSIX semantics was LustreFS, which, for related reasons,
| is also an operability nightmare.
| sabslikesobs wrote:
| See also their User Stories: https://juicefs.com/en/blog/user-
| stories
|
| I'm not an enterprise-storage guy (just sqlite on a local volume
| for me so far!) so those really helped de-abstractify what
| JuiceFS is for.
| staticassertion wrote:
| Do people really trust Redis for something like this? I feel like
| it's sort of pointless to pair Redis with S3 like this, and it'd
| be better to see benchmarks with metadata stores that can provide
| actual guarantees for durability/availability.
|
| Unfortunately, the benchmarks use Redis. Why would I care about
| distributed storage on a system like S3, which is all about
| consistency/durability/availability guarantees, just to put my
| metadata into Redis?
|
| It would be nice to see benchmarks with another metadata store.
| tuhgdetzhh wrote:
| I think they should replace Redis with Valkey or even better
| use rocksdb.
| tuhgdetzhh wrote:
| If tested various Posix FS projects over the years and everyone
| has their shortcomings in one way or the other.
|
| Although the maintainers of these projects disagree, I mostly
| consider them as a workaround for smaller projects. For big data
| (PB range) and critical production workloads I recommend to bite
| the bullet and make your software nativley S3 compatible without
| going over a POSIX mounted S3 proxy.
___________________________________________________________________
(page generated 2026-01-15 23:00 UTC)