[HN Gopher] Open source cloud file system. Posix, HDFS and S3 co...
___________________________________________________________________
Open source cloud file system. Posix, HDFS and S3 compatible
Author : wiradikusuma
Score : 168 points
Date : 2023-02-08 10:05 UTC (12 hours ago)
(HTM) web link (juicefs.com)
(TXT) w3m dump (juicefs.com)
| tyingq wrote:
| >JuiceFS has introduced S3 gateway since v0.11. The feature is
| implemented based on the MinIO S3 Gateway.
|
| The MinIO S3 Gateway was deprecated:
| https://blog.min.io/deprecation-of-the-minio-gateway/
|
| I don't know if JuiceFS is treating it like a fork that they are
| maintaining, or if they have other plans.
| daviesliu wrote:
| We have a fork of MinIO at https://github.com/juicedata/minio,
| which will be maintained by us.
| mdaniel wrote:
| Damn, I was hoping it was pre the AGPL cut-over, especially
| in light of juicefs's Apache 2
| sillysaurusx wrote:
| You may want to specify reasons for wanting a "free as in
| beer" license, otherwise it just sounds greedy. (Not a
| swipe at you; I've fallen into this trap more than once.)
| rubiquity wrote:
| We have to do better to support open source with money.
| GitHub likes don't pay the bills. AGPL is fine.
| daviesliu wrote:
| Yes, JuiceFS uses the Apache 2 fork [1] directly (master
| branch), but also provide a full featured S3 gateway
| (gateway branch) under AGPL for people' choice.
|
| [1] https://github.com/juicedata/minio/tree/master
| ltbarcly3 wrote:
| Is it 'Posix Compatible' or 'Posix' aka 'Posix compliant'?
|
| It's incredibly hard to make a distributed posix compatible
| filesystem since you run into CAP. I believe (but am not certain)
| you are caching locally in violation of Posix or you are signing
| up for arbitrarily long stalls and a ton of latency on every
| read/write. (I'm not certain because I'm not sure what Posix
| specifies wrt stale reads and other cache consistency
| requirements between sync's)
|
| It would be interesting to hear what the tradeoffs are here, but
| assuming they are explicit and can be designed around this seems
| very useful.
| jeffbee wrote:
| It is not posix anything. It provides a compatibility layer
| that makes open, close, read, and write work but other than
| that does not provide the type of features that would allow you
| to deliver mail on it with qmail or whatever. It is incredibly
| misleading to advertise it that way.
|
| As you say there is no free lunch with distributed filesystems.
| Application programmers have to program their way around the
| fact that something like posix atomic writes with multiple
| writers is never going to work, and that the only way to get
| reasonable efficiency out of the thing is to defer work until
| the file is closed.
| omgwtfusb wrote:
| I can't seem to find docs in a language other than Chinese on the
| site
| danpalmer wrote:
| The docs are in English and Chinese, there's a language
| selector in the top right.
|
| Perhaps if your computer/browser's language isn't set to
| English it defaults to Chinese?
| cynicalsecurity wrote:
| Is the company Chinese or Taiwanese?
| daviesliu wrote:
| Juicedata Inc is a US company, was registered in Delaware.
| The founding team are Chinese.
|
| ps, I'm the founder of Juicedata.
| tecleandor wrote:
| https://juicefs.com/en/about-us About
| Juicedata Inc. Founded in April 2017,
| Juicedata is a globally oriented innovated distributed file
| system company. The team consists of senior architects,
| genius engineers, and consulting experts who have worked in
| the field of distributed systems for many years. The team
| members located across Hangzhou, Shanghai, Xiamen, and
| other cities, used to serve Facebook, Databricks, Tencent,
| Alibaba, Zhihu, Xiaohongshu, Douban, and other well-known
| high-tech enterprises around the world.
| Juicedata was jointly invested by China Growth Capital and
| Foothill Ventures.
| omgwtfusb wrote:
| I could not find it on iOS Safari, might be an UI bug then.
| daviesliu wrote:
| The button to switch language is at the bottom of right-top
| menu, we will fix that.
| jeroenhd wrote:
| The button doesn't appear on the mobile theme it seems
| (went to
| https://juicefs.com/docs/zh/community/introduction/ after
| clicking the "Community edition docs" link at the home
| page).
|
| Maybe consider also changing the link on the English
| homepage to the English documentation?
| throwawayapples wrote:
| Is this basically a non-POSIX* FUSE for S3 and/or Redis?
|
| * POSIX implies a whole lot of guarantees, like atomic file
| renames/moves, that definitely don't seem to be included here.
| truth_seeker wrote:
| Adopted SeaweedFS few months back. Never looked back since then.
| It's fast even on HDD disks.
|
| https://github.com/seaweedfs/seaweedfs#introduction
| wazoox wrote:
| That looks awesome! What else did you try before going to
| SeaweedFS?
| nullwarp wrote:
| Seaweed is excellent, use it in a bunch of places both big and
| small!
| kyledrake wrote:
| I love this project and I'd love to switch to it. Hopefully
| constructive feedback: The big issue I always run into with
| this stuff is what am I supposed to do if something goes wrong?
| I think the project documentation people would be wise to
| document procedures to do when certain things go wrong and how
| you should deal with them, such as if a server or two fail, or
| there's some unexpected corruption. Without that, "distributed
| storage" systems really feel incomplete to me. Storage is
| usually "mission critical" and they had a procedure for every
| single thing that could go wrong on the Apollo mission.
| throwawayapples wrote:
| By default, it starts up exposing the file server on all
| interfaces, not just localhost, AKA insecure by default.
|
| Such a poor security choice, makes me question the entire
| project.
| afandian wrote:
| Can you share your experience? What were the alternatives? Did
| you consider it against AWS S3?
| siliconc0w wrote:
| I was actually building something similar to Juice using S3 as an
| object store and optionally using redis(fast) or s3(slow) for
| metadata storage. Basically a log structured filesystem using
| rolling hash chunk encoding and delegations. I kinda stopped when
| I found juice (and to some extent seaweed) as they were much
| further along. If you need shared storage and don't have crazy
| performance requirements it makes a lot of sense to separate out
| metadata and just throw blobs into object storage.
| filleokus wrote:
| How does it compare to other players in the area? E.g Ceph,
| Gluster or Seaweed? (I'm no expert myself, only used those as a
| consumer of already setup systems)
|
| EDIT: There is a whole comparison section in the docs that I
| missed:
| https://juicefs.com/docs/community/comparison/juicefs_vs_cep...
| rch wrote:
| The comparisons are helpful, but I'm curious why Ozone wasn't
| included, since that seems like the most directly comparable
| alternative.
| daviesliu wrote:
| Apache Ozone is not POSIX compatible, even with the File
| System Optimized format [1].
|
| https://ozone.apache.org/docs/current/feature/prefixfso.html
| rch wrote:
| That's a fair point. Thanks!
| adql wrote:
| Bit weird comparison. Like sure CephFS doesn't support S3-like
| access... because the object store is a separate service that
| also runs on top of Ceph/RADOS store
| tinco wrote:
| It is weird, but it's also a valid use case. I can imagine
| someone wanting to pull files from a FS that was populated as
| a regular POSIX filesystem through an S3 api. I'm not sure if
| you can access the CephFS files from the underlying Ceph
| store easily.
| remram wrote:
| You could run Minio just like they do.
| daviesliu wrote:
| It's doable to run a MinIO gateway on top of CephFS mount
| point, but that will has performance issue, especially
| for multipart-upload and copy. That's why we put MinIO
| and JuiceFS client together and use some internal API to
| do zero-copy uploads.
| dark-star wrote:
| It compares pretty well to the proprietary WAFL/FabricPool
| technology by NetApp:
|
| - have "hot" blocks on local storage (SSD cache) - "cold"
| blocks are stored on S3 - POSIX semantics on top
|
| Having worked with NetApp technology for >10 years this is a
| welcome addition on the Open-Source side of things.
| jenka93 wrote:
| Does it support POSIX ACLs as well?
| daviesliu wrote:
| The JuiceFS Cloud supports ACL, but open source one does not
| support it yet.
| PlutoIsAPlanet wrote:
| Anyway to mount with a forced UID/GID of all files?
|
| Useful in container scenarios.
| daviesliu wrote:
| This is an experimental feature to do this, still working
| on it.
| imhoguy wrote:
| Can it do full encryption from client (transfer + at-rest) with
| Fuse?
|
| Currently I use ext4 image + LUKS + NBD over SSH tunel, it works
| but is extremely slow.
| daviesliu wrote:
| Yes, the data can be encrypted [1] by the client before sending
| to S3, but the metadata is not encrypted.
|
| [1] https://juicefs.com/docs/community/security/encrypt
| Lucasoato wrote:
| Do you know if strong read-after-write consistency is supported
| (as in s3)? Is an atomic put-if-absent method supported in
| JuiceFS (as in Azure blob storage)? If so, this could be a really
| cool platform for formats like Delta.io :)
| FlyingGeta wrote:
| It seems not, instead it provides 'close-to-open' consistency,
| as documented here:
| https://juicefs.com/docs/community/cache_management/#data-co...
| remram wrote:
| That is the same as S3 then, once an upload is complete it is
| seen by all other clients.
| daviesliu wrote:
| JuiceFS supports create-if-not-existed by using the Java SDK
| (HDFS compatible), so I guess it should work well with
| Delta.io.
| Jamie9912 wrote:
| I tried putting Postgres on JuiceFS and let's just say.. it
| didn't perform very well
| prpl wrote:
| postgres + rocks fdw would be a more interesting test case (or
| any LSM DB)
| mbreese wrote:
| Is that a use case they are really targeting though? Their
| splash page mentions big data with model generation and genomic
| sequencing as examples. I can really only speak to genetic
| sequencing. The IO pattern for these workflows is almost all
| streaming reads/writes. Random access takes too long when you
| are reading/writing 100-500GB files.
|
| Postgres doesn't like running on NFS either to be fair.
| carlhjerpe wrote:
| > Postgres doesn't like running on NFS either to be fair.
|
| https://news.ycombinator.com/item?id=19119991
|
| Is that still the case after this? Or is it tribal knowledge?
| daviesliu wrote:
| Yes, JuiceFS is not a good choice for PG, unless if you don't
| care the performance.
|
| One interesting use case is the backup of MySQL [1].
|
| [1] https://juicefs.com/docs/cloud/backup_mysql_in_juicefs/
| VikingCoder wrote:
| > 99.99999999% (10 9s) reliability SLA
|
| Can someone tell me in practical terms what that means?
|
| 1 second of unreliability every 317 years?
| daviesliu wrote:
| 99.99999999% reliability means you will not loss more than one
| byte in every 10 GB in a year.
|
| JuiceFS uses S3 as the underlying data storage, so S3 provides
| this durability SLA.
| andrewstuart2 wrote:
| As you allude to in your response, that's usually referred to
| as durability, not reliability. The home page could probably
| use an update there to reflect that terminology.
| [deleted]
| andrewxdiamond wrote:
| Important to note that S3 does not have any Durability SLA.
| We promise Durability and take it extremely seriously, but
| there is no SLA. Much more of an SLO
| riku_iki wrote:
| It sounds like not very practical metrics, since losing one
| byte often makes whole dataset useless (encryption, checksums
| failures).
| ravi-delia wrote:
| It's an average- presumably they don't smear files across
| disks byte by byte, since that would be insane. But with
| drives randomly breaking, at some point every copy of at
| least one file will go at once. With, say, a terabyte of
| files over a thousand years, you'd expect to lose a total
| number of files equal to 100Kb. So probably not even one,
| with some small chance of losing half a drive.
| riku_iki wrote:
| I think probability to lose any data in 100tb should be
| good metric.
| 908B64B197 wrote:
| As in there's no durability guarantee for the data? I can
| expect data loss at a rhythm of 1b per GB per year?
| CodesInChaos wrote:
| It's unavoidable that too many disk failures in quick
| succession lead to data-loss. For example if you store two
| copies, your durability rests on being able to detect a
| disk failure and create another copy, before the sole
| remaining version dies as well.
| juliangoldsmith wrote:
| "What do you mean you mean it can't recover from a 100%
| disk failure rate?
|
| At least it's all in RAID 0, so the data's safe."
| margorczynski wrote:
| Regarding the topic of "cloud storage" - could someone tell me if
| Juice or maybe MinIO would be a good solution to: 1. Storing
| multimedia data (image/video) uploaded by an user - here I would
| guess it can either hit it directly or via the backend for auth
| 2. Should be accessible by an URL exposed outside of the docker-
| compose so it doesn't need to go through the backend REST API 3.
| Some form of authentication based on the JWT token in the Header
| - or maybe as this is a MVP simply generating a long enough
| random string will be enough
|
| Or should I simply use nginx + filesystem and not overcomplicate?
|
| I hear everywhere S3 but as it's a pet project don't want to go
| the AWS route, instead maybe a Hetzner VPS with docker-compose to
| run the whole setup with an external Postgres instance.
| prpl wrote:
| webdav, oauth2_proxy, nginx. That's all you need. You can
| create your own issuer or also use dex I think.
|
| Fancy authnz is easy to do with openresty instead of vanilla
| nginx.
|
| Alternatively just use own cloud/nextcloud
___________________________________________________________________
(page generated 2023-02-08 23:01 UTC)