[HN Gopher] Mountpoint - file client for S3 written in Rust, fro...
___________________________________________________________________
Mountpoint - file client for S3 written in Rust, from AWS
Author : ranman
Score : 142 points
Date : 2023-03-14 18:12 UTC (4 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| BrianHenryIE wrote:
| JungleDisk was backup software I used ~2009 that allowed mounting
| S3. They were bought by Rackspace and the product wasn't updated.
| Seems to be called/part of Cyberfortress now.
|
| Later I used Panic's Transmit Disk but they removed the feature.
|
| Recently I'd been looking at s3fs-fuse to use with gocryptfs but
| haven't actually installed it yet!
|
| https://github.com/s3fs-fuse/s3fs-fuse
|
| https://github.com/rfjakob/gocryptfs
| metadat wrote:
| How does this compare to rclone, performance wise?
|
| https://rclone.org
| IceWreck wrote:
| They should benchmark it against rclone
| sfritz wrote:
| There's a similar project under awslabs for using S3 as a
| FileSystem within the Java JVM: https://github.com/awslabs/aws-
| java-nio-spi-for-s3
| hkgjjgjfjfjfjf wrote:
| [dead]
| mlindner wrote:
| There's some really confusing use of unsafe going on.
|
| For example I'm not sure what they're doing here:
|
| https://github.com/awslabs/mountpoint-s3/blob/main/mountpoin...
| klodolph wrote:
| I want a better client for Google Cloud Storage, too, while we're
| at it. The Python gcloud / gsutil stuff is mediocre on the best
| of days.
| thangngoc89 wrote:
| In theory, you can just use this library since GC Storage
| supports S3 protocol. But in practice, I'm not sure
| seabrookmx wrote:
| `gsutil ...` was pretty bad (it is python like you say, and
| based on a very outdated fork of boto2).
|
| I've had really good luck with `gcloud storage ...` though,
| which takes essentially the same CLI args. It's much faster and
| IIRC written in golang.
| Dinux wrote:
| This is _exactly_ what I need. The current python scrips are good
| enough but a rust utility would be preferable
| toomuchtodo wrote:
| > This is an alpha release and not yet ready for production use.
| We're especially interested in early feedback on features,
| performance, and compatibility. Please send feedback by opening a
| GitHub issue. See Current status for more limitations.
| ary wrote:
| This is really interesting and something I've been thinking about
| for a while now. The SEMANTICS[1] doc details what is and isn't
| supported from a POSIX filesystem API perspective, and this
| stands out: Write operations (write, writev,
| pwrite, pwritev) are not currently supported. In the future,
| Mountpoint for Amazon S3 will support sequential writes, but with
| some limitations: Writes will only be supported
| to new files, and must be done sequentially. Modifying
| existing files will not be supported. Truncation will
| not be supported.
|
| The sequential requirement for writes is the part that I've been
| mulling over whether or not it's actually required in S3. Last
| year I discovered that S3 can do transactional I/O via multipart
| upload[2] operations combined with the CopyObject[3] operation.
| This should, in theory, allow for out of order writes, existing
| partial object re-use, and file appends.
|
| [1]
| https://github.com/awslabs/mountpoint-s3/blob/main/doc/SEMAN...
|
| [2]
| https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuove...
|
| [3]
| https://docs.aws.amazon.com/AmazonS3/latest/API/API_CopyObje...
| [deleted]
| FullyFunctional wrote:
| Forgive the question but I never quite understood the point of
| S3. It seems it's a terrible protocol but it's designed for
| bandwidth. Why couldn't they have used something like, say, 9P
| or Ceph? Surely I'm missing something fundamental.
|
| EDIT: In my personal experience with S3 it's always been super
| slow.
| spmurrayzzz wrote:
| I think theres an interesting story in software ecosystems
| where there are two flavors of applications (which coexist)
| that prefer object stores over filesystems and vice versa.
| Good reference point for this I think exists in many modern
| video transcoding infrastructures.
|
| Using something like FSx [1] gives you a performant option
| for the use cases when the tooling involved prefers
| filesystem semantics.
|
| [1] https://aws.amazon.com/fsx/lustre/
| beebmam wrote:
| Same with my experience. Not a fan
| fnordpiglet wrote:
| Simple api
|
| Absurdly cheap storage
|
| Extremely HA
|
| Absurdly durable
|
| Effectively unlimited bandwidth
|
| Effectively unbounded storage without reservation or other
| management
|
| Everything supports its api
|
| It's not a file system. It's a blob store. It's useful for
| spraying vast amounts of data into it and getting vast
| amounts of data out of it at any scale. It's not low latency,
| it's not a block store, but it is really cheap and the
| scaling of bandwidth and storage and concurrency make it
| possible to build stuff like snowflake that couldn't be built
| on Ceph in any reasonable way.
| aseipp wrote:
| Because you don't have to allocate any fixed amount up front,
| and it's pay as you go. At the time when the best storage
| options you could get were fixed-size hard drives from VPS
| providers, this was a big change, especially on both the
| "very small" and "very large" ends of the spectrum. It has
| always spoken HTTP with a relatively straightforward request-
| signing scheme for security, so integration at the basic
| levels is very easy -- you can have signed GET requests,
| written by hand, working in 20 minutes. The parallel
| throughput (on AWS, at least) is more than good enough for
| the vast, vast majority of apps assuming they actually design
| with it in mind a little. Latency could improve (especially
| externally) but realistically you can just put an HTTP
| caching layer of some sort in front to mitigate that and
| that's exactly what everybody does.
|
| Ceph was also released many years after S3 was released. And
| I've never seen a highly performant 9P implementation come
| anywhere close to even third party S3 implementations. There
| was nothing for Amazon to copy. That's why everyone else
| copied Amazon, instead.
|
| It's not the most insanely hyper-optimized thing _from the
| user POV_ (HTTP, etc) and in the past some semantics were
| pretty underspecified e.g. before full consistency guarantees
| several years ago, you only got "read your writes" and
| that's it. But it's not that hard to see why it's popular,
| IMO, given the historical context and use cases. It's hard to
| beat in the average case for both ease of use and commitment.
| Scarbutt wrote:
| S3 is slow but at the same time low cost, if you want fast
| AWS has other alternatives but pricier.
| mlhpdx wrote:
| This is misleading. S3 is also incredibly fast. The former
| when you're sequentially writing (or reading) objects and
| the latter when concurrently writing (or reading) vast
| numbers of them.
| cyberax wrote:
| That depends on what you consider "fast". EFS (the
| "serverless" NFS) has sub-millisecond operation latency.
| S3 is more in the 10-20ms range for most operations, with
| occasional spikes.
|
| BTW, if you need a pure Go client for NFSv4 (including
| AWS EFS), feel free to check my:
| https://github.com/Cyberax/go-nfs-client
| mlhpdx wrote:
| We can write vast numbers and volume of objects to S3 per
| second using concurrent processes (spawn 1000 lambda
| invocations and try it). As long as I have the network
| bandwidth, I can push stuff essentially as fast as I
| want. Is that true for EFS? Handle limits. Network
| interface limits. Protocol limits.
|
| I'm not saying that S3 is perfect or even good for most
| workloads. However, it is most excellent when the
| workload fits.
| chucky_z wrote:
| Yea it kind of is! I've used EFS in real-world scenarios
| with more than 1,000 concurrent readers/writers. EFS's
| costs are just otherworldly compared to S3. If you _need_
| that interface though, it 's a good (albeit expensive)
| choice.
| the8472 wrote:
| They have NFS (called EFS), but it's about 10x more
| expensive.
| netfortius wrote:
| Good luck mounting EFS in Windows.
| mdaniel wrote:
| Do you mean EFS specifically, or you find that NFS
| doesn't work? Because it was my recollection that Windows
| included NFS machinery natively
| acdha wrote:
| I wouldn't give a number because the pricing models are
| fairly different and the real cost will depend on how
| you're using it and how easy it is to shift your access
| patterns. On my apps using EFA, that 10x is more like
| .8-1.1x -- an easy call versus rewriting a bunch of code.
| ary wrote:
| When S3 was released the Internet was very different. Two of
| the things that stood out were:
|
| 1. It offered a resilient key/object store over HTTP.
|
| 2. By the standards of the day for bandwidth and storage it
| was (and to a certain extent still is) _very_ inexpensive.
|
| Since then much of AWS has been built on the foundation of S3
| and so its importance has changed from merely being a tool to
| basically a pervasive dependency of the AWS stack. Also, it
| very much is designed for objects larger than 1KB and for
| applications that need durable storage of many, many large
| objects.
|
| The key benefit, at least according to AWS marketing, is that
| you don't have to host it yourself.
| supriyo-biswas wrote:
| The problem is S3 is just a lexicographically ordered key
| value store with (what I suspect is) key-range partitions[1]
| for the key part and Reed-Solomon encoded blobs for the value
| part. In other words, it's a glorified NoSQL database with no
| semantics that you'd typically expect of a file system, and
| therefore repeated writes are slow because any modification
| to an object involves writing a new version of the key along
| with its new object.
|
| [1] https://martinfowler.com/articles/patterns-of-
| distributed-sy...
| rakoo wrote:
| S3 is straight HTTP, the most widespread API. It can be
| directly used on the browser, has libraries in pretty much
| every language, and can reuse the mountain of available
| software and frameworks for load-balancing, redirections,
| auth, distributed storage etc
| ignoramous wrote:
| > _Forgive the question but I never quite understood the
| point of S3._
|
| S3 and DynamoDB are essentially a decoupled BigTable; in that
| both are KV databases: One is used for high performance,
| small obj workloads; the other for high throughput, large obj
| workloads.
| deathanatos wrote:
| > _In my personal experience with S3 it's always been super
| slow._
|
| Numbers? I feel like it's been a while, but my experience was
| it is in the 50ms latency range. That's fast enough that you
| can do most things. Your page loads might not be _instant_ ,
| but 50ms is fast enough for a wide range of applications.
|
| The big mistake I see though is a lack of connection pooling:
| I find code going through the entire TCP connection setup,
| TLS setup, just for a single request, tearing it all down,
| and repeating. boto also enouranges some code patterns which
| result in GET bucket or HEAD object requests which you don't
| need and can avoid; none of this gives you good latency.
| mlhpdx wrote:
| I think you're spot on: using multipart uploads, different
| sections of the ultimate object can be created out of order.
| Unfortunately, though, that's subject to restrictions that
| require you to ensure all but the last part are sufficiently
| sized.
|
| I'm a little disappointed that this library (which is supposed
| to be "read optimized") doesn't take advantage of S3 Range
| requests to optimize read after seek. The simple example is a
| zip file in S3 for which you want only the listing of files
| from the central directory record at the end. As far as I can
| tell this library reads the entire zip to get that. I have some
| experience with this[1][2].
|
| [1] https://github.com/mlhpdx/seekable-s3-stream [2]
| https://github.com/mlhpdx/s3-upload-stream
| simooooo wrote:
| Wouldn't you be maintaining your own list of what is in the
| zip offline at this point?
| [deleted]
| wanderingmind wrote:
| For anyone looking to mount S3 as file system, I will suggest
| giving rclone a shot. It can mount, copy and do all file
| operations not just on s3 but on a wide range of cloud providers,
| you can also declare a remote as encrypted so it does client side
| encryption
| bravura wrote:
| How does this compare to goofys?
| [deleted]
| ranman wrote:
| A simple, high-throughput file client for mounting an Amazon S3
| bucket as a local file system.
| favourable wrote:
| Something similar that I've been using for a while now for an S3
| filesystem: Cyberduck[0]
|
| [0] https://cyberduck.io/s3/
| mNovak wrote:
| In a similar vein, I've been using ExpanDrive [0] for a while.
| Though admittedly it's only suitable for infrequent access /
| long term storage type use.
|
| [0] https://www.expandrive.com/desktop/
| lightlyused wrote:
| I just wish cyberduck would get a more standard UI. It is so
| win95 vb. Otherwise works great!
| supriyo-biswas wrote:
| After teaching customers for years that S3 shouldn't be mounted
| as a filesystem because of its whole object-or-nothing semantics,
| and even offering a paid solution named "storage gateway" to
| prevent issues between FS and S3 semantics, it's rather
| interesting they'd release a product like this.
|
| Amazon should really just fix the underlying issue of semantics
| by providing a PatchObjectPart API call that overwrites a
| particular multipart upload chunk with a new chunk uploaded from
| the client. CopyObjectPart+CompleteMultipartUpload still requires
| the client to issue CopyObjectPart calls for the entire object.
| wmf wrote:
| Eventually AWS always gives customers what they want even if
| it's a "bad idea".
| mlhpdx wrote:
| Indeed.
| znpy wrote:
| Bad ideas are very relative.
|
| Some bad ideas work extremely well if they fit your use case,
| you understand very well the tradeoffs and you're building
| safeguards (disaster recovery).
|
| Some other companies try to convince (force?) you into a
| workflow or into a specific solution. Aws just gives you the
| tools and some guidance on how to use them best.
| nostrebored wrote:
| Distributed patching becomes hell. You need transactional
| semantics and files are not laid out well to help you define
| invariants that should reject the transaction.
| supriyo-biswas wrote:
| There is no reason why the descriptor of objects can't be
| updated with a new value that has all of the old chunks and a
| new one, since S3 doesn't do deduplication anyway the other
| chunks may be resized internally with an asynchronous process
| that gets rid of the excess data corresponding to the now
| overridden chunk.
| dimatura wrote:
| It would be interesting to see how this compares to other
| solutions in this space, such as s3fs (the FUSE driver, not the
| python package), goofys, and the rclone mount feature, among
| others. This certainly has less features (notably, mounts are
| read-only!).
| tolmasky wrote:
| Couldn't tell from the README, does this do any sort of cache
| management or LRU type thing? In other words, does it fetch the
| underlying S3 object in real time, and then eventually eject them
| from memory and/or the backing FS when they haven't been used for
| a while?
| renewiltord wrote:
| `catfs` is a FUSE FS that can do this for you. You'll need some
| changes to make it work well. I'll have a friend upstream them
| soon, but they're easy to make yourself.
|
| Could replace `goofys` with this and then stick `catfs` in
| front.
___________________________________________________________________
(page generated 2023-03-14 23:02 UTC)