[HN Gopher] Garage - An S3 object store so reliable you can run ...
___________________________________________________________________
Garage - An S3 object store so reliable you can run it outside
datacenters
Author : ibobev
Score : 685 points
Date : 2025-12-19 15:40 UTC (1 days ago)
(HTM) web link (garagehq.deuxfleurs.fr)
(TXT) w3m dump (garagehq.deuxfleurs.fr)
| SomaticPirate wrote:
| Seeing a ton of adoption of this after the Minio debacle
|
| https://www.repoflow.io/blog/benchmarking-self-hosted-s3-com...
| was useful.
|
| RustFS also looks interesting but for entirely non-technical
| reasons we had to exclude it.
|
| Anyone have any advice for swapping this in for Minio?
| dpedu wrote:
| I have not tried either myself, but I wanted to mention that
| Versity S3 Gateway looks good too.
|
| https://github.com/versity/versitygw
|
| I am also curious how Ceph S3 gateway compares to all of these.
| zipzad wrote:
| I'd be curious to know how versitygw compares to rclone serve
| S3.
| skrtskrt wrote:
| When I was there, DigitalOcean was writing a complete
| replacement for the Ceph S3 gateway because its performance
| under high concurrency was awful.
|
| They just completely swapped out the whole service from the
| stack and wrote one in Go because of how much better the
| concurrency management was, and Ceph's team and codebase C++
| was too resistant to change.
| jiqiren wrote:
| Unrelated, but one of the more annoying aspects of whatever
| software they use now is lack of IPv6 for the CDN layer of
| DigitalOcean Spaces. It means I need to proxy requests
| myself. :(
| Implicated wrote:
| > but for entirely non-technical reasons we had to exclude it
|
| Able/willing to expand on this at all? Just curious.
| NitpickLawyer wrote:
| Not the same person you asked, but my guess would be that it
| is seen as a chinese product.
| lima wrote:
| RustFS appears to be very early-stage with no real
| distributed systems architecture:
| https://github.com/rustfs/rustfs/pull/884
|
| I'm not sure if it even has any sort of cluster consensus
| algorithm? I can't imagine it _not_ eating committed writes
| in a multi-node deployment.
|
| Garage and Ceph (well, radosgw) are the only open source
| S3-compatible object storage which have undergone serious
| durability/correctness testing. Anything else will most
| likely eat your data.
| dewey wrote:
| What is this based on, honest question as from the landing
| page I don't get that impression. Are many committers
| China-based?
| NitpickLawyer wrote:
| https://rustfs.com.cn/
|
| > Beijing Address: Area C, North Territory, Zhongguancun
| Dongsheng Science Park, No. 66 Xixiaokou Road, Haidian
| District, Beijing
|
| > Beijing ICP Registration No. 2024061305-1
| dewey wrote:
| Oh, I misread the initial comment and thought they had to
| exclude Garage. Thanks!
| misnome wrote:
| They seem to have gone all-in on AI, for commits and ticket
| management. Not interested in interacting with that.
|
| Otherwise, the built in admin on one-executable was nice, and
| support for tiered storage, but single node parallel write
| performance was pretty unimpressive and started throwing
| strange errors (investigating of which led to the AI ticket
| discovery).
| scottydelta wrote:
| From what I have seen in the previous discussions here (since
| and before Minio debacle) and at work, Garage is a solid
| replacement.
| klooney wrote:
| Seaweed looks good in those benchmarks, I haven't heard much
| about it for a while.
| chrislusf wrote:
| Disclaim: I work on SeaweedFS.
|
| Why skipping SeaweedFS? It rank #1 on all benchmarks, and has a
| lot of features.
| dionian wrote:
| can you link benchmarks
| chrislusf wrote:
| It is in the parent comment.
| meotimdihia wrote:
| I confirm this, I used SeaweedFS to serve 1M users daily with
| 56 million images / ~100TB with 2 servers + HDD only, while
| Minio can't do this. Seaweedfs performance is much better
| than Minio's. The only problem is that SeaweedFS
| documentation is hard to understand.
| magicalhippo wrote:
| SeaweedFS is also _so_ optimized for small objects, it can
| 't store larger objects (max 32GiB[1]).
|
| Not a concern for many use-cases, just something to be
| aware of as it's not a universal solution.
|
| [1]: https://github.com/seaweedfs/seaweedfs?tab=readme-ov-
| file#st...
| chrislusf wrote:
| Not correct. The files are chunked into smaller pieces
| and spread to all volume servers.
| ted_dunning wrote:
| SeaweedFS is very nice and takes quite an effort to lose
| data.
| ai-christianson wrote:
| I love garage. I think it has applications beyond the standard
| self host s3 alternative.
|
| It's a really cool system for hyper converged architecture where
| storage requests can pull data from the local machine and only
| hit the network when needed.
| singpolyma3 wrote:
| I'd love to hear what configuration you are using for this
| Powdering7082 wrote:
| No erasure coding seems like a pretty big loss in terms of how
| much resources do you need to get good resiliency & efficiency
| munro wrote:
| I was looking at using this on an LTO tape library, it seems
| the only resiliency is through replication, but this was my
| main concern with this project, what happens with HW goes bad
| lxpz wrote:
| If you have replication, you can lose one of the replica,
| that's the point. This is what Garage was designed for, and
| it works.
|
| Erasure coding is another debate, for now we have chosen not
| to implement it, but I would personally be open to have it
| supported by Garage if someone codes it up.
| hathawsh wrote:
| Erasure coding is an interesting topic for me. I've run
| some calculations on the theoretical longevity of digital
| storage. If you assume that today's technology is close to
| what we'll be using for a long time, then cross-device
| erasure coding wins, statistically. However, if you factor
| in the current exponential rate of technological
| development, simply making lots of copies and hoping for
| price reductions over the next few years turns out to be a
| winning strategy, as long as you don't have vendor lock-in.
| In other words, I think you're making great choices.
| Dylan16807 wrote:
| I question that math. Erasure coding needs less than half
| as much space as replication, and imposes pretty small
| costs itself. Maybe we can say the difference is
| irrelevant if storage prices will drop 4x over the next
| five years? But looking at pricing trends right now...
| that's not likely. Hard drives and SSDs are about the
| same price they were 5 years ago. The 5 years before that
| SSDs were seeing good advancements, but hard drive prices
| only advanced 2x.
| fabian2k wrote:
| Looks interesting for something like local development. I don't
| intend to run production object storage myself, but some of the
| stuff in the guide to the production setup
| (https://garagehq.deuxfleurs.fr/documentation/cookbook/real-w...)
| would scare me a bit:
|
| > For the metadata storage, Garage does not do checksumming and
| integrity verification on its own, so it is better to use a
| robust filesystem such as BTRFS or ZFS. Users have reported that
| when using the LMDB database engine (the default), database files
| have a tendency of becoming corrupted after an unclean shutdown
| (e.g. a power outage), so you should take regular snapshots to be
| able to recover from such a situation.
|
| It seems like you can also use SQLite, but a default database
| that isn't robust against power failure or crashes seems
| suprising to me.
| igor47 wrote:
| I've been using minio for local dev but that version is
| unmaintained now. However, I was put off by the minimum
| requirements for garage listed on the page -- does it really
| need a gig of RAM?
| archon810 wrote:
| The current latest Minio release that is working for us for
| local development is now almost a year old and soon enough we
| will have to upgrade. Curious what others have replaced it
| with that is as easy to set up and has a management UI.
| mbreese wrote:
| I think that's part of the pitch here... swapping out Minio
| for Garage. Both scale a lot more than for just local
| development, but local dev certainly seems like a good use-
| case here.
| lxpz wrote:
| It does not, at least not for a small local dev server. I
| believe RAM usage should be around 50-100MB, increasing if
| you have many requests with large objects.
| dsvf wrote:
| I always understood this requirement as "garage will run fine
| on hardware with 1GB RAM total" - meaning the 1GB includes
| the RAM used by the OS and other processes. I think that most
| current consumer hardware that is a, potential garage host,
| even on the low end, has at least 1GB total RAM.
| moffkalast wrote:
| That's not something you can do reliably in software,
| datacenter grade NVMe drives come with power loss protection
| and additional capacitors to handle that gracefully. If power
| is cut at the wrong moment the partition may not be mountable
| afterwards otherwise.
|
| If you really live somewhere with frequent outages, buy an
| industrial drive that has a PLP rating. Or get a UPS, they tend
| to be cheaper.
| crote wrote:
| Isn't that the entire point of write-ahead logs, journaling
| file systems, and fsync in general? A roll-back or roll-
| forward due to a power loss causing a partial write is
| completely expected, but _surely_ consumer SSDs wouldn 't
| just completely ignore fsync and blatantly lie that the data
| has been persisted?
|
| As I understood it, the capacitors on datacenter-grade drives
| are to give it more flexibility, as it allows the drive to
| issue a successful write response for cached data: the
| capacitor guarantees that _even with a power loss_ the write
| will still finish, so for all intents and purposes it has
| been persisted, so an fsync can return without having to wait
| on the actual flash itself, which greatly increases
| performance. Have I just completely misunderstood this?
| Nextgrid wrote:
| > ignore fsync and blatantly lie that the data has been
| persisted
|
| Unfortunately they do:
| https://news.ycombinator.com/item?id=38371307
| btown wrote:
| If the drives continue to have power, but the OS has
| crashed, will the drives persist the data once a certain
| amount of time has passed? Are datacenters set up to take
| advantage of this?
| Nextgrid wrote:
| > will the drives persist the data once a certain amount
| of time has passed
|
| Yes, otherwise those drives wouldn't work at all and
| would have a 100% warranty return rate. The reason they
| get away with it is that the misbehavior is only a
| problem in a specific edge-case (forgetting data written
| shortly before a power loss).
| unsnap_biceps wrote:
| Yes, the drives are unaware of the OS state.
| unsnap_biceps wrote:
| you actually don't need capacitors for rotating media,
| Western Digital has a feature called "ArmorCache" that uses
| the rotational energy in the platters to power the drive
| long enough to sync the volatile cache to a non volatile
| storage.
|
| https://documents.westerndigital.com/content/dam/doc-
| library...
| toomuchtodo wrote:
| Very cool, like the ram air turbine that deploys on
| aircraft in the event of a power loss.
| patmorgan23 wrote:
| Good I love engineers
| Aerolfos wrote:
| > but surely consumer SSDs wouldn't just completely ignore
| fsync and blatantly lie that the data has been persisted?
|
| That doesn't even help if fsync() doesn't do what
| developers expect: https://danluu.com/fsyncgate/
|
| I think this was the blog post that had a bunch more stuff
| that can go wrong too: https://danluu.com/deconstruct-
| files/
|
| But basically fsync itself (sometimes) has dubious
| behaviour, then OS on top of kernel handles it dubiously,
| and then even on top of that most databases can ignore
| fsync erroring (and lie that the data was written properly)
|
| So... yes.
| lxpz wrote:
| If you know of an embedded key-value store that supports
| transactions, is fast, has good Rust bindings, and does
| checksumming/integrity verification by default such that it
| almost never corrupts upon power loss (or at least, is always
| able to recover to a valid state), please tell me, and we will
| integrate it into Garage immediately.
| BeefySwain wrote:
| (genuinely asking) why not SQLite by default?
| lxpz wrote:
| We were not able to get good enough performance compared to
| LMDB. We will work on this more though, there are probably
| many ways performance can be increased by reducing load on
| the KV store.
| skrtskrt wrote:
| Could you use something like Fly's Corrosion to shard and
| distribute the SQLite data? It uses a CRDT
| reconciliation, which is familiar for Garage.
| lxpz wrote:
| Garage already shards data by itself if you add more
| nodes, and it is indeed a viable path to increasing
| throughput.
| tensor wrote:
| Keep in mind that write safety comes with performance
| penalties. You can turn off write protections and many
| databases will be super fast, but easily corrupt.
| srcreigh wrote:
| Did you try WITHOUT ROWID? Your sqlite implementation[1]
| uses a BLOB primary key. In SQLite, this means each
| operation requires 2 b-tree traversals: The BLOB->rowid
| tree and the rowid->data tree.
|
| If you use WITHOUT ROWID, you traverse only the
| BLOB->data tree.
|
| Looking up lexicographically similar keys gets a huge
| performance boost since sqlite can scan a B-Tree node and
| the data is contiguous. Your current implementation is
| chasing pointers to random locations in a different
| b-tree.
|
| I'm not sure exactly whether on disk size would get
| smaller or larger. It probably depends on the key size
| and value size compared to the 64 bit rowids. This is
| probably a well studied question you could find the
| answer to.
|
| [1]: https://git.deuxfleurs.fr/Deuxfleurs/garage/src/comm
| it/4efc8...
| lxpz wrote:
| Very interesting, thank you. It would probably make sense
| for most tables but not all of them because some are
| holding large CRDT values.
| asa400 wrote:
| Other than knowing this about SQLite beforehand, is there
| any way one could discover that this is happening through
| tracing?
| rapnie wrote:
| I learned that Turso apparently have plans for a rewrite
| of libsql [0] in Rust, and create a more 'hackable'
| SQLite alternative altogether. It was apparently
| discussed in this Developer Voices [1] video, which I
| haven't yet watched.
|
| [0] https://github.com/tursodatabase/libsql
|
| [1] https://www.youtube.com/watch?v=1JHOY0zqNBY
| agavra wrote:
| Sounds like a perfect fit for https://slatedb.io/ -- it's
| just that (an embedded, rust, KV store that supports
| transactions).
|
| It's built specifically to run on object storage, currently
| relies on the `object_store` crate but we're consdering
| OpenDAL instead so if Garage works with those crates (I
| assume it does if its S3 compatible) it should just work
| OOTB.
| evil-olive wrote:
| for Garage's particular use case I think SlateDB's "backed
| by object storage" would be an anti-feature. their usage of
| LMDB/SQLite is for the metadata of the object store itself
| - trying to host that metadata within the object store runs
| into a circular dependency problem.
| fabian2k wrote:
| I don't really know enough about the specifics here. But my
| main points isn't about checksums, but more something like
| WAL in Postgres. For an embedded KV store this is probably
| not the solution, but my understanding is that there are data
| structures like LSM that would result in similar robustness.
| But I don't actually understand this topic well enough.
|
| Checksumming detects corruption after it happened. A database
| like Postgres will simply notice it was not cleanly shut down
| and put the DB into a consistent state by replaying the write
| ahead log on startup. So that is kind of my default
| expectation for any DB that handles data that isn't ephemeral
| or easily regenerated.
|
| But I also likely have the wrong mental model of what Garage
| does with the metadata, as I wouldn't have expected that to
| be ever limited by Sqlite.
| lxpz wrote:
| So the thing is, different KV stores have different trade-
| offs, and for now we haven't yet found one that has the
| best of all worlds.
|
| We do recommend SQLite in our quick-start guide to setup a
| single-node deployment for small/moderate workloads, and it
| works fine. The "real world deployment" guide recommends
| LMDB because it gives much better performance (with the
| current status of Garage, not to say that this couldn't be
| improved), and the risk of critical data loss is mitigated
| by the fact that such a deployment would use multi-node
| replication, meaning that the data can always be recovered
| from another replica if one node is corrupted and no
| snapshot is available. Maybe this should be worded better,
| I can see that the alarmist wording of the deployment guide
| is creating quite a debate so we probably need to make
| these facts clearer.
|
| We are also experimenting Fjall as an alternate KV engine
| based on LSM, as it theoretically has good speed and crash
| resilience, which would make it the best option. We are
| just not recommending it by default yet, as we don't have
| much data to confirm that it works up to these
| expectations.
| patmorgan23 wrote:
| Valkey?
| __turbobrew__ wrote:
| RocksDB possibly. Used in high throughput systems like Ceph
| OSDs.
| johncolanduoni wrote:
| I've used RocksDB for this kind of thing in the past with
| good results. It's very thorough from a data corruption
| detection/rollback perspective (this is naturally much easier
| to get right with LSMs than B+ trees). The Rust bindings are
| fine.
|
| It's worth noting too that B+ tree databases are not a
| fantastic match for ZFS - they usually require extra tuning
| (block sizes, other stuff like how WAL commits work) to get
| performance comparable to XFS/ext4. LSMs on the other hand
| naturally fit ZFS's CoW internals like a glove.
| VerifiedReports wrote:
| It's "key/value store", FYI
| abustamam wrote:
| Wikipedia seems to find "key-value store" an appropriate
| term.
|
| https://en.wikipedia.org/wiki/Key%E2%80%93value_database
| VerifiedReports wrote:
| See above.
| kqr wrote:
| It's not a store of "keys or values", no. It's a store of
| key-value pairs.
| VerifiedReports wrote:
| A key-value store would be a store of one thing: key
| values. A hyphen combines two words to make an adjective,
| which describes the word that follows: A
| used-car lot A value-added tax A
| key-based access system
|
| When you have two exclusive options, two sides to a
| situation, or separate things; you separate them with a
| slash: An on/off switch A
| win/win situation A master/slave arrangement
|
| Therefore a key-value store and a key/value store are
| quite different.
| kqr wrote:
| All of your slash examples represent either-or
| situations. A swich turns it on or off, the situation is
| a win in the first outcome or a win in the second
| outcome, etc.
|
| It's true that key-value store shouldn't be written with
| a hyphen. It should be written with an en dash, which is
| used "to contrast values or illustrate a relationship
| between two things [... e.g.] Mother-daughter
| relationship"
|
| https://en.wikipedia.org/wiki/Dash#En_dash
|
| I just didn't want to bother with typography at that
| level of pedanticism.
| VerifiedReports wrote:
| No, they don't. A master/slave configuration (of hard
| drives, for example) involves two things. I specifically
| included it to head off the exact objection you're
| raising.
|
| "...the slash is now used to represent division and
| fractions, as a date separator, in between multiple
| alternative or related terms"
|
| -Wikipedia
|
| And what is a key/value store? A store of related terms.
|
| And if you had a system that only allowed a finite
| collection of key values, where might you put them? A
| key-value store.
| kqr wrote:
| The hard drives are either master or slave. A hard drive
| is not a master-and-slave.
| DonHopkins wrote:
| Which is infinite of value is zero.
| yupyupyups wrote:
| Depending on the underlying storage being reliable is far from
| unique to garage. This is what most other services do too,
| unless we're talking about something like Ceph which manages
| the physical storage itself.
|
| Standard filesystems such as ext4 and xfs don't have data
| checksumming, so you'll have to rely on another layer to
| provide integrity. Regardless, that's not garage's job imo.
| It's good that they're keeping their design simple and focus
| their resources on implementing the S3 spec.
| nijave wrote:
| The assumption is nodes are in different fault domains so it'd
| be highly unlikely to ruin the whole cluster.
|
| LMDB mode also runs with flush/syncing disabled
| doctorpangloss wrote:
| https://git.deuxfleurs.fr/Deuxfleurs/garage/src/branch/main-...
|
| this is the reliability question no?
| lxpz wrote:
| I talked about the meaning of the Jepsen test and the results
| we obtained in the FOSDEM'24 talk:
|
| https://archive.fosdem.org/2024/schedule/event/fosdem-2024-3...
|
| Slides are available here:
|
| https://git.deuxfleurs.fr/Deuxfleurs/garage/src/commit/4efc8...
| agwa wrote:
| Does this support conditional PUT (If-Match / If-None-Match)?
| codethief wrote:
| https://news.ycombinator.com/item?id=46328218
| faizshah wrote:
| One really useful usecase for Garage for me has been data
| engineering scripts. I can just use the S3 integration that every
| tool has to dump to garage and then I can more easily scale up to
| cloud later.
| Eikon wrote:
| Unfortunately, this doesn't support conditional writes through
| if-match and if-none-match [0] and thus is not compatible with
| ZeroFS [1].
|
| [0] https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/1052
|
| [1] https://github.com/Barre/ZeroFS
| chrislusf wrote:
| I work on SeaweedFS. It has support for these if conditions,
| and a lot more.
| wyattjoh wrote:
| Wasn't expecting to see it hosted on forgejo. Kind of a breath of
| fresh air to be honest.
| thhck wrote:
| BTW https://deuxfleurs.fr/ is one of the most beautiful website I
| have ever seen
| codethief wrote:
| It's beautiful from an artistic point of view but also rather
| hard to read and probably not very accessible (haven't checked
| it, though, since I'm on my phone).
| isoprophlex wrote:
| Works perfectly on an iphone. I can't attest to the
| accessibility features, but the aesthetic is absolutely
| wonderful. Something I love, and went for on my own
| portfolio/company website... this is executed 100x better
| tho, clearly a labor of love and not 30 minutes of shitting
| around in vi.
| self_awareness wrote:
| Well it's ASCII-themed but it's completely unreadable in
| terminal links/lynx.
| apawloski wrote:
| Is it the same consistency model as S3? I couldn't see anything
| about it in their docs.
| lxpz wrote:
| Read-after-write consistency : yes (after PutObject has
| finished, the object will be immediately visible in all
| subsequent requests, including GetObject and ListObjects)
|
| Conditionnal writes : no, we can't do it with CRDTs, which are
| the core of Garage's design.
| skrtskrt wrote:
| Does RAMP or CURE offer any possibility of conditional writes
| with CRDTs? I have had these papers on my list to read for
| months, specifically wondering if it could be applied to
| Garage
|
| https://dd.thekkedam.org/assets/documents/publications/Repor.
| .. http://www.bailis.org/papers/ramp-sigmod2014.pdf
| lxpz wrote:
| I had a very rapid look at these two papers, it looks like
| none of them allow the implementation of compare-and-swap,
| which is required for if-match / if-none-match support.
| They have a weaker definition of a "transaction". Which is
| to be expected as they only implement causal consistency at
| best and not consensus, whereas consensus is required for
| compare-and-swap.
| skrtskrt wrote:
| ack - makes sense, thank you for looking!
| topspin wrote:
| No tags on objects.
|
| Garage looks really nice: I've evaluated it with test code and
| benchmarks and it looks like a winner. Also, very straightforward
| deployment (self contained executable) and good docs.
|
| But no tags on objects is a pretty big gap, and I had to shelve
| it. If Garage folk see this: please think on this. You obviously
| have the talent to make a killer application, but tags are table
| stakes in the "cloud" API world.
| lxpz wrote:
| Thank you for your feedback, we will take it into account.
| topspin wrote:
| Great, and thank you.
|
| I really, really appreciate that Garage accommodates running
| as a single node _without_ work-arounds and special
| configuration to yield some kind of degraded state. Despite
| the single minded focus on distributed operation you no doubt
| hear endlessly (as seen among some comments here,) there are,
| in fact, traditional use cases where someone will be
| attracted to Garage _only_ for the API compatibility, and
| where they will achieve availability in production sufficient
| to their needs by means other than clustering.
| VerifiedReports wrote:
| What are "tags on objects?"
| topspin wrote:
| https://docs.aws.amazon.com/AmazonS3/latest/userguide/object.
| ..
|
| Arbitrary name+value pairs attached to S3 objects and
| buckets, and readily available via the S3 API. Metadata,
| basically. AWS has some tie-ins with permissions and other
| features, but tags can be used for any purpose. You might
| encode video multiple times at different bitrates, and store
| the rate in a tag on each object, for example. Tags are an
| affordance used by many applications for countless purposes.
| VerifiedReports wrote:
| Thanks! I understand what tags are, but not what an
| "object" was in this context. Your example of multiple
| encodings of the same video seems very good.
| JonChesterfield wrote:
| Corrupts data on power loss according to their own docs. Like
| what you get outside of data centers. Not reliable then.
| lxpz wrote:
| Losing a node is a regular occurrence, and a scenario for which
| Garage has been designed.
|
| The assumption Garage makes, which is well-documented, is that
| of 3 replica nodes, only 1 will be in a crash-like situation at
| any time. With 1 crashed node, the cluster is still fully
| functional. With 2 crashed nodes, the cluster is unavailable
| until at least one additional node is recovered, but no data is
| lost.
|
| In other words, Garage makes a very precise promise to its
| users, which is fully respected. Database corruption upon power
| loss enters in the definition of a "crash state", similarly to
| a node just being offline due to an internet connection loss.
| We recommend making metadata snapshots so that recovery of a
| crashed node is faster and simpler, but it's not required per
| se: Garage can always start over from an empty database and
| recover data from the remaining copies in the cluster.
|
| To talk more about concrete scenarios: if you have 3 replicas
| in 3 different physical locations, the assumption of at-most
| one crashed node is pretty reasonable, it's quite unlikely that
| 2 of the 3 locations will be offline at the same time.
| Concerning data corruption on a power loss, the probability to
| lose power at 3 distant sites at the exact same time with the
| same data in the write buffers is extremely low, so I'd say in
| practice it's not a problem.
|
| Of course, this all implies a Garage cluster running with 3-way
| replication, which everyone should do.
| jiggawatts wrote:
| So if you put a 3-way cluster in the same building and they
| lose power together, then what? Is your data toast?
| lxpz wrote:
| If I make certain assumptions and you respect them, I will
| give you certain guarantees. If you don't respect them, I
| won't guarantee anything. I won't guarantee that your data
| will be toast either.
| Dylan16807 wrote:
| If you can't guarantee _anything_ for all the nodes
| losing power at the same time, that 's really bad.
|
| If it's just the write buffer at risk, that's fine. But
| the chance of overlapping power loss across multiple
| sites isn't low enough to risk all the existing data.
| InitialBP wrote:
| It sounds like that's a possibility, but why on earth would
| you take the time to setup a 3 node cluster of object
| storage for reliability and ignore one of the key tenants
| of what makes it reliable?
| JonChesterfield wrote:
| That is a much stronger guarantee than your documentation
| currently claims. One site falling over and being rebuilt
| without loss is great. One site losing power, corrupting the
| local state, then propagating that corruption to the rest of
| the cluster would not be fine. Different behaviours.
| lxpz wrote:
| Fair enough, we will work on making the documentation
| clearer.
| ekjhgkejhgk wrote:
| Anybody understand how this compares with Vast?
| allanrbo wrote:
| I use Syncthing a lot. Is Garage only really useful if you
| specifically want to expose an S3 drop in compatible API, or does
| it also provide other benefits over syncthing?
| lxpz wrote:
| They are not solving the same problem.
|
| Syncthing will synchronize a full folder between an arbitrary
| number of machines, but you still have to access this folder
| one way or another.
|
| Garage provides an HTTP API for your data, and handles
| internally the placement of this data among a set of possible
| replica nodes. But the data is not in the form of files on disk
| like the ones you upload to the API.
|
| Syncthing is good for, e.g., synchronizing your documents or
| music collection between computers. Garage is good as a storage
| service for back-ups with e.g. Restic, for media files stored
| by a web application, for serving personal (static) web sites
| to the Internet. Of course, you can always run something like
| Nextcloud in front of Garage and get folder synchronization
| between computers somewhat like what you would get with
| Syncthing.
|
| But to answer your question, yes, Garage only provides a
| S3-compatible API specifically.
| sippeangelo wrote:
| You use Syncthing for object storage?
| supernes wrote:
| I tried it recently. Uploaded around 300 documents (1GB) and then
| went to delete them. Maybe my client was buggy, because the S3
| service inside the container crashed and couldn't recover - I had
| to restart it. It's a really cool project, but I wouldn't really
| call it "reliable" from my experience.
| awoimbee wrote:
| How is garage for a simple local dev env ? I recently used
| seaweedfs since they have a super simple minimal setup compared
| to garage which seemed to require a config file just to get
| started.
| adamcharnock wrote:
| Copy/paste from a previous thread [0]:
|
| We've done some fairly extensive testing internally recently and
| found that Garage is somewhat easier to deploy in comparison to
| our existing use of MinIO, but is not as performant at high
| speeds. IIRC we could push about 5 gigabits of (not small) GET
| requests out of it, but something blocked it from reaching the
| 20-25 gigabits (on a 25g NIC) that MinIO could reach (also 50k
| STAT requests/s, over 10 nodes)
|
| I don't begrudge it that. I get the impression that Garage isn't
| necessarily focussed on this kind of use case.
|
| ---
|
| In addition:
|
| Next time we come to this we are going to look at RustFS [1], as
| well as Ceph/Rook [2].
|
| We can see we're going to have to move away from MinIO in the
| foreseeable future. My hope is that the alternatives get a boost
| of interest given the direction MinIO is now taking.
|
| [0]: https://news.ycombinator.com/item?id=46140342
|
| [1]: https://rustfs.com/
|
| [2]: https://rook.io/
| hardwaresofton wrote:
| Please also consider including SeaweedFS in the testing.
| __turbobrew__ wrote:
| I wouldn't use rook if you solely want S3. It is a massively
| complex system which you really need to invest in understanding
| or else your cluster will croak at some point and you will have
| no idea on how to fix it.
| breakingcups wrote:
| IS there a better solution for self-healing S3 storage that
| you could recommend? I'm also curious what will make a rook
| cluster croak after some time and what kind of maintenance is
| required in your experience.
| adastra22 wrote:
| ceph?
| yupyupyups wrote:
| Rook is ceph.
| adamcharnock wrote:
| Not used it yet, but RustFS sounds like it has self healing
|
| https://docs.rustfs.com/troubleshooting/healing.html
| __turbobrew__ wrote:
| I have unfortunately got a ceph cluster in a bad enough
| state that I just had to delete the pools and start from
| scratch. It was due to improper sequencing when removing
| OSDs, but that is kindof the point is you have to know what
| you are doing to know how to do things safely. For the most
| part I have so far learned by blundering things and
| learning hard lessons. Ceph clusters when mistreated can
| get into death spirals that only an experienced
| practitioner can advert through very carefully modifying
| cluster state through things like upmaps. You also need to
| make sure you understand your failure domains and how to
| spread mons and osds across the domains to properly handle
| failure. Lots of people don't think about this and then one
| day a rack goes poof and you didn't replicate your data
| across racks and you have data loss. Same thing with mons,
| you should be deploying mons across at least 3 failure
| domains (ideally 3 different datacenters) to maintain
| quorum during an outage.
| nine_k wrote:
| They explicitly say that top performance is not a goal: <<high
| performances constrain a lot the design and the infrastructure;
| we seek performances through minimalism only>>
| (https://garagehq.deuxfleurs.fr/documentation/design/goals/)
|
| But it might be interesting to see where the time is spent. I
| suspect they may be doing fewer things in parallel than MinIO,
| but maybe it's something entirely different.
| NL807 wrote:
| >I get the impression that Garage isn't necessarily focussed on
| this kind of use case.
|
| I wouldn't be surprised if this will be fixed sometime in the
| future.
| throwaway894345 wrote:
| > We can see we're going to have to move away from MinIO in the
| foreseeable future.
|
| My favorite thing about all of this is that I had just invested
| a ton of time in understanding MinIO and its Kubernetes
| operator and got everything into a state that I felt good
| about. I was nearly ready to deploy it to production when the
| announcement was released that they would not be supporting it.
|
| I'm somewhat surprised that no one is forking it (or I haven't
| heard about any organizations of consequence stepping up
| anyway) instead of all of these projects to rebuild it from
| scratch.
| johncolanduoni wrote:
| Somewhat unrelated, but I just looked at the RustFS docs
| intro[1] after seeing it here. It has this statement:
|
| > RustFS is a high-performance, distributed object storage
| software developed using Rust, the world's most popular memory-
| safe language.
|
| I'm actually something of a Rust booster, and have used it
| professionally more than once (including working on a primarily
| Rust codebase for a while). But it's hard to take a project's
| docs seriously when it describes Rust as "the world's most
| popular memory-safe language". Java, JavaScript, Python, even
| C# - these all blow it out of the water in popularity and are
| unambiguously memory safe. I've had a lot more segfaults in
| Rust dependencies than I have in Java dependencies (though both
| are minuscule in comparison to e.g. C++ dependencies).
|
| [1]: https://docs.rustfs.com/installation/
| b112 wrote:
| I feel dumbfounded. All I've ever heard from rust users, is
| the equivalent of football fans running up, waving pendants
| in my face and screaming. So much so, that everything else
| said seems like the wild fantasies of "our team gonna win".
|
| Then things like this appear:
|
| https://www.phoronix.com/news/First-Linux-Rust-CVE
|
| And I'm all warm and feeling schadenfreude.
|
| To hear "yes, it's safer" and yet not "everyone on the planet
| not using rust is a moron!!!", is a nice change.
|
| Frankly, the whole cargo side of rust has the same issues
| that node has, and that's silly beyond comprehension. Memory
| safe is almost a non-concern, compared to installing random,
| unvetted stuff. Cargo vet seems barely helpful here.
|
| I'd want any language caring about security and code safety,
| to have a human audit every single diff, on every single
| package, _and_ host those specific crates on locked down
| servers.
|
| No, I don't care about "but that will slow down development
| and change!". Security needs to be first and front.
|
| And until the Rust community addresses this, and its
| requirement for 234234 packages, it's a toy.
|
| And yes, it can be done. And no, it doesn't require money.
| Debian's been doing just this very thing for decades, on a
| far, far, far larger scale. Debian developers gatekeep. They
| package. They test and take bug reports on specific packages.
| This is a solved problem.
|
| Caring about 'memory safe!' is grand, but ignoring the rest
| of the ecosystem is absurd.
| sporkland wrote:
| Not dismissing your point, but Looking at the article, it
| looks like it's in rust unsafe code. Which seems to me to
| be a point that the rest of the rust code is fine but the
| place where they turned off the static safety the language
| provides they got bit.
| b112 wrote:
| Hey! Can't I just enjoy my schadenfreude in peace?
|
| I guess the takeaway is that, doubly so, trusting rust
| code to be memory safe, simply because it is rust isn't
| sensible. All its protections can simple be invalidated,
| and an end user would never know.
| teiferer wrote:
| Um I doubt Debian maintainers look at every single line of
| code in the packages they maintain.
| necovek wrote:
| Debian has been doing this for decades, yes, but it is
| largely a volunteer effort, and it's become a meme how slow
| Debian is to release things.
|
| I've long desired this approach (backporting security
| fixes) to be commercialized instead of the always-up-to-
| date-even-if-incompatible push, and on top of Red Hat,
| Suse, Canonical (with LTS), nobody has been doing it for
| product teams until recently (Chainguard seems to be doing
| this).
|
| But, if you ignore speed, you also fail: others will build
| less secure products and conquer the market, and your
| product has no future.
|
| The real engineering trick is to be fast and build new
| things, which is why we need supply chain commoditized
| stewards (for a fee) that will solve this problem for you
| and others at scale!
| PunchyHamster wrote:
| > Debian has been doing this for decades, yes, but it is
| largely a volunteer effort, and it's become a meme how
| slow Debian is to release things.
|
| which is a bit silly considering that if you want fast,
| most packages land in testing/unstable pretty quickly.
| necovek wrote:
| But then you as a consumer/user of Debian packages need
| to stay on top of things when they change in backwards-
| incompatible ways.
|
| I believe the sweet spot is Debian-like stable as the
| base platform to build on top of, and then commercial-
| support in a similar way for any dependencies you must
| have more recent versions on top.
| PunchyHamster wrote:
| > But then you as a consumer/user of Debian packages need
| to stay on top of things when they change in backwards-
| incompatible ways.
|
| If you need latest packages, you have to do it anyway.
|
| > I believe the sweet spot is Debian-like stable as the
| base platform to build on top of, and then commercial-
| support in a similar way for any dependencies you must
| have more recent versions on top.
|
| That if the company can build packages properly. Also too
| old OS deps sometimes do throw wrench in the works.
|
| Tho frankly "latest Debian Testing" have far smaller
| chance breaking something than "latest piece of software
| that couldn't figure out how to upstream to Debian"
| necovek wrote:
| The difference is between staying on stable and cherry-
| picking the latest for what you really do need, and being
| on everything latest.
|
| The latter has a huge maintenance burden, the former is
| the, as I said already, sweet spot. (And let's not talk
| about combining stable/testing, any machine I tried that
| on got into an non-upgradeable mess quickly)
|
| I am not saying it is easy, which is exactly why I think
| it should be a commercial service that you pay for for it
| to actually survive.
| dotancohen wrote:
| > supply chain commoditized stewards (for a fee)
|
| I agree with this, but the open source licenses allow
| anyone who purchases a stewarded implementation to
| distribute it freely.
|
| I would love to see a software distribution model in
| which we could pay for vetted libraries, from bodies that
| we trust, which would become FOSS after a time period -
| even a month would be fine.
|
| There are flaws in my argument, but it is a safer option
| than the current normal practices.
| SEJeff wrote:
| One might even call the rust community a "cargo cult"
| teiferer wrote:
| It's hard to take a project seriously if it focuses so much
| on the language it's written in. As a user, I don't care.
| Show me the results (bug tracker with low rate of issues),
| that's what I care about. Whether you program in Rust or C or
| Java or assembly or PHP.
| limagnolia wrote:
| As a potential user of an open source project, I care a
| fair bit what language it is implemented in. As an open
| source project, I preffer projects in languages and
| ecosystems I am familair and comfortable with. I may need
| to fix bugs, add features, or otherwise make contributions
| back to the project, and thus I am more likely to pick a
| solution in a language I am comfortable with than in a
| language I am not as comfortable with, given my other needs
| and priorities are met.
| PunchyHamster wrote:
| The docs of it and the marketing is a bit of a mess, tho I'm
| just gonna blame that on culture barrier as the devs are
| chinese
| woodruffw wrote:
| I agree, although I'm guessing they're measuring "most
| popular" as in "most beloved" and not as in "most used."
| That's the metric that StackOverflow puts out each year.
| riedel wrote:
| >Secure: RustFS is written in Rust, a memory-safe language,
| so it is 100% secure
|
| [0]
|
| qed
|
| [0] https://docs.rustfs.com/installation/
| Emjayen wrote:
| Those rates are peanuts considering that a decade ago
| saturating 40G, per core, was more than reasonable via standard
| userspace networking, with atleast a few copies in the
| datapath.
| PunchyHamster wrote:
| passing blocks of memory around vs referencing
| filesystem/database, ACLs, authentication and SSL
| Roark66 wrote:
| Having just finished a "hobby size" setup of Rook-Ceph on 3
| n100 mini pcs, with every service to fit in a couple hundred MB
| of ram (one service needs up to 3Gb when starting, but then
| runs around 250MB) I'd ask why not ceph?
|
| At work I'm typically a consumer of such services from large
| cloud providers. I read in few places how "difficult" it is,
| how you need "4GB minimum RAM for most services" and how
| "friends do not let friends run Ceph below 10Gb".
|
| But this setup runs on a non dedicated 2.5Gb interface (there
| is VLAN segmentation and careful QoSing).
|
| My benchmarks show I'm primarily network latency and bandwidth
| limited. By the very definition you can't get better than that.
|
| There were many factors why I chose Ceph and not Garage,
| Seaweed or MinIo. (One of the biggest is that ceph does 2 birds
| with one stone for me - block and object).
| PunchyHamster wrote:
| Ceph is far higher on RAM usage and complexity. Yeah if you
| need block storage in addition it's a good choice, but for
| anything smaller than half a rack of devices it's kinda
| overkill
|
| Also from our experience the docs outright lie about ceph's
| OSD memory usage and we've seen double or more than what docs
| claim (8-10GB instead of 4)
| PunchyHamster wrote:
| My small adventure with rustfs is that it is somewhat
| underbaked at the moment.
|
| And also it is already rigged for a rug-pull
|
| https://github.com/rustfs/rustfs/blob/main/rustfs/src/licens...
| evil-olive wrote:
| yeah, their docs _look_ pretty comprehensive, but there 's a
| disturbing number of 404s that scream "not ready for prime-
| time" to me.
|
| from https://rustfs.com/ if you click Documentation, it takes
| you to their main docs site. there's a nav header at the top,
| if you click Docs there...it 404s.
|
| "Single Node Multiple Disk Installation" is a 404. ditto
| "Terminology Explanation". and "Troubleshooting > Node
| Failure". and "RustFS Performance Comparison".
|
| on the 404 page, there's a "take me home" button...which also
| leads to a 404.
| eduardogarza wrote:
| I use this for booting up S3-compatible buckets for local
| development and testing -- paired up with s5cmd, I can seed 15GB
| and over 60,000 items (seed/mock data) in < 60s... have a perfect
| replica of a staging environment with Docker containers (api, db,
| cache, objects) all up in less than 2mins. Super simple to set up
| for my case and been working great.
|
| Previously I used LocalStack S3 but ultimately didn't like the
| lack of persistance thats not available on the OSS verison. MinIO
| OSS is apparently no longer maintained? Also looked at SeaweedFS
| and RustFS but from a quick reading into them this once was the
| easiest to set up.
| chrislusf wrote:
| I work on SeaweedFS. So very biased. :)
|
| Just run "weed sever -s3 -dir=..." to have an object store.
| eduardogarza wrote:
| I'll try it!
| k__ wrote:
| Half-OT:
|
| Does anyone know a good open source S3 alternarive that's easily
| extendable with custom storage backends?
|
| For example, AWS offers IA and Glacier in addition to the
| defaults.
| onionjake wrote:
| Storj supports arbitrary configured backends each with
| different erasure coding, node placement, etc.
| yupyupyups wrote:
| Garage is amazing! But it would be even more amazing if it had
| immutable object support. :)
|
| This is used for ransomware resistant backups.
| tenacious_tuna wrote:
| Anyone know if it's possible to bandwidth-limit the sync
| operations? I'd love to set up garage instances across my
| families' houses to act as a distributed backup, but I don't want
| to hose their (or my) down/uplink during awake hours. Having
| redundant selfhosted S3like storage would solve many problems for
| me, but I really need that capability.
| BOOSTERHIDROGEN wrote:
| I use juicefs
| ianopolous wrote:
| @lxpz It would be great to do a follow up to this blog post with
| the latest Peergos. All the issues with baseline bandwidth and
| requests have gone away, even with federation on. The baseline is
| now 0, and even many locally initiated requests will be served
| directly from a Peergos cache without touching S3.
|
| https://garagehq.deuxfleurs.fr/blog/2022-ipfs/
|
| Let's talk!
| PunchyHamster wrote:
| For someone recently migrating from minio, caveats
|
| * no lifecycle management of any kind - if you're using it for
| backups you can't set "don't delete versions for 3 months", so if
| anyone takes hold of your key, you backups are gone. I relied on
| minio's lifecycle management for that but it's feature missing in
| garage (and to be fair, most other) S3
|
| * no automatic mirroring (if you want to have second copy in
| something other than garage or just don't want to have a cluster
| but rather have more independent nodes)
|
| * ACLs for access are VERY limited - can't make a key access only
| sub-path, can't make a "master key" (AFAIK, couldn't find an
| option) that can access all the buckets so the previous point is
| also harder - I can't easily use rclone to mirror entire instance
| somewhere else unless I write scrip iterating over buckets and
| adding them bucket by bucket to key ACK
|
| * Web hosting features are extremely limited so you won't be able
| to say set CORS headers for the bucket
|
| * No ability to _set_ keys - you can only generate on inside
| garage or import garage-formatted one - which means you can 't
| just migrate storage itself, you have to re-generate every key.
| It also makes automating it harder, in case of minio you can pre-
| generate key and pass then fed it to clients and to the minio key
| command, here you have to do the dance of "generate with tool" ->
| "scrape and put into DB" -> put onto clients.
|
| Overall I like the software a lot but if you have setup that uses
| those features, beware.
| coldtea wrote:
| > _no lifecycle management of any kind - if you 're using it
| for backups you can't set "don't delete versions for 3 months",
| so if anyone takes hold of your key, you backups are gone_
|
| If someone gets a hold of your key, can't they also just change
| your backup deletion policy, even if it supported one?
| PunchyHamster wrote:
| > If someone gets a hold of your key, can't they also just
| change your backup deletion policy, even if it supported one?
|
| Minio have full on ACLs so you can just create a key that can
| only write/read but not change any settings like that.
|
| So you just need to keep the "master key" that you use for
| setup away from potentially vulnerable devices, the "backup
| key" doesn't need those permissions.
___________________________________________________________________
(page generated 2025-12-20 23:01 UTC)