Shared S3 backed filesystem reddit / selfhosted ---------------------------------------------------------------------- adamshand · 2023-10-09 06:37 I'm looking for an S3 backed filesystem that I could use for persistent data for containers in a Docker Swarm. My homelab only has a few light users, so performance doesn't have to be amazing. Likewise, I don't care about perfect fail over, I'm mostly just after cheaper storage and convenience. There are a few interesting pieces of software in this space: - ObjectiveFS - commercial, posix, looks great, but cheapest license is US$80/month - JuiceFS - open source, posix, S3 & HDFS, - Goofyfs - open source, posix-ish, multiple clients can read/write - S3QL - open source, posix, only single client can read/write - Rclone (mount/sync) - open source, posix(?) Wondering if anyone else has built something like this, or used a similar piece of software? Any experiences would be great, thanks!! UPDATE: I'm aware of Min.IO and GarageHQ. What I'm trying to figure out is how to connect S3 storage to multiple Docker Hosts/Containers in a way that allows applications which don't natively support S3 (eg. Vaultwarden or Jellyfin) to use it. And secondarily, in a way that allows Docker Swarm containers can fail over between nodes. ---------------------------------------------------------------------- dominikwh94 · 2023-10-09 06:44 I use MinIO as S3 backup target ---------------------------------------------------------------------- tcassaert · 2023-10-09 06:45 I'm currently using JuiceFS in my homelab for shared storage between my Nomad nodes. The apps that use it, are running without issues. My S3 backend is a Minio setup. ---------------------------------------------------------------------- adamshand · 2023-10-09 08:08 Very cool, thanks! A couple questions - how resource heavy is it? - does it with databases (Postgres, SQLite?) ---------------------------------------------------------------------- codeagency · 2023-10-09 08:18 Min.io ticks all my boxes. We are using it for many years in production. Works in simple docker, Swarm and kubernetes. Also taking it along into Kubernetes was easy, they have a kubernetes operator. ---------------------------------------------------------------------- adamshand · 2023-10-09 08:28 I thought minio was just an S3 provider? How are you using it with something like Vaultwarden? ---------------------------------------------------------------------- josemcornynetoperek · 2023-10-09 08:31 Min.io ---------------------------------------------------------------------- codeagency · 2023-10-09 08:36 It's more than just S3 (as protocol) It's a full object storage platform. You can build a storage pool with minio and at same time it gives you an S3 compatible endpoint. But you can also use with other storage pooling like longhorn, ceph, rook, etc... I haven't used it with Swarm. I used it with docker ane than moved on to k3s kubernetes with rancher + longhorn + minio ---------------------------------------------------------------------- adamshand · 2023-10-09 08:40 I see minfs, but it looks dead ... https://github.com/minio/minfs ---------------------------------------------------------------------- Defiant-Ad-5513 · 2023-10-09 09:07 Maybe Ceph is a good option ---------------------------------------------------------------------- adamshand · 2023-10-09 09:14 Thanks. Looks like it integrates nicely with k8s, but I'd like to avoid that complexity at home! ---------------------------------------------------------------------- adamshand · 2023-10-09 09:17 Thanks! It might be, I've looked at it a few times, last time I looked the problem I ran into was that it requires an entire block device which isn't easy on a cheap VPS. If there's a way to get it to work with a generic S3 it would be *soo* useful ... could access via laptop, VPS, homelab ... the dream! But maybe I'm asking too much. :-) ---------------------------------------------------------------------- codeagency · 2023-10-09 09:19 I'm not using k8s. I'm using k3s. K3s with rancher is pretty easy. And fast, mich faster than the big k8s. There is even an easy tool k3d which runs k3s in docker containers if you want to play around with this first. Rancher gives you a web UI to manage the cluster, deploy, run backups, and has a catalog to 1-click install apps like Prometheus, grafana, databases, etc... ---------------------------------------------------------------------- limeunderground · 2023-10-09 09:21 I like S3QL but it is best as a backup/archival filesystem, I would not run anything direct on it - except backup scripts. ---------------------------------------------------------------------- Defiant-Ad-5513 · 2023-10-09 09:34 You can use the v3 version of the deamon to use a directory an example i found is [https://github.com/fatlan/Ceph-Docker- Compose/](https://github.com/fatlan/Ceph-Docker- Compose/blob/master/docker-compose.yaml) ---------------------------------------------------------------------- Stetsed · 2023-10-09 12:28 I currently love GarageHQ, it's similar to minio but is made for selfhosting as its made to be able to handle less than ideal conditions(low CPU power, high network latency etc). I use it and it works great having 2 nodes in my homelab and then 1 node remotely to act as a safety node for if my homelab burns dowm ---------------------------------------------------------------------- Old-Satisfaction-564 · 2023-10-09 13:55 I use a lot minio for storage and I have explored a few routes to mount and access it in a docker container, basically there are 2 apps that provide this functionality rclone and s3fs you can than use a docker volume plugin with both apps: [https://rclone.org/docker/](https://rclone.org/docker/) [https://hub.docker.com/r/mochoa/s3fs-volume- plugin](https://hub.docker.com/r/mochoa/s3fs-volume-plugin) for me the plugin was not a reliable solutions, it was prone to failures over time, but it is the easiest one. You can mount the s3 bucket inside the docker container using both tools, here an example with s3fs: [https://stackoverflow.com/questions/51405965/can-i-use-a-docker- volume-with-s3fs](https://stackoverflow.com/questions/51405965/can-i- use-a-docker-volume-with-s3fs) This is really the best method reliable, the tools are updated every time you update the container, however it requires to modify the container. the third method is to create a docker volume, mount the s3 bucket inside, and share the docker volume, like in this example: [https://www.appsloveworld.com/amazon-s3/6/mount-s3fs-as-docker-volume ?expand\_article=1](https://www.appsloveworld.com/amazon-s3/6/mount- s3fs-as-docker-volume?expand_article=1) this probably needs some work for swarm. The fourth solution (not useful for swarm) is to mount the s3 bucket on the host with rclone or s3fs and remount it inside the docker container, this clearly can be problematic sometimes. ---------------------------------------------------------------------- flaming_m0e · 2023-10-09 15:24 Came here to recommend Garage. It's awesome ---------------------------------------------------------------------- FunDeckHermit · 2023-10-09 18:01 I've tried SeaweedFS, RClone and Garage. RClone is an awesome project but not focussed on S3. SeaweedFS is an awesome project but way too complex. Try [Garage](https://garagehq.deuxfleurs.fr/) or MinIO. ---------------------------------------------------------------------- adamshand · 2023-10-09 19:10 I've looked at Garage, and it seems great! But it's just an S3 provider, right? What I'm trying to figure out is how to connect S3 storage to multiple nodes in a way that allows applications which don't natively support S3 (eg. Vaultwarden or Jellyfin) to use it. And secondarily, in a way that allows Docker Swarm containers can fail over between nodes. ---------------------------------------------------------------------- adamshand · 2023-10-09 19:17 Thanks for the reply! I hadn't considered building S3FS into the container itself. A bit of a pain, but it seems like a really slick solution because it will take care of unmounting / remounting as part of the container fail over. I've mostly used bind mounts, so I'm not very familiar with Docker volumes. Why is this better than mounting the bucket on the host and then remounting in the container? ---------------------------------------------------------------------- adamshand · 2023-10-09 19:17 Thanks for the feedback! ---------------------------------------------------------------------- adamshand · 2023-10-09 19:20 I'm fine with WebDAV, but I'm not aware of cheap cloud storage via WebDAV? Are you selfhosting a DAV server or did you find good cloud storage? Are you using the WebDAV backend to provide storage to multiple hosts & container failover? Thanks!! ---------------------------------------------------------------------- adamshand · 2023-10-09 19:21 Thanks, will have a look! ---------------------------------------------------------------------- adamshand · 2023-10-09 19:24 I've been trying to avoid k*s. :-) But the way it handles backend storage does seem awesome. Maybe it's time to put a weekend aside and try to get a working k3s setup. Hrm ... ---------------------------------------------------------------------- codeagency · 2023-10-09 19:48 I was same like you until 5 years ago. Until I tipped my toe in the water and understood kubernetes solved many other challenges I had. You can make it as simple as you need or turn it into a monster yourself. If you stick to basic principles it's not that difficult. The only "hard" part is relearning some parts that work different in kubernetes vs Plain docker . The biggest "gotchas" and mistakes I had in the beginning was the networking, load balancer service, etc... But there are some very courses that show you perfectly all the differences. Once you got that in your head, it's amazing. Auto failover, auto scaling, rancher supports multi cloud, ... I recently learned about KEDA which lets you scale down to ZERO containers and spin up again once traffic hits your application. Based on Prometheus metrics. Just pure madness this is even possible and very easy. ---------------------------------------------------------------------- adamshand · 2023-10-09 20:10 That's pretty tempting ... ---------------------------------------------------------------------- FunDeckHermit · 2023-10-09 20:17 I've self-hosted a simple WebDAV server using RClone. Then use Rclone mount for directories on other systems. Did you find the [Docker Volume plugin?](https://rclone.org/docker/) ---------------------------------------------------------------------- adamshand · 2023-10-09 22:20 Awesome resource, thank you!