Distributed / scale out storage solutions reddit / selfhosted ---------------------------------------------------------------------- Party_9001 · 2023-12-14 08:53 Hello, I'm experimenting with scaling out my storage over the winter break and would like some guidance. I'll intially only be using it on unimportant data for learning purposes, but I hope to get it 'production ready' sooner or later. I'm mainly trying to reduce network usage during reads instead of georedundancy / HA. I have 2 primary locations and I want to RAID 0 them, as in half a file goes to location A, the other half goes to location B. This way I'll only have to read half the file over WAN. I'm not looking for pooling / JBOD where one complete file goes to location A, then the next file goes to location B because that would mean I have to read the entire file slowly over WAN half the time, making things inconsistent. The sites are connected via VPN, and the machines are running Windows as the host OS but linux VMs are an option. (Can't run windows as a guest OS unfortunately). I'd like to access the pool with SMB / SSH but other options like S3 are also fine. I made a list of solutions and tried some of them, but there's too many to try each in depth. In no particular order; 1. SeaweedFS : Seems promising and I'm working on trying it out. You can tell it which machines are on which rack + datacenter which should help with the data locality thing, but I haven't gotten it to work the way I want yet. I wish there was a nice gui to manage the nodes ans change settings but it looks like there's only a dashboard and a webdav. 2. Ceph : Most recommended, but I cannot figure out how to install it for the life of me. I'm getting a random assortment of errors on WSL and a 'full' VM, not sure what I'm doing wrong. It seems hella complicated too. I've tried ceph on proxmox once and it was pretty straight forward. I ran into some issues but that's probably because I was running 3 instances of proxmox on Hyper-V on windows and making a cluster out of that. Might be worth revisting? 3. MooseFS : Erasure coding is paid 4. CubeFS : Documentation is pretty good, seems to be able to do what I want, except it doesn't compile properly. Maybe it's a 'me' problem similar to my issues with ceph, I have no clue. The webui seems to be Chinese only, but google translate can probably deal with that. 5. LizardFS : No updates since 2017 6. fastdfs : I have no clue what this is 7. BeeGFS : HA / resiliency is paid. Not sure if it's replication based or if it's erasure coding, but regardless it's paywalled. 8. Quobyte : Erasure coding is paid 9. LeoFS : No updates since 2020 10. MinFS : No updates since 2021 11. Nvidia AIstore : Haven't looked into it much, seems complicated. 12. GPFS : Not sure how it's licensed, but it's from IBM so I'm sure there's a catch somewhere 13. OpenIO / Openstack Swift : I nearly had an aneurysm last year trying to figure out openstack before giving up. 14. Longhorn : Not sure if it's Kubernetes specific? 15. Minio (+ JuiceFS?) : Minio does S3, not sure what juicefs is supposed to do. Minio seems to be bad at dealing with small files (not a problem) and I seem to recall it having a 120TB limit for free users but I can't find it anywhere. I'm only storing less than a terabyte for the tests so it shouldn't be an issue, but I'd like to know the restrictions up front. 16. OrangeFS : Never heard of it, not a lot of talk about it. 17. GlusterFS : It seems like people prefer Ceph over GlusterFS. And there's some vague mentions of stuff being deprecated / not maintained ---------------------------------------------------------------------- lilolalu · 2023-12-14 09:02 Unfortunately you failed to mentioned what you are trying to achieve by this... I think also you are mixing up "on prem" distributed storage and Internet-connected distributed storage. In any case, take a look at Tahoe-LAFS and/or gridsync https://leastauthority.com/blog/gridsync-a-new-graphical-user- interface-gui-to-s4-coming-soon/ ---------------------------------------------------------------------- ElevenNotes · 2023-12-14 10:34 None of the storage options you have work over a residential WAN VPN link. Most need RTT < 5ms, so DIA or dark fiber between sites at 10Gbps+ to even barely function. I think you are better of with a simple site-to-site replication instead of clustered object storage, unless you mean clustered object storage on a single site, but even then, you need most 5 or more nodes for a proper HA setup. ---------------------------------------------------------------------- Party_9001 · 2023-12-14 11:10 Ah interesting. Didn't know the performance requirements were that high. I was hoping to learn on my own hardware, but I can rent a VPS or two with 10Gbe and learn off that instead. What should I try? ---------------------------------------------------------------------- Party_9001 · 2023-12-14 11:15 > Unfortunately you failed to mentioned what you are trying to achieve by this... I want to set up my storage in such a way that when I read a file, half of if is always from local storage and the other half is from remote. > I think also you are mixing up "on prem" distributed storage and Internet-connected distributed storage. Is it an issue because of latency? > In any case, take a look at Tahoe-LAFS and/or gridsync Oh I did hear some mentions of tahoe but didn't know much about it. Gridsync looks nice but it looks like I have to set things like the erasure coding setting manually. Is that right? ---------------------------------------------------------------------- lilolalu · 2023-12-14 11:33 To be honest I just used tahoe-lafs, I don't know which parameters of the Tahoe LAFS storage grid can be configured via the Gridsync GUI. When setting up a Tahoe LAFS storage grid via the CLI, you can configure things like redundancy etc very fine grained. There are also tahoe-lafs docker containers, maybe that a middle ground between gridsync and direct install, especially if it's on windows. If this is supposed to go through complex network setups with NAT etc. maybe setting up something like a Tailscale Mesh VPN may be beneficial on top of that - but I guess that would apply to all distributed solutions you are looking at. I don't think that something like GlusterFS is designed for storage were half of your storagr pool is connected remotely through a "slow" vpn connection. I was under the impression it's use case is more independent storage servers which are connected through high speed connections on premise. ---------------------------------------------------------------------- Party_9001 · 2023-12-14 11:47 > When setting up a Tahoe LAFS storage grid via the CLI, you can configure things like redundancy etc very fine grained. The gridsync page makes it sound like it's very difficult to manually set up, but I'll take a look at the docker containers too. Thanks! > If this is supposed to go through complex network setups with NAT etc. Tailscale is already set up and ready to go! Lol > I don't think that something like GlusterFS is designed for storage were half of your storagr pool Yeah none of these were designed with that in mind. It seems like data is replicated between sites and then spread across servers that are 'local' to each other. Theoretically it should work remotely too, but I don't know how much of a performance hit it'll cause. So I'm asking around for some opinions ---------------------------------------------------------------------- lilolalu · 2023-12-14 12:08 You sound like someone who knows admin stuff, so I don't think setting up tahoe-lafs is an issue for you. The Gridsync GUI (how I understand it) is basically for the end-user who has never heard about the existence of CLI's and such. ---------------------------------------------------------------------- Party_9001 · 2023-12-14 12:24 >You sound like someone who knows admin stuff, so I don't think setting up tahoe-lafs is an issue for you I'm just a university student. My major is in mechanical engineering not IT ;-; It's a hobby lol > The Gridsync GUI (how I understand it) is basically for the end-user who has never heard about the existence of CLI's and such. Yeah the gridsync ui seems... Too simple? I'm trying to find a middle ground that isn't insanely difficult to set up, or easy but doesn't do what I want. ---------------------------------------------------------------------- iavael · 2023-12-14 13:38 For learning you can use local 1g network or even VMs. Most distributed storages just won't work well if latency is high (because it affects performance of their quorum algorithms) or if your load exceeds your network bandwidth. But for playground 1g is more than enough. Just don't use it for production. And BTW keep in mind that bandwidth != low latency. ---------------------------------------------------------------------- chrislusf · 2023-12-14 19:38 I work on SeaweedFS. It was designed to be simpler and more scalable. Let me know if you have any questions or just ask in the GitHub discussion, or SeaweedFS slack channels.