I tried to virtualize / containerize everything, but there's still services I had to run on bare metal. Thoughts? reddit / selfhosted ---------------------------------------------------------------------- Theon · 2023-11-12 09:13 So in the last iteration of my home/community server (NAS + some common selfhosted services), I tried to virtualize/dockerize everything, and I pretty much succeeded... Except for *everything* to do with the NAS. I've got a server running Debian, with a couple HDDs with ZFS on them. That server is also running a kvm/qemu hypervisor and Docker, so the vast majority of services are in there, but everything that needs to touch the ZFS pool, I had to set up on the server directly, since neither Docker nor VMs can access (share) the hard drives directly. That is, rotating (GFS) backups, SMB shares, SMART reporting, overall monitoring - it's actually a nontrivial amount of stuff that I don't have stored anywhere but in the state of the system itself. It all works fine, but I don't like how scattered this is. My ideal is that I just have: 1) my VMs 2) docker compose files 3) docker volumes, and those three "are" my server. Right now, I've got this, *and* a bunch of hand-written `systemd` services, some `sanoid` config, `smartd` config, `smbd` config (user/pass, perms, etc)... I don't think it makes sense to have a VM that actually does the NASing, since then I'd have to... network mount the share on the host from the guest so that Docker can access it? I imagine there'd be some performance loss too. I dunno, I didn't come up with any solution that wouldn't end up being all twisted and circular in the end, but I don't think what I've got is the best possible solution either. Maybe I'm thinking wrong about this, and I should just set up Ansible so that my main server config is reproducible? Or have two physical machines to begin with? I'm interested to hear what you think :) ---------------------------------------------------------------------- ElevenNotes · 2023-11-12 10:00 If you don't want any of that: Build a dedicated NAS, and don't use your NAS as a hypervisor. ---------------------------------------------------------------------- stupv · 2023-11-12 10:33 Yeah, proxmox is what he's looking for ---------------------------------------------------------------------- ElevenNotes · 2023-11-12 10:52 Is he? Why use a hypervisor as a storage server? Unless you talk HCI, that's a clear no from me. ---------------------------------------------------------------------- stupv · 2023-11-12 10:59 He's using a NAS as a bad hypervisor as it is - migrating to proxmox, running storage and share management as a containerised service there, and hosting his apps and services in a fit-for-purpose OS with better support for various deployment types ---------------------------------------------------------------------- ElevenNotes · 2023-11-12 11:18 I think you missed the point where I advertise not to use any hypervisor as a storage server unless you do HCI. ---------------------------------------------------------------------- stupv · 2023-11-12 11:54 If this was a commercial setup, not a homelab, I'd agree with you ---------------------------------------------------------------------- autogyrophilia · 2023-11-12 13:24 Do not use the hypervisor. Use a VM or container. Also, even with HCI I would advise agains using it directly. ---------------------------------------------------------------------- mpopgun · 2023-11-12 16:41 I had a similar thought with my last upgrade. I went with proxmox hci and ceph storage. Somebody rise mentioned HCI...it's been really great for me I have 6 nodes at the present. I just bought them one at a time as the budget allowed. I started out with Lenovo tiny and now I use HP Elitedesk 800 mini g6. Same thing with storage. I have 2 storage pools, one of my ssd/nvme storage where I boot my OS vhds from and then my slow storage for media (hdds). If I need more ram or compute I just add another node to the cluster and spread out my VMs. If I need more storage, add it to one of the nodes in the cluster and ceph will redistribute the files evenly across the drives while online. I map my 2 cephfs to all of my vms so the primary vhd has the os, Docker, etc...then if I need fast storage it's already mapped and so is my slow storage. I map all my Docker containers storage to fast and slow cephfs. All of the data it's stored at the hypervisor level with proxmox and all the vms, containers, and laptops/desktops access the cephfs. You can run a windows vm, mount the two cephfs storages and run your favorite backup software and back all your data up off site. Another benefit I was looking for is not having to rebuild this thing every few years when hardware ages out. Because hci uses nodes...I just had a new node with a newer used eBay computer and when the oldest one dies just move the vms...ceph cares for all the data migration. Proxmox makes ceph and cephfs extremely easy to deploy. You can expose your file share via iscsi or cephs. Windows will need the ceph/dokan driver installed...or iscsi. That's the only small inconvenience. My two cents...hope he helps! ---------------------------------------------------------------------- Mag37 · 2023-11-12 21:43 >everything that needs to touch the ZFS pool, I had to set up on the server itself What do you mean by that? If you mean drive/pool overhead stuff, that is natural - I think that is only accessible by the host or if you fully pass the drive controller to a VM. But if you mean general read/write usage that should be just fine between all layers of abstraction through NFS/SSHFS/SMB? I wanted to split my NAS from my services, ran on different hardware at first but now Ive got split workloads, some on the NAS. I run TrueNAS Scale with qemu/kvm VMs (Fedora Server) that I host services on through podman and docker - acessing the NAS ZFS datasets through mounts. Then a separate machine with Proxmox + VMs, mounting the NAS data and exposing to LXCs and Dockers. I've not done any benchmarks but for my use its good enough. ---------------------------------------------------------------------- Frozen_Gecko · 2023-11-12 22:10 I run a dedicated nas machine in my homelab. Greatly simplifies the infrastructure. ---------------------------------------------------------------------- stupv · 2023-11-12 23:04 OP has 1 device, i was making commentary on single-device setups in a homelab. If you've got multiple devices and thus can dedicate bare metal to a NAS...yes thats obviously a great thing to have ---------------------------------------------------------------------- cr1tic · 2023-11-13 02:08 I tried this since I have enough machines but I wound up abandoning it because of the performance hit when running disk intensive stuff via network rather than directly on the host. Besides optimizing NFS or samba as best as possible (which still really suck compared to native disks) is there anything that can be done? How do cloud providers offer such good disk speeds through their networked storage devices? ---------------------------------------------------------------------- ElevenNotes · 2023-11-13 02:51 For home use, like a few VM's, any decent NAS should suffice. If you have more intensive IO needs 10G helps a lot, but you are correct that local storage is always faster. So how do we do it? Simple: HCI or RDMA or both. Can you do this at home too? Yes, but only really if you need it. Not using HDD but only SSD, especially NVMe helps a lot to get beyond 100G for storage access via network. RDMA keeps the CPU free and allowa fir maximum throughout. RDMA exists for most storage protocols like NFSoRDMA and SMB Direct. You can also simply use NVMeOF to access the NVMe devices via network like local storage. ---------------------------------------------------------------------- cr1tic · 2023-11-13 03:02 Thanks, had to look a lot of that up but it all effectively tracks. So unless I'm using hardware or infrastructure specifically designed for the task, the cheapest upgrade I can do with my consumer grade network and nas is to go from 2.5g to 10g on the NAS and compute machines? This was more or less my take away last time I looked into this, which sucks, because I'd love to turn my QNAP device into a very dumb storage box and have nothing else on it. ---------------------------------------------------------------------- ElevenNotes · 2023-11-13 06:13 NAS = Network attached storage, is a dumb storage box. It’s because of vendors like Synology who added applications and now containers that people start using it like a server, which it isn’t (not enough RAM, low powered CPU). The 10G upgrade is only useful to you if your NAS can handle that much IO which it will not if it does not have SSD caching (better NVMe caching). Adding a 10G NIC to a system that can only provide 2’400 IO will not change anything. A normal SATA disk has up to 600 IO, a normal NVMe has up to 100’000 IO, these numbers just don’t compare.