Proxmox over docker or with? reddit / selfhosted ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 11:33 I’m starting my journey in self hosting soon. I wanted to use Debian or Ubuntu server with docker to run my services, but I’ve been seeing a lot of people discouraging doing this. Why would that be? Would I be better off running proxmox if yes how would I do it? Meaning do I run each service in a vm or lcx or how to group them? ---------------------------------------------------------------------- Treamtx · 2023-12-21 11:50 This heavily depends on various things. 1. what are you comfortable with 2. what exactly are your goals/requirements 3. kinda depends on what hardware you plan to use and what services u plan to run I for myself run everything on a proxmox host. Docker runs in an lxc container. The proxmox approach gives u the ability for snapshots and rollbacks, and the ability to easily nuke entire vms if something goes bad. But for real if you have little to no experience just try both ways. That way you can decide what fit your needs In the end thats the fun in selfhosted. Just try it ;) ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 11:53 Do you run all docker services in one lcx or do you divide them? As for experience I have a little bit of experience with docker and none with proxmox. Goals for me idk just running my own services and maintaining it easily without having to worry much about something breaking bad brining everything down. ---------------------------------------------------------------------- stephenc01 · 2023-12-21 12:01 I have a similar setup. 1 lxc with docker for the arr family and 1 for everything else. Just for backup restore purposes. Look at this for scripts to automate the setup. https://tteck.github.io/Proxmox/ ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 12:14 Thanks for sharing. 👍 ---------------------------------------------------------------------- lesigh · 2023-12-21 13:48 Proxmox host. One Ubuntu server vm with docker installed. 30+ compose services . Easy breezy snapshot backups. Ability to run windows vms or other Linux vms on the side ---------------------------------------------------------------------- Treamtx · 2023-12-21 13:59 I kinda divided them in prod, lab, and dev. ---------------------------------------------------------------------- AnApexBread · 2023-12-21 14:11 >Do you run all docker services in one lcx or do you divide them? I don't run them all in one but I also don't split them to one each. It all depends on importance. If it's a container that's really important I'll run it in a dedicated VM that way I don't accidentally screw it up. For all my testing applications or low priority I'll run in the same VM ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 14:12 Ok I get you. Thanks for the help ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 14:14 Do you run any other vms? And how often do you need the snapshots? I want to know how often things break with this many services. And may I also ask what you’re running? ---------------------------------------------------------------------- lesigh · 2023-12-21 15:10 I run a lot of VMS. I have several Windows and Linux. I run Proxmox backup snapshots weekly and they auto prune x amount as needed. It's just easier and have peace of mind in case anything breaks. It takes 2 min to setup. As long as I provision enough resources to my docker vm my setup never really breaks or goes down Major services are Plex , *arrs, nextcloud, game servers My entire Plex system is automated. My friends and family login via organizrr or Discord and they request anything they want and everything auto downloads the correct format and gets added to plex ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 15:18 Nice, thanks for the info appreciated. 👍 ---------------------------------------------------------------------- mikewilkinsjr · 2023-12-21 16:34 Echoing what users have said below but from a slightly different perspective: I have tended to split up services based on whether they are dependent on a core service or whether they -are- the core service. For instance: 1. I have DNS running in its own container so that an issue with the docker VM doesn't suddenly break connectivity for clients. 2. I have LDAP services running in a separate container for the same reason as #1. 3. I have \*arr services, along with Paperless, running behind Traefik on a docker VM. 4. I have separate dev/production VMs for Docker so I can try updates and config changes without disrupting my fiancé's home office and/or getting killed because Plex isn't working. :D ---------------------------------------------------------------------- notdoreen · 2023-12-21 16:54 I have a Proxmox host with an Ubuntu server VM (with Docker containers on it). Also a Windows server VM and any VMs I want to play around with + test. I have not done any snapshots or backups of my VMs in 2 years and I know 100% that I should ---------------------------------------------------------------------- TheCaptain53 · 2023-12-21 17:25 If you have a decent mount of server resource (lots of CPU + mem) and value flexibility, ProxMox is the way to go. If you only ever plan on using Docker on a VM, that's fine, but you at least have the option of spinning up additional VMs should it be required. With a server running Debian/Ubuntu directly, you would have to do this through KVM, which can be difficult to understand for beginners. That's not to say that a server running vanilla Ubuntu or Debian is bad. For example, my home server just runs vanilla Ubuntu Headless, the reason being is I knew I was only going to run native applications or Docker, no VMs. If I'm literally never going to run VMs, I don't need ProxMox, so I didn't bother. Installing straight Ubuntu or Debian isn't bad, but going for ProxMox does give you more flexibility. Just depends on what you want to achieve. As a side note: if you want to run Docker, make sure you have a VM setup for this. Docker was not designed to run in an LXC. ---------------------------------------------------------------------- -my_dude · 2023-12-21 17:40 Proxmox IS Debian you can do what you want purely in Debian if you want to build it yourself. Use what you're comfortable with or curious about. It's a homelab, use it to learn what you want not what weird nerds tell you. I personally run my NAS and docker containers in separate VMs. Debian for docker, TrueNAS core for the NAS. ---------------------------------------------------------------------- Ursa_Solaris · 2023-12-21 17:43 I'm gonna be the contrarian voice here. I just run a raw Debian server with Docker. I don't plan to run any VMs so I don't need a hypervisor interface, I don't need snapshots and all this junk because I just use Borgbase to backup offsite. If you want something to play with for learning purposes, do it on a separete test machine you don't rely on for your actual services. Keep play and production separate, and keep production as simple as you can while still accomplishing your goals. ---------------------------------------------------------------------- pastelfemby · 2023-12-21 17:55 Learn to do what you want on standard linux, proxmox is only a limiting UI to the standard open tools. It has only been a restrictive headache in negative ways especially as you continue to grow, honestly never looked back after dumping it. Not to mention a lot of their features are woefully behind what the open source tools they utilize actually offer. Seriously I would take any standard linux distro over proxmox any day. No need to wastefully allocate a VM entirely to docker. And for those who bring up 'but limiting resources', do it properly in docker. ---------------------------------------------------------------------- ewixy750 · 2023-12-21 17:59 Exactly my setup too on multiple nodes Proxmox the Ubuntu server, installed docker and using portainer to manage my containers It gives you flexibility and you can add home assistant as a VM on proxmox giving you the ability to have the full experience too Just expect proxmox to take 1 to 2gb of ram by itself ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 18:12 How much cpu power should I expect it taking for a setup like yours? ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 18:13 What is Borgbase? Do you use it to backup docker, if so how? ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 18:15 So let me check if I understand your setup correctly: you group them in a VM if they need each other otherwise they go on separate machines, right? ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 18:16 It's interesting, haven't heard people talking about using windows server much around here. What do you use it for? Is there something were it's better than linux at? ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 18:18 I can't run docker in an LXC? + I'm planning on running my server on a mini-pc and my pc for now the specs are PC: 6c/12t 4.7 Ghz 32 GB mem Mini PC: 8c/16t 4.4 Ghz-ish at start probably 16 (will upgrade it to 64 when I have money for it) ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 18:22 Who are you calling weird nerds??? You're also in this sub. 😅 ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 18:23 oof! How bad was your experience with Proxmox? ---------------------------------------------------------------------- -my_dude · 2023-12-21 18:24 We're all weird nerds here, myself included. Don't listen to a word I say... ---------------------------------------------------------------------- ewixy750 · 2023-12-21 18:32 Frankly, I have a 4th Gen intel i5 and it's way enough for my usage. I think I don't go above 15/20% or CPU usage, with like 20 containers and home assistant. Only limitation is the RAM, 8gb would be limiting a lot but doable to start and test, anything more is comfortable ---------------------------------------------------------------------- TheCaptain53 · 2023-12-21 18:43 Up to you, really. They're both probably okay to run ProxMox. And if they're not... well, you can always trash it and reinstall. ---------------------------------------------------------------------- Ursa_Solaris · 2023-12-21 18:45 I do use it for Docker, yeah. There's these projects called [BorgBackup](https://www.borgbackup.org/) and [Borgmatic](https://torsion.org/borgmatic/). Basically BorgBackup is a minimal program that just creates very efficient snapshots of a folder and sends it to a remote receiver, and Borgmatic is a really simple way to configure and automate it. It's all encrypted before it leaves your system as well, so your data is safe. It preserves permissions and other metadata, too. [BorgBase](https://www.borgbase.com/) is a cheap service run by the creator of Borg, to act as offsite storage for those backups. Lots of services work with it, but I like this one because it funds the actual project. I pay about ~$25/yr for 250GB of storage (and the data is *highly* compressed and each upload only uploads the difference in the snapshot, so that 250GB goes a long way) and I use this to back up all my container volumes, except for my big media. I literally just point it at `/var/lib/docker/volumes`, configure it to keep a certain number of snapshots, and that's basically it. Following the instructions, it took me maybe an hour to register a free tier (10GB) account, learn how it works and set it up, and now it runs like clockwork securing my data, and it'll alert me via email if the upload fails for any reason. On my end, it's just a single config file and a line in my root account's crontab. Couldn't be any simpler. ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 18:51 LMAO 🤣 ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 18:52 Ok, thank you so much for your help. ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 18:53 Thanks, 👍 ---------------------------------------------------------------------- gamerjazzar · 2023-12-21 18:55 Wow, seems like an amazing service. I'll try my best to give it a try. Thanks for the help, you sure did put some effort in this answer and the quality of it speaks for itself. Thanks ---------------------------------------------------------------------- mikewilkinsjr · 2023-12-21 19:20 Kind of. I wrote that previous response in a bit of a hurry. Hopefully the below will make my answer a bit clearer. Basically, if it's infrastructure that I want up if I need to reboot the docker VM --> LXC. Here is my current hardware (not strictly needed for this answer but might be helpful): 1. Proxmox hosts x3 (pve01,pve02,pve03) 2. Ceph shared storage (optional, but handy for doing maintenance/hardware issues) 3. NICs for both client and cluster traffic ​ Here is my setup: 1. LXC running Adguard Home for ad-blocking 2. LXC running Technitium for local/reverse lookups (I could technically just do ad-blocking here but I like AGH) 3. LXC running OpenLDAP 4. Docker VM #1 - Development 1. Traefik 2. \*arr services 3. Paperless 5. Docker VM #2 - Live/Production 1. Traefik 2. \*arr services 3. Paperless ---------------------------------------------------------------------- GOVStooge · 2023-12-21 19:40 Ubuntu(or whatever linux flavor you like) + docker. I'm running plex, the arrs, and a few other things like vaultwarden and mealie. I back up docker config dirs and ubuntu configs with a dockerized Duplicacy(or duplicati, whatever you like). Ubuntu runs my NAS by serving NFS/SMB shares. I view hypervisors almost as outdated tech that fell to containerization. ---------------------------------------------------------------------- Ok_Expert_5751 · 2023-12-21 20:10 Your mileage may vary from many people on /selfhosted. Some of us started before docker was a thing. We were overjoyed when we could migrate things from bare metal to VMs. Personally, I run two docker VMs on Proxmox; docker-prod and docker- devel. Docker-prod is for my production services. I never touch them unless I have a very good reason to do so. Docker-devel is my sandbox for experimentation. ---------------------------------------------------------------------- notdoreen · 2023-12-21 20:14 For me it's a matter of habit. I enjoy the Windows drag and drop file system so all of my torrent downloads go there. I also find it easier to use with the *arr apps because I don't have to do additional work setting up permissions. When I first started with selfhosting my server was a Windows 10 host. I reluctantly started using Linux for Docker containers and now SSH. Eventually I'll probably get rid of the Windows server VM but it works for what I need for now. ---------------------------------------------------------------------- Richmondez · 2023-12-21 20:39 Learning is much easier with virtualisation where you can divide one physical machine up into VMs that you don't need to care about bare metal is the limiting factor when you grow your lab, with VM infrastructure you can add more bare metal hosts and even move VMs from old hardware to new hardware without having to rebuild everything from scratch. There is a reason most businesses don't host on bare metal anymore. ---------------------------------------------------------------------- tenekev · 2023-12-21 21:01 If you don't touch your setup, things are very unlikely to break. If you thinker a lot, stability depends on your ability to implement changes. I snapshot every night at 5am when nothing and noone does anything on my network. Also before risky changes. ---------------------------------------------------------------------- MadeWithPat · 2023-12-21 21:10 This is the way. ---------------------------------------------------------------------- c0delama · 2023-12-21 21:27 How do you backup your 'big media library' then? ---------------------------------------------------------------------- Ursa_Solaris · 2023-12-21 22:31 Currently, I don't. Nothing in there is need-to-have. Right now I just rely on ZFS RAIDZ2 to protect it. If I have data loss there, it'll suck, but it's not the end of the world. I've toyed with the idea of long-term tape storage, but that's a "if I had a whole lot of money and time" kind of project. ---------------------------------------------------------------------- Frozen_Gecko · 2023-12-21 23:20 I'm in both camps. I have a proxmox machine with some vm's on it. A rocky linux machine with mostly docker containers. And bare metal opnsense and bare metal truenas. I have a bit of everything. Just do whatever seems fun to you. You can always just switch it up if you have a good backup strategy. ---------------------------------------------------------------------- gamerjazzar · 2023-12-22 05:12 Thanks for the clear answer. ---------------------------------------------------------------------- SilentDecode · 2023-12-25 16:12 I have ESXi with Docker running. Docker runs on their own low power hardware. I have a 7th gen NUC running Debian 12 with Docker. And a Dell R730 running ESXi, for the stuff that needs more beefy hardware.