Initial Setup of Proxmox Homelab
reddit / selfhosted

----------------------------------------------------------------------
cheeky_panduh · 2023-12-23 22:05

Hey folks,

I'm looking to set up Proxmox on bare metal and create multiple
virtual machines (VMs) for different services like Ubuntu/Debian
servers for various purposes. My aim is to use this as a general-
purpose homelab, running services like firewall, VPN, media server,
file hosting, networking/monitoring and some FOSS software. I also
want to access some services in my private network from the outside
(maybe via Cloudflare Zero Trust Tunnels).

Here's the current hardware I'm working with: HP EliteDesk 800 G5 SFF
(specs: https://support.hp.com/us-en/document/c06414352)

- CPU: Intel i5 9500
- GPU: Intel UHD Graphics 630
- PSU: 250 W (80 PLUS Platinum)
- RAM: Crucial Pro DDR4 2666 (2 x 32GB)
- SSD SATA: Crucial MX500 (1 x 500GB)
- SSD NVME: Crucial P3 (2 x 2TB)
- HDD: Seagate Exos X18 (2 x 16TB)

For services, I'm planning to use:

- Firewall: OPNsense
- VPN: OpenVPN / WireGuard - what is the more robust solution?
- Media: Jellyfin, Sonarr, Radarr, Prowlarr
- File Hosting and Synchronization: Nextcloud, PhotoPrism
- Networking and Monitoring: AdGuard / PiHole, Cloudfare, Nginx Proxy
Manager / Traefik, Uptime Kama, Grafana
- Password Manager: Vaultwarden
- Dashboard: Heimdall / Homarr
- Software Development: GitLab
- Download Clients: Deluge
- Misc: LocalAI, Paperless, Kasm Workspaces, Kavita, Audiobookshelf,
FreshRSS, Invidious (YouTube), Teddit (Reddit)

I'm comfortable with Docker, so I'm thinking of running multiple
Docker hosts in dedicated VMs (like Ubuntu/Debian server) and managing
the services with Portainer.
However, I'm unsure about using Kubernetes and Rancher instead. Is it
worth learning these tools for enabling HA clusters which would result
in a more robust setup?

I'm also undecided on storage / backup solutions. I've heard good
things about TrueNAS Scale since it has built-in periodic snapshoting
and simplified file sharing system (NFS) but using it as a guest VM
might involve extra effort using an HBA PCIe card to passthrough the
disks (HDDs). Is it worth it?

As for file systems I'm also not an expert. What would be a reasonable
configuration for this system?
I've heard ZFS has great features but consumer-grade SSDs might not be
ideal (wearing on heavy workload), so I should only use ZFS for the
HDDs. Any thoughts on this?

In terms of VM setup, should I dedicate a VM for each service category
for better isolation and file sharing? Are there any best practices
you recommend?

Lastly, do you think the CPU and RAM I have are sufficient for these
tasks?
What are your thoughts on the assigned storage dedication (SATA SSD
(EXT4) for boot drive, NVME SSDs (EXT4) for VMs, HDDs (ZFS) for file
storage and backups)?

Thanks a bunch for your input!

----------------------------------------------------------------------
clintkev251 · 2023-12-23 22:17

>I'm comfortable with Docker, so I'm thinking of running multiple
Docker hosts in dedicated VMs (like Ubuntu/Debian server) and managing
the services with Portainer

This is where I started

>However, I'm unsure about using Kubernetes and Rancher instead. Is it
worth learning these tools for enabling HA clusters which would result
in a more robust setup?

This is where I ended up.

Mostly because I didn't love how inefficient it was to allow for VM
failover on a small cluster. That said, docker is about 100x easier to
learn and manage compared to k8s. It's not insurmountable to learn if
you already understand containerization and the payoff is huge, but
it's something that you should really consider how much time you're
willing to sink into.

The rest I can't really comment on. I don't run any significant
amounts of storage on my proxmox nodes outside of rook-ceph and with
k8s I just create the largest possible VM that a given node can handle
and let k8s handle scheduling workloads. If you were using just docker
I'm sure a handful of smaller VMs would be a good idea, though there
are probably people with more experience on that front than me

----------------------------------------------------------------------
Eveley · 2023-12-24 21:08

How did you setup your kubernetes cluster? Do you run multiple VM and
make it each a node for HA in case one VM goes down ? Or are you
running it in a single node?

----------------------------------------------------------------------
clintkev251 · 2023-12-24 22:41

I don't see any benefit in running multiple k8s VMs on a single piece
of physical hardware (except for the purpose of separating CP and
worker nodes). I find it to be much less likely for a single VM to
fail rather than the host itself, so you're not really improving your
availability by having multiple nodes tied to a single failure domain.

I have 2 proxmox nodes which each run 1 k8s (talos) VM which are
combined control plane & workers. I also have 2 other hosts where I
run talos on bare metal, one of which is another combined worker and
control plane, the other of which is a pure worker