Finally happy with my home lab reddit / selfhosted ---------------------------------------------------------------------- Prettynewham · 2023-12-28 01:00 Finally happy with how I've got my home lab set up. I've been working with different hardware and software to try and find the right balance between power usage and ability to run what I need. Here in the UK, energy prices are currently insane! I spent a lot of time on the monitoring setup with Prometheus, Alertmanager and Loki, it's working really well. I didn't include much detail on the diagram of the monitoring setup, but feel free to ask any questions. Keen to hear your feedback, suggestions or questions. Merry Christmas homelabbers, keep on labbin'! ---------------------------------------------------------------------- 9Hero · 2023-12-28 01:49 Out of curiosity, what was the reasoning behind choosing Oracle Cloud vs. other providers? ---------------------------------------------------------------------- Reverent · 2023-12-28 01:53 Just a note, layer 3 switches are a different concept from managed switches. Layer 3 switches are essentially big routers, most switches are layer 2 switches (or used as layer 2 even if they have layer 3 capabilities). ---------------------------------------------------------------------- Prettynewham · 2023-12-28 01:54 Yes, you're right, it's a managed switch with L3 VLAN routing capabilities, as opposed to a layer 3 switch. ---------------------------------------------------------------------- Reverent · 2023-12-28 02:03 Yeah, quick google says it's got some internal QoS capabilities it can turn on (though trying to troubleshoot that at the *switch* sounds like a nightmare I would sidestep entirely). ---------------------------------------------------------------------- Prettynewham · 2023-12-28 02:05 I tend to stick to the simple features, just set up the VLANS on the ports and it works as I need it to. Not needed to dive deeper in to the features. As a principal though I agree, a switch should be fairly "dumb" unless you really have a specific need. ---------------------------------------------------------------------- Prettynewham · 2023-12-28 02:06 Just because of the generous free tier, 4vCPUs and 24GB RAM instance hosted in London. 100% free. I've heard people had their account cancelled before, but apparently to prevent it, upgrade your account and just stick with the free tier and you'll be golden. ---------------------------------------------------------------------- guptaxpn · 2023-12-28 04:33 Backups? ---------------------------------------------------------------------- Prettynewham · 2023-12-28 04:51 For backups I've got Duplicati running on TrueNAS. It backs up the shared NAS data, container config / storage to Backblaze E2 (S3 compatible storage). Jenkins runs daily jobs to backup the Opnsense, TrueNAS and home assistant config in to an NFS backup mount (the same one that's backed up to Backblaze). It prunes old backups so it doesn't get too full. There are also snapshots taken within TrueNAS itself, as well as replication to just a local USB M.2 drive enclosure. It doesn't back up everything to local storage, just critical stuff, same with Backblaze. Anything I can't easily reproduce or redowload gets multiple local backups (if you consider a snapshot a backup) and an off site one too. I'm yet to update my diagram with the backup details, but I shall in the future. ---------------------------------------------------------------------- guptaxpn · 2023-12-28 05:02 I feel like people fail to share their backup schemes and that's really holding me back from implementing a real one. I have no idea what I'm doing. ---------------------------------------------------------------------- Prettynewham · 2023-12-28 05:04 Feel free to PM me and I can go in to further detail on what I've done and try and give you some advice based on your requirements 🙂 ---------------------------------------------------------------------- rickman1011 · 2023-12-28 05:59 Would you mind sharing more details on your supermicro OPNSense box? ---------------------------------------------------------------------- Lirionex · 2023-12-28 07:48 If you're using bare prometheus/alertmanager I highly recommend migrating to the grafana agent in flow mode + grafana mimir + MinIO. Idk why but to me it always felt like my metrics aren't "save" in a single or even a federated prometheus environment. But using agent, mimir and minio feels so much more stable and secure because its so easily scalable. ---------------------------------------------------------------------- Theon · 2023-12-28 08:01 /r/homelab ---------------------------------------------------------------------- xCharg · 2023-12-28 08:10 Why auth app is in the cloud? ---------------------------------------------------------------------- Prettynewham · 2023-12-28 09:43 Because for it to work as I needed it to (be my SSO for everything) that I included one or two internet SaaS applications, these solutions wouldn't have direct access to my network so I'd have a few options: - Expose my home network to the internet to the internet via tunnelin: This would work mostly fine, apart from I'd also need to expose port 636 for LDAPS (couple of apps use this). Also my thinking was that ifor my public facing apps, if they're on a separate server and one of my public facing apps is compromised, then you've not straight away got to the crown jewels - so to speak - you'd have to move laterally across the network with some fairly specific firewall rules in place which means there isn't much of an attack surface. - Expose my home network to the internet via port forwarding: To me this has all the same (acceptable risks) as above, but with the addition that you're straight in to the management network (where everything is hosted and my data is stored) if you compromise an app. I also try as much as possible to not port forward anything if I can, I've managed to do that even setting up a SIP IP phone. - Creating a separate server in the cloud: This of course is the option I went with due to what I mentioned above. It may be overkill and a bit risk-averse, but it works well for now and let's me sleep better at night. I do think there should be some demarcation line between internal and public facing apps, but for most this could likely be a separate docker network. ---------------------------------------------------------------------- Prettynewham · 2023-12-28 09:55 Sure, hardware wise it's a Supermicro 1U chassis with a X11SSH-LN4H motherboard, 16GB DDR4 RAM and running an E3-1220 v5. It's got a fairly small SSD for the OS too. I did a lot of tinkering in the BIOS (of both the rackmount servers I use), and while I'm not sure about the power usage of this specifically, all together (both servers + network equipment) it chugs along at around the 140W mark. ---------------------------------------------------------------------- Prettynewham · 2023-12-28 10:09 I'll look in to this, thanks! 🙂 ---------------------------------------------------------------------- rickman1011 · 2023-12-28 15:34 Interesting, thank you. I’m planning on adding an additional node to my proxmox cluster dedicated to OPNSense and adding either HA or at least redundancy for the more crucial services, such as DNS and AD that I’m currently running on a R730xd and some intel NUCs. I’ve been contemplating what hardware is really necessary for this, I was of the understanding that VPN client, VPN server, suricada and the rest of your security stack would require more resources to operate at your line speed than your build. I have 5gbps fiber and 5G failover WAN2, in my head I’ve been toying with a mid spec R430 or R630 with 64-128GB RAM. Besides a similar security stack I’m going to be running 1-2 WireGuard servers and 4-6 WireGuard clients on OPNSense. Am I nuts and could get the performance I need with way less? ---------------------------------------------------------------------- [deleted] · 2023-12-28 16:57 [removed] ---------------------------------------------------------------------- Prettynewham · 2023-12-28 18:34 Wireguard VPNs come with very little overhead if (if I recall correctly) your CPU has AES-NI instruction set, most fairly recent CPUs do. So I wouldn't worry about VPNs, they'll work fine and I have used standard wireguard on Opnsense before I switched to Tailscale. Even with the numbers of VPN servers and clients you suggested, I think my setup could easily run that (based only on my experience so far with 1 VPN server and client - it basically used no resources 🤣) Suricata, set up just to monitor the WAN interface using Hyperscan runs up to about 40% CPU usage when utilising the full 1G line speed. I think this is acceptable. I also have Suricata monitoring my 4G WAN failover, which isn't included in this diagram. With regards to virtualizing, I assume you already know, but you'll see a slight performance decrease, but only very slight. As long as you do pass through on your NICs which are running your WAN / LAN you'll be fine. I ended up moving away from virtualizing my firewall because I'm always tinkering with my hypervisor and containers. With mine being a physical appliance now, I can take down my hypervisor and the internet still works - keeps my wife happy! ---------------------------------------------------------------------- Prettynewham · 2023-12-28 18:36 You're not nuts, but you certainly don't need anything close to 64 / 128GB RAM, unless you're running something I'm not aware of. 5Gbps line speed will of course require a good CPU if you're running Suricata on your WAN interface. ---------------------------------------------------------------------- No-Refrigerator5648 · 2023-12-28 20:54 Could you tell me how you got such resources from Oracle in the free tier? I can see only 1 vCPU and 1 GB RAM. ---------------------------------------------------------------------- juanclack · 2023-12-28 21:27 Would you mind explaining the reasoning behind using nested virtualization with Proxmox for the LXC containers? Your setup inspires me. ---------------------------------------------------------------------- Prettynewham · 2023-12-28 21:39 Literally no other reason than I like the Proxmox UI for managing LXC / LXD. Plus it sets up all the networking nicely for you (the bridge you'll need if you want containers on your LAN IP ranges). I did try it with Canonicals LXD UI, which is a really nice UI, I just prefer Proxmox. I like having LXC containers for quick places to play around in, test new things away from my "production" docker host, and without having to spin up a whole new VM. TL;DR: Not a huge amount of logic behind it, I just like the Proxmox UI. ---------------------------------------------------------------------- Prettynewham · 2023-12-28 21:41 Use the Ampre instances (arm64), not the standard ones 🙂 ---------------------------------------------------------------------- BeefcakeUK · 2023-12-28 22:20 What did you use to make the diagram? ---------------------------------------------------------------------- rickman1011 · 2023-12-28 22:31 Great. Thank you for both informative comments! I had that amount of ram in mind for two reasons. A) Relatively cheap used server DDR4 availability, B) Extra headroom to shuffle services off of my main hypervisor if necessary ---------------------------------------------------------------------- Prettynewham · 2023-12-28 22:36 Draw.io, always used to use this in my job doing cloud architecture, so felt right even if it isn't entirely the cloud. It's web based and free! ---------------------------------------------------------------------- Prettynewham · 2023-12-28 23:39 If you've got the budget and can stomach the power bills, then go for it! It'll definitely run your firewall software with ease, whether you go for a lower power smaller build like mine and keep it a hardware appliance, or you virtualise it on a bigger server is down to choice. One note from me about clustering, and even iSCSI stuff is that I'd steer away from it unless you've got a 10G backbone. That's not to say you can't do it, but to move a VM with it's storage, that could be a looonngg time over a 1G link. You generally circumvent that by having your iSCSI LUNS presented to your hypervisors, so you essentially just move the VM config and not the data. Again, 10G I'd recommend if you're going to have a decent number of iSCSI volumes - but it's certainly not required. When I clustered Proxmox, I found myself never really migrating VMs. 1. Firewall - Had PCI passthrough, so couldn't move that to a new hypervisor. 2. Containers host - Generally has USB devices connected - UPS for monitoring, ZigBee dongle for lights and switches. So also couldn't be moved. ---------------------------------------------------------------------- juanclack · 2023-12-29 00:41 Thanks for the response! My main host is just Ubuntu with some containers for now. Have another I set up with VMWare to migrate to but with the Broadcom acquisition I’m thinking of moving over to Proxmox. ---------------------------------------------------------------------- AdministrativeBit115 · 2023-12-29 05:13 Nice mate ! Just curious if all were backed with a UPS and an estimate how much it cost to run those devices monthly? If it goes on 24/7 ish. Thank you ---------------------------------------------------------------------- HoustonBOFH · 2023-12-29 05:39 If we are going to get specific, layer 2 is switch. Layer 3 is router. "Layer 3 switch" is a marketing term with no real meaning. It can be damn near anything. ---------------------------------------------------------------------- Reverent · 2023-12-29 05:58 No? Layer 2 is the data link layer. A layer 2 switch can do this. A layer 3 switch can also do this. A router can also do this. What a layer 2 switch can't do is layer 3 operations, such as routing. Rule of thumb is the protocol involving mac addresses? Then it's layer 2. Is the protocol involving IP addresses? Then it's layer 3. It's not product based. ---------------------------------------------------------------------- LoganJFisher · 2023-12-29 06:07 I've never used Draw.io but I'd like to replicate this for myself. Could you share the file for this so I can modify it? ---------------------------------------------------------------------- HoustonBOFH · 2023-12-29 13:41 Read your 7 layer model again. Layer 2 is switching. Layer 3 is routing. "Layer 3 switch" is a marketing term. Sadly, it is an accepted one now. ---------------------------------------------------------------------- Prettynewham · 2023-12-29 14:14 Yeah so the downstairs rack is backed up with an APC 1400w UPS. From the wall the entire rack, servers and all networking equipment is drawing around 140w. The other switches and the NVR are backed up by the Amazon basics UPS, it's basically just a really chonky extension lead with a big lead acid battery in it, gives me an extra hour or so on the NVR / Middle floor switches and IP phone. I'm not sure of the energy usage of those ones, but I doubt it's a whole lot. So costs roughly £30 per month in energy usage to keep the whole thing running. ---------------------------------------------------------------------- Prettynewham · 2023-12-29 16:08 Proxmox is great in general, I like it. Only problem I ever had with it was when I clustered it, it seemed to be unstable, nodes dropping, etc. I suspect it was probably a "me problem" though. ---------------------------------------------------------------------- Reverent · 2023-12-29 21:28 I would except the packets fell out of my router on the way to my desktop because apparently I need a switch in between to do the layer 2 part 🙄.