[HN Gopher] Ask HN: Show Your Homelab and Home Server Setup for ...
___________________________________________________________________
Ask HN: Show Your Homelab and Home Server Setup for Inspiration
I've been tinkering with quite a bit of self-hosting and home-lab
stuff, starting with a few Raspberry Pis, a 10+ year-old MacMini,
and a few laptops-as-servers. I'm willing to make mistakes, learn,
and be inspired by what you do. I believe many of us would love
your tips, tricks, and all the gotchas in between.
Author : Brajeshwar
Score : 27 points
Date : 2023-12-22 16:00 UTC (7 hours ago)
| firemelt wrote:
| try /r/homelab
| KomoD wrote:
| My homelab is a single 4U with Proxmox and a crappy GPU I never
| got to work
|
| The 4U sits on the floor in a "closet", I think you can picture
| it
| piotrke wrote:
| I use Intel NUC. It is small, quiet and you can choose some
| components as you like. For a homelab, be sure to have a CPU that
| supports virtualization (VT-x), so you can play with VMs. On top
| of that Proxmox.
|
| Like someone already mentioned /r/homelab for more ideas.
| hollerith wrote:
| Many comments on Reddit and such complain about the noise made
| by their NUC, and I hate my NUC8i3BEH because of the noise the
| cooling fan makes. I think some people are sensitive to
| constant sounds of constant frequency and some are not. I.e.,
| for me it is not about decibels, but more about spikes in the
| sound spectrum at a certain frequency that persists as long as
| the computer is powered on.
| manuel_w wrote:
| I feel you. I was very disappointed with the noise made by my
| nuc7i5bnh. Did a bit of research and found out that a) the
| default BIOS fan settings are crap and b) the installed fan
| is crap as well.
|
| Found some recommendations for the fan [1] and BIOS fan
| settings [2] on the internet and got my NUC completely
| silent. Perhaps that works for you as well?
|
| [1] https://www.amazon.de/gp/product/B07C8H5WHP (Out of stock
| and does not fit your model anyway, but perhaps the brand has
| fans suitable for your nuc.) [2]
| https://community.intel.com/t5/Intel-NUCs/NUC7i5BNK-fan-
| spee..., https://community.intel.com/t5/Intel-
| NUCs/Lautst%C3%A4rke-L%...
|
| Did a bit of research and found the following one getting
| recommended: https://www.amazon.de/gp/product/B07C8H5WHP
|
| Installed it and it was SO MUCH quieter.
| Foreignborn wrote:
| Wow I feel like something may be wrong with me.
|
| This isn't the first comment I've read about the noise, but
| I've had an 8i5beh for almost 5 years (just replaced a dead
| fan last week!) and a 13anhi5. I never notice them, they're
| essentially fanless to me.
|
| Currently sits at 40db in my bedroom with the fan at 100%
| 24/7.
|
| Is it a high pitched whine? Or is it like... unfavorable
| harmonics? I have an old brocade switched that had a very
| discordant sound, I did end up replacing those fans.
| bigbuppo wrote:
| Bought three Dell T420s for like $120 each. Loaded them up with
| ram. They're actually quieter than some NUCs in normal operation.
| On the down side, they have obnoxiously bright blue LEDs on the
| back and the little LCD display. A bit of rubylith tape took care
| of that.
| justinclift wrote:
| > rubylith tape
|
| Hadn't heard of that stuff before. Looks cool. :)
|
| * https://en.wikipedia.org/wiki/Rubylith
|
| * https://www.youtube.com/watch?v=q_zInegHR40
| brainlessdev wrote:
| I bought a Biostar J4125NHU motherboard and put it in a small
| case with two 2.5 HDDs and two SSDs and am using it to download
| shows and movies from a Usenet provider and stream them with
| Jellyfin.
|
| The motherboard was a huge pain to work with, and I had to return
| two (!) units; I made the third one work. The first two would not
| boot in the same configuration.
|
| I set up the box with NixOS. Here is its flake:
| https://github.com/fnune/bilbo
|
| The NixOS experience was fun!
| noaoh wrote:
| I've got a NAS, a really nice NetGear router, and four mini PCs
| ewweezdsd wrote:
| Used corporate mini-pcs make excellent home servers if you don't
| need too much storage (typically they support two SSDs), and want
| low power consumption & noise and small form factor. Right now
| ones with 7th or 8th gen i5 are a pretty good deal, often around
| 50-120 USD on ebay, a bit more in the EU. Idle power cosumption
| is about 10W. If you want to play around with virtualization, a
| higher-end model with Intel NIC is recommended due to some Linux
| driver issues (at least with Proxmox).
|
| https://www.servethehome.com/introducing-project-tinyminimic...
|
| Corporate thin clients with Pentium J5005 or similar also make
| decent raspberry pi replacements, usually just not as good price-
| power ratio as proper minipc. Certain models have pcie slot,
| which makes them ideal DIY routers when fitted with pcie NIC &
| pfsense/opnsense as OS. If you're using a consumer router, such
| project can make lots of sense.
|
| For NAS, I recommend building your own. Personally I use Asrock
| J5040 mini-itx board with Node 304 case and Unraid as OS, houses
| 4x 3.5" drives, so far happy with it. For maximum reliability you
| might want a mothebroard that supports EEC memory though.
| Hamuko wrote:
| I have a Synology DS918+. Not exactly the best bang for your
| buck, but the form factor is tiny and fairly good fit for a small
| apartment. For access, I have a PiVPN setup on a Dell Wyse 3040
| because Raspberry Pis were extremely hard to obtain. Eaton 5S
| 550i for keeping both powered.
|
| Need to get a new router though. I have a Mikrotik hAP ac and it
| seems like it's hanging onto dear life.
| sgarland wrote:
| It's been through quite a few iterations, but the current setup
| is a 35U Sysracks (not filled, I just like the space) with:
| * 3x Dell R620 running K3os on clustered Proxmox * 1x X11
| Supermicro 2U with ZFS handling the disk array * 1x X9
| Supermicro 2U, also with ZFS, that is a backup target for the
| other one (WoL to ingest, then shutdown) * UniFi Dream
| Machine Pro * UniFi 24 port PoE 1G switch * 2x UniFi
| UAP-AC-PRO APs (not in the cabinet, obvs) * APC UPS with
| battery extension
|
| The Dells have Samsung PM863 3.84 TB drives for Ceph, which
| Proxmox handles. They have some ancient 2.5" enterprise pull SSDs
| for boot.
|
| The Supermicros have some cheap ADATA NVMe SSDs for boot (the X9
| has a modded BIOS to make this possible).
|
| In general I like it, but the current in-progress work is
| rewriting all of the Ansible playbook that generates QEMU images
| for Proxmox, and setting up PXE so I can grab IPs on boot for new
| images. The latter will enable me to shift off of K3os in favor
| of Talos. Technically I can do so now, but it requires manual
| input to set the IP addresses.
|
| Beyond that, I want to get a 10G switch so a. RBD-backed devices
| are faster b. ZFS send/recv is faster c. Just because.
| justinclift wrote:
| With your clustered Ceph Promox, does Ceph there provide the
| storage to your VM's running in Promox?
|
| And if so, how resilient are the running VMs to the (temporary)
| loss of a Promox node?
|
| Asking just because I'm trying out some clustered Ceph Promox
| stuff currently, and with 3 nodes (so far) the VMs aren't
| coping well the loss of a node. They seem to either freeze,
| fault or hard reset (after a while) when it occurs.
|
| I'm going to try with 5 nodes next and see how it goes. Am new
| to both Ceph and Proxmox, so it could also be something to do
| with my current newbie-ness. :)
| sgarland wrote:
| At the moment, Ceph is a tangential storage solution only
| used for testing (the bulk of my pods use Longhorn), but yes,
| that's what it does - Proxmox sets up Ceph on the NVMe drives
| on its nodes, and then exposes the pool over RBD to VMs when
| you add a disk. In that regard, it's very much like EBS from
| the perspective of the VM.
|
| Nothing stops me from shifting to Ceph for existing
| workloads, but I have it in my head that I need to shift the
| K8s hosting to Talos first. I have used it successfully to
| run MySQL and Postgres instances FWIW. Never had any issues
| with either of those, even with abrupt VM shutdowns.
|
| A VM, if set up as HA, can be live-migrated to a different
| node to perform maintenance with zero downtime. If the node
| just goes away, of course, you can't do much about that.
|
| I confess I haven't used Ceph heavily while dropping a node,
| so unfortunately I can't give you a good answer on that
| behavior, other than ensuring that you have adequate
| bandwidth for both Ceph and Proxmox. I know they both say 10G
| networks are preferable. FWIW I have a completely separate
| switch (UniFi Flex Mini) for Proxmox's corosync; in theory
| the main switch would be fine, but for an extra $30 I had
| more peace of mind.
| justinclift wrote:
| Thanks, that's all good info.
|
| Yeah, the live migration bit is a useful looking feature.
| I'm just investigating ways to reduce single points of
| failure. :)
|
| Live migration is working fine with storage off-the-
| cluster. eg local iSCSI server sharing storage to the
| nodes. But then the iSCSI server is the failure point.
|
| Almost done setting up the 5 node cluster, so will test
| that shortly. Then probably need to read a bunch more docs.
| ;D
| jawmes8 wrote:
| Ansible caught my interest! I setup a dev server and have been
| wondering about using Ansible to install my development
| dependencies on a new OS
| sgarland wrote:
| You're welcome to take a look [0] at what I use now; it's
| designed to be used with Packer to build QEMU images for
| Proxmox.
|
| Also it's not really set up how I now prefer to do things,
| hence the total rewrite that's underway. When that's done,
| I'll archive this repo with a link to the new one.
|
| [0]: https://github.com/stephanGarland/packer-proxmox-
| templates
| replwoacause wrote:
| Dell Optiplex 7060 Micro i7 running Proxmox on bare metal. Works
| awesome and lets me experiment very quickly spinning up VMs for
| each of my projects.
| marginalia_nu wrote:
| The Marginalia search engine ran for over a year on a AMD Ryzen 9
| 3900X with 128 GB non-ECC RAM and a mix of NAS drives and SSDs of
| various types ranging from consumer crap, through enterprise
| drives, and even an Optane. 16 TB mechanical storage, 4 TB SSD.
| All on domestic broadband.
|
| It survived the HN front page and even bigger traffic spikes. Did
| blip out when Elon Musk tweeted a link to one of my blog posts,
| but only momentarily.
|
| Now that server's been relegated to run a test environment and to
| perform various odd jobs.
|
| I do think everyone interested in programming should have some
| sort of server in the house. Being able to run processing jobs
| for a few days really does radically expand what you're able to
| do.
| sphars wrote:
| I'm repurposing an old laptop (an Asus X53SV with an i7, 8GB RAM,
| and 1TB HDD) for a simple server at home that I use for
| Plex/Jellyfin, Navidrome, and some self-hosted webapps, all using
| Docker. Can't handle transcoding media much, so I do that
| separately. Obviously it's an old laptop, a decade old at this
| point, but it handles everything I use it for.
| nonameiguess wrote:
| I'm not gonna show anything but I've done quite a bit. My house
| is a four-story townhouse with the second floor being a single
| room where the kitchen, living room, and dining room are in an
| open floorplan. We put most of our stuff there. My wife built
| shelving into the corner of the den area, wrapping the corner and
| going up to the 12-foot ceiling. She also added a rail-mounted
| rolling ladder, so we've got a nice library setup. Beneath that
| is locking cabinets for craft material and I keep spare
| electronics and cabling in it. We have the same model of cabinet
| mounted underneath the nook for our television where the cable
| hookup is and that's where I keep most of the homelab.
|
| I've got an OPNSense appliance router running FreeBSD. I made no
| attempt to modify it. I tried to build a Linux router but
| realized I can't make anything that takes up this little space
| and the PCIE-mounted NICs to get the number of ethernet ports
| costs a lot more than buying an appliance. I built the NAS server
| myself, using ASRock Rack mini-ITX motherboard with AMD CPUs that
| have graphics integrated onto the chip. It's got a 1 TB SSD cache
| and 8 spinning drives. The machines I use as servers are six
| Minisforum small form-factor PCs, similar to NUCs but fanless,
| cheaper, and with AMD 6-core processors. I don't think they
| outperform or anything, but having more cores makes it easier to
| pin many VMs. These plus the NAS server and television are
| plugged into two Cisco switches that support 10 GbE and two UPSes
| that typically give me about 30-40 minutes in the event of the
| frequent power outages we get in Texas.
|
| I've been tempted forever to try building a "real" server, but
| they're power-hungry and loud and way more than I need. The small
| form-factor PCs have done the job and I can run them in a closed
| cabinet that looks identical to all my other cabinets. The only
| modification I needed to do was install USB fans in the cabinets
| themselves to ventilate the heat, but they don't make any level
| of noise I can perceive.
|
| I've got Aruba Instant-On WiFi access points, one for each floor
| of the house. They run five separate networks, one for work
| devices, one for IoT, one for televisions that aren't hard-wired,
| one for guests, and one for a main non-work WiFi network.
| Everything except the main network is forbidden from sending
| packets to local IPs. I don't know there's much benefit outside
| of that, but it allows me to set the television network to be
| optimized for streaming, low QoS on the IoT and guest networks,
| and make the main network WiFi 6. It's also pretty funny to ban
| porn on the guest network and see which houseguests notice and
| complain about it.
|
| Being in a townhouse, the only walls I have running floor to
| ceiling are either shared or external, which means they're very
| tightly insulated, and running cable between floors was a pretty
| serious challenge. If you're ever going to do it, I would
| recommend doing it immediately upon moving in, before you do
| anything else, before you even move in furniture. I'm pretty
| serious about cable management and keeping things neat, so I run
| everything I can through walls and/or floors. No cable is loose
| except at the last mile.
|
| As for the self-hosted services, I don't use any sort of on-prem
| management layer like vSphere or Proxmox or anything. It's all
| Arch Linux with libvirt running on hardware-accelerated KVM. I at
| least automated the Arch builds by putting provisioning scripts
| on USB drives with a "cidata" label since the Arch installer
| comes with cloud-init and you can use this for unattended
| installs by just in two drives at first boot instead of one. Most
| everything else runs on Kubernetes, with Longhorn as a storage
| provider. I use Ansible playbooks to install Kubernetes and the
| applications are installed and configured with GitOps. So the
| external services are a Git server and Minio on the NAS acting as
| a backup target for anything that will backup to S3 as well as a
| package mirror and image registry so I can provision everything
| without Internet access. I load balance the control plane with
| kube-vip and Ingress with MetalLB, using the L2 advertisement
| features.
|
| Unbound on my router is configured as the default DNS for the
| LAN. This recurses to NextDNS. I block outbound port 53 to try
| and ensure everything is actually using this, but there isn't
| much you can do to block DNS over HTTPs without a MITM proxy.
| Pretty much any known ad source, telemetry, tracking domains, is
| all blackholed at both the Unbound and NextDNS layers. It doesn't
| seem to break too much. The Paramount+ app stopped working, but
| their actual content is streamable through Prime Video using the
| same subscription. I should probably just cancel it, but they
| have SEC and NFL football that I still geek out for plus I've
| been rewatching Aeon Flux and Daria and they own the MTV back
| catalog.
| q0uaur wrote:
| just running my old desktop PC with proxmox.
|
| most important thing for my homelab i ever did was starting to
| mess around with VMs. first with virt-manager to have a gui, then
| just plain kvm (actually probably qemu) on the commandline using
| scripts to auto-install vms, and now a combination of Proxmox and
| Ansible to make a vm with just a few keystrokes. The freedom of
| messing around as much as i want without danger of breaking my
| system is the best.
|
| next most important was setting up dns and dhcp automation -
| whenever the ansible playbook makes a vm, it immediately reserves
| a static ip for that mac and makes a DNS entry for that vm's name
| pointing to it's ip - so i can easily reach all my vm's by name
| without having to go through the effort of setting that up. great
| for when a test vm becomes permanent and needs to be accessible.
| this combined with using ansible to automatically create my user
| and put my ssh pubkey into it's authorized-hosts file makes
| everything SO smooth, no friction at all to work on different
| vm's and experiment.
|
| also, when the ssd proxmox was installed on died, it was super
| easy to restore all the vm's.
|
| next step is probably automating public dns, port forwarding and
| reverse proxy settings - so far i am doing that manually, but i
| just got a new phone and refuse to log into google on it, so i've
| had to set up a bunch of services, kinda running my own cloud,
| and they all need port forwards and reverse proxy entries.
| speaking of reverse proxy... it's a wonderful tool, set up
| ssl/https ONCE on it and use certbot to manage certs in one
| place, and you can test any service by just forwarding it to the
| vm without ssl, no hassle. for critical things i guess i'd still
| recommend setting up ssl for your internal network too, defense
| in depth like.
| tetris11 wrote:
| RPi4 running DietPi, with a 1TB USB drive, all strapped to the
| back of the TV. Syncthing for backing up phone media, Hugo for my
| blog posts, HomeAssistant for my ZigBee devices, Jellyfin for my
| media stack which network mounts my Synology NAS (auto
| wake/suspend), Moonlight client to game in the living room with
| PlayStation six axis controller, private git. All reverse proxied
| behind a CG-NAT to a single public IP4 VM I pay maybe 10EUR /
| year for.
| astromechza wrote:
| Personally, I've really enjoyed using a 1U 16 core Atom server as
| a single node Kubernetes cluster. I started on raspberry PIs and
| Synology NASs too but consolidated them earlier this year and
| didn't look back. It runs a whole bunch of stuff now and I
| blogged about the setup at
| https://bensblog.meierhost.com/20230705-home-lab-infrastruct....
|
| Totally understand that it's a bit more expensive than second
| hand hardware and smaller elements so I'd only suggest going down
| that path once you know you'll get the utility out of it.
|
| I've used it to replace my reliance on Google photos/docs and
| lean more on self hosting, though that means that backups, disk
| mirrors, and run books for restoring everything is double
| important!
| extragood wrote:
| I've been running TrueNAS for years on commodity hardware that's
| been upcycled from my personal workstation after major upgrades.
|
| Current iteration is: a skylake era CPU, 16 gb ddr4, 4x 14TB WD
| Reds, 2x 512 GB SSD, and a m.2 boot drive
|
| Used primarily for shared storage, backups, media serving, and
| lightweight docker containers (home assistant, git tea, etc) in
| an Ubuntu VM.
|
| It's been super stable (paired with a UPS) and was surprisingly
| economical for what it is. I've also learned a lot in the
| process.
|
| I do wish my CPU had a few more cores though.
| CommieBobDole wrote:
| Mine is mostly various cast-off enterprise hardware, living in a
| semi-finished room in the basement. From bottom to top:
|
| - Mid-'90s 42U IBM server cabinet
|
| - Two 1500VA APC UPSes (non-rackmount)
|
| - 1U Dell R610 (PFSense firewall)
|
| - 1U Dell R620 (ESXi host)
|
| - 2U Dell R720XD (60TB NFS/CIFS storage server)
|
| - 4U rackmount case containing a Windows gaming/workstation PC
|
| - 4U rackmount case containing a Linux workstation (the previous
| iteration of the gaming/workstation PC)
|
| - Black Box KVM for the two workstations
|
| - TP-Link unmanaged 10GB switch
|
| - Cisco managed 1GB switch
|
| - Cable modem
|
| - Two 1U PDUs plugged into the UPSes - half-ass dual power path
| for the machines that support it.
|
| It's a pretty nice setup; I have a bunch of system service VMs on
| the ESXi host (IDS, Splunk, etc) and can spin up a new one in a
| few minutes whenever I have a new project or want to try
| something out. The cables from the KVM go through the floor to my
| office on the main floor so I don't have to listen to fans and
| can switch with a key combination. And of course I have plenty of
| storage for whatever - I back up all of my VMs and my
| workstations to it, I can download pretty much anything I need
| to, etc.
|
| And since it's been asked before, yes, electricity is oddly
| inexpensive where I live.
___________________________________________________________________
(page generated 2023-12-22 23:02 UTC)