If you could start from scratch what would you do differently? reddit / selfhosted ---------------------------------------------------------------------- nosliw_rm · 2023-11-20 20:02 One thing i've learned over the years is the way I did something when I was learning is usually not the best way of doing it, and I later go back and redo everything now that I have gained experience and knowledge. I have decided to ditch google photos, and was lucky enough to snag a free rackmount 2017 server from work with an i7 installed and 2 6tb drives on the way. But now comes the hard part of deciding what software I am going to end up learning on and hopefully living with. First and foremost I want a photo backup service, and I have debated between immich and xpenology, I also know that I want to run Pihole and would really like to self host my own website documenting my projects, even if no one will ever look at it. ​ If you had to start from the beginning, which OS, which container manager and which containers would you build. I would love the recommendations from those who walked so that I can run ---------------------------------------------------------------------- No_Dragonfruit_5882 · 2023-11-20 20:13 I wouldnt be so stupid and put cash into 10 gbit nics again, since i swapped them after 1 month for 40 gbit nics ---------------------------------------------------------------------- tech2but1 · 2023-11-20 20:22 2 things. Document everything. Regarding picking a different OS though, if I was starting again now then maybe yes. However the choices were different when I started so what made sense some amount of years ago might not make sense now. Do I want a different hypervisor? Yes. Do I regret picking the one I use? No. ---------------------------------------------------------------------- nosliw_rm · 2023-11-20 20:23 Hah good to know ---------------------------------------------------------------------- No_Dragonfruit_5882 · 2023-11-20 20:25 And the documentation. Totally forgott about it since i dont have any ---------------------------------------------------------------------- No_Dragonfruit_5882 · 2023-11-20 20:26 Offtopic as fuck, but what did you use to document your stuff? Onenotes etc? ---------------------------------------------------------------------- SnidelyRemarkable · 2023-11-20 20:27 Check out BookStack ---------------------------------------------------------------------- Accomplished-Lack721 · 2023-11-20 20:28 I started off using Qnap's Container Station, not quite understanding how anything worked. CS is workable, especially in the newer versions, but I wouldn't have tried to do anything with individual container installs and menu/config options in the UI. Instead, I'd have gone right into its Applications section, where I could create (and edit/recreate) "applications" using Docker Compose yaml (basically how Portainer does stacks). I'd have developed skills and understandinh I could transfer to Docker Compose or other GUI front-ends a lot faster that way. ---------------------------------------------------------------------- No_Dragonfruit_5882 · 2023-11-20 20:29 I love you man #nohomo. Installing it right now ---------------------------------------------------------------------- tech2but1 · 2023-11-20 20:31 DokuWiki. ---------------------------------------------------------------------- M-fz · 2023-11-20 20:36 Never offer Jellyfin to friends / family so I could downgrade to a much smaller, cost efficient server. Hell if I didn’t have to stream outside the house I’d probably use a raspberry pi for the rest, those things are super capable now. ---------------------------------------------------------------------- MRobi83 · 2023-11-20 20:37 I'm sitting here thinking of my servers and all the services I host and wondering what I would actually document. I use proxmox and name containers based on their content. I use the notes to record the IP of the container. And whatever it is running in that container already has its own documentation. So I feel like if I were to document things I'd be like "Sonarr... Running in Arrs container. Documents here: LINK" which wouldn't give me much lol So what am I missing here? ---------------------------------------------------------------------- No_Dragonfruit_5882 · 2023-11-20 20:43 If you got family members etc you should document everything for noobs. Imagine beeing hit by a Bus and Service etc doesnt work anymore for your family. Or just to get some Personal data from you after you passed. I didnt document anything yet because i know what everything does etc. But after thinking about it, i should at least write some tutorials for logging into proxmox / Updating the Firewall etc ---------------------------------------------------------------------- tech2but1 · 2023-11-20 20:46 I document exactly what I did to install everything, what commands I used, how I set up the environment, how I set up the cron for the backup, the exact restore procedure etc. Literally everything. I guess what you're referring to is more like just installing a package on something, so like for instance I don't document every package I installed in DSM. Also some general thing I do on every install, setting up the environment in Ubuntu server (user group, adding me as sudo etc). My lab is a lab, as in there's lots of things that are test/dev for things I do for work so a lot of it is experimental/research/trial and error. An example of where I fucked up here, before I documented things I installed Nginx Proxy Manager in Debian then found out I needed to update it. Fucked if I can work out how I installed or got it working outside of docker so I was stuck with it until I learned docker so I could get an updated supported version in the lab. ---------------------------------------------------------------------- MRobi83 · 2023-11-20 20:53 I feel if I ever got hit by a bus that my family wouldn't even try to keep things going. My stuff would just get unplugged and sold. ---------------------------------------------------------------------- No_Dragonfruit_5882 · 2023-11-20 21:02 Bruh. Well they can do it with mine, but then 2 companys wont work and they would starve to death. Depends on what you are hosting ofcourse, but still having some data of a person that is gone is at least something to hold onto ---------------------------------------------------------------------- tech2but1 · 2023-11-20 21:04 Mine might but then they'd realise that their password managers don't work, their cloud files are gone and the Windows VM they run their businesses from are AWOL. It'd get plugged back in pretty quick once they join the dots! ---------------------------------------------------------------------- Ursa_Solaris · 2023-11-20 21:15 I already apply these rules myself, but these are the five major things I emphasize to everyone. 1. Don't overcomplicate things. You don't need Proxmox on every machine "just in case". Sometimes, a system can be single purpose. Just using Debian is often good enough; if you need a single VM later, you can do that on any distro. This goes for adding services, too. Docker makes it very easy to spin things up to play with, but you should also know when to put things down. Don't get carried away, you'll just make more work for yourself and end up slacking and/or giving up. 2. Don't put all your eggs in one basket if you can avoid it. For instance, something like Home Assistant should run on its own system. If you rely heavily on your NAS, your NAS should be a discrete system. Anything you rely on should be resilient so that your tinkering doesn't leave you high and dry. 3. Be careful who you let in. First, anybody with access to your systems is a potential liability to your security, and so you must choose your tenants carefully. Second, if others come to rely on your systems, that drastically reduces your window to tinker unless you have a dedicated testbench. Sharing your projects with others is fun and good experience, but it must be done cautiously and with properly set expectations. You don't want to be on the receiving end of an angry phonecall because you took Nextcloud down while playing around. 4. Document when it's fresh in your mind, not later. In fact, most of the time you should document it *before* you do it. If things don't go according to plan, make minor adjustments. And *update docs when things change*. What you think is redundant info today might save your ass tomorrow. 5. Don't rely on anything you don't understand. If it works, and you don't know how or why it works on at least a basic level, don't simply accept it and move on. Figure it out. Don't just copy and paste, don't just buy a solution. If you don't know it, you don't control it. ---------------------------------------------------------------------- John_Mason · 2023-11-20 21:15 I would’ve just sprung for a 12TB drive out of the gate for my Synology NAS/home server. I started with a 4TB, and because it’s only a two-drive system, I can’t really get any more space (if I want to keep redundancy) unless I now buy a new drive. My second drive is now 12TB for redundancy, but it’s not using 8TB of that. ---------------------------------------------------------------------- dazchad · 2023-11-20 21:18 > Totally forgott about it since i dont have any makes sense ---------------------------------------------------------------------- No_Dragonfruit_5882 · 2023-11-20 21:20 Iam starting now xD. Already got my first page written xD. Prolly 600 more to go ---------------------------------------------------------------------- dazchad · 2023-11-20 21:20 You can preemptively transcode your media to a more well supported version (1080p h264 SDR) and disable live transcoding on your Jellyfin setup. That way Jellyfin will work as a glorified file server. ---------------------------------------------------------------------- Raithmir · 2023-11-20 21:22 I'd start off with three SFF PC's instead of a single more expensive server. Most people don't need expensive servers running Xeons and ECC memory. More capacity across the the PC's with better redundancy and HA for important services. ---------------------------------------------------------------------- dazchad · 2023-11-20 21:24 > Don't overcomplicate things The amount of people that wants to deploy a Kubernetes cluster is mind bending. Even small-medium businesses more often than not don't need it, let alone your 2-device homelab. ---------------------------------------------------------------------- root42_ · 2023-11-20 21:34 This. I originally started out with a stack of Dell servers. They were noisy and power hungry. I was able to replace all of that with a single NUC like device (Beelink to be specific). ---------------------------------------------------------------------- nosliw_rm · 2023-11-20 21:47 That’s the eventual plan but I’m working with what I got for the moment ---------------------------------------------------------------------- tech2but1 · 2023-11-20 21:52 Well, most of don't *need* a server of any sort! I started out with a single desktop PC running IPCop, then added another old desktop running FreeNAS and it just kept growing. I didn't set out to have multiple servers with multiple Xeons and hundreds of GB of ECC memory, it just sort of happened! ---------------------------------------------------------------------- ElevenNotes · 2023-11-20 21:55 Till you swap them for 100G only to replace them again for 200G till you run out of PCIe lanes 😜. ---------------------------------------------------------------------- No_Dragonfruit_5882 · 2023-11-20 21:59 Ohh god. 88 pcie lanes..... ---------------------------------------------------------------------- Bagel42 · 2023-11-20 22:00 Document and get a more powerful server. ---------------------------------------------------------------------- ElevenNotes · 2023-11-20 22:01 It really sucks if you have a dual port 200G with 8 NVMe drives in the same system. ---------------------------------------------------------------------- iRawrz · 2023-11-20 22:02 I'm so split on this myself. I have my Plex server shared with close to 20+ people. It's almost always in use, it maybe gets 5 hours of idling a week. I often think about how much I could downsize if I stopped. But they also will pass money my way from time to time. I just had a series of failures and multiple people passed me more than enough money to replace the failed hardware without me even asking for it. That wouldn't happen if I kept it to myself... I'm slowly revoking access to people I haven't talked to in multiple years but I don't think I'll ever be able to stop sharing outright, I feel obligated now that others have pitched in. ---------------------------------------------------------------------- No_Dragonfruit_5882 · 2023-11-20 22:03 Already got 6.... In case i max them out with networking please send me into a psychiatry ---------------------------------------------------------------------- Richmondez · 2023-11-20 22:04 Counter point to not using proxmox for everything, there are terraform/opentofu providers for proxmox so beyond installing the hypervisor itself you can make all your infrastructure instantly rebuildable in a repeatable fashion. You say document everything, using techniques like infrastructure and configuration as code makes your setup self documenting. ---------------------------------------------------------------------- Diligent_Ad_9060 · 2023-11-20 22:15 Using native zfs encryption rather than LUKS containers as part of the zpools. ---------------------------------------------------------------------- Raithmir · 2023-11-20 22:15 But will an 8 way Xeon system with 2TB of RAM be enough to run a Minecraft server? Should I get two in case I want to run Plex as well? 😂 ---------------------------------------------------------------------- zfa · 2023-11-20 22:25 > I also know that I want to run Pihole Jeez, already planning your next steps to create future rework. Check out AdGuard Home, Technitium DNS, Blocky etc before committing to pi- hole. ---------------------------------------------------------------------- Jealy · 2023-11-20 22:28 I went from an i5 PC I liberated from work, to a Xeon server I liberated from work, and the server uses less power. ---------------------------------------------------------------------- Cylian91460 · 2023-11-20 22:35 Reinstall it, it's very easy when you run everything on arch. It can do it in like 1h and another hour for setting up everything. ---------------------------------------------------------------------- isawasahasa · 2023-11-20 22:40 I've used Windows, OpenMedia, Synology, Virtual Machines, Rancher, Truenas Scale, but Unraid is by far the best. For me here are the benefits; Hardware compatibility is awesome. It works well with my Dell Server and even picks up on thermal/bios status. It boots from a thumb drive so your 2 disks will be dedicated to your data. Simple to backup config to zip file. Software is awesome. It treats docker containers like apps so you just install it, config storage/networking, and it just works. Updates are simple. Update containers with new images with a single click. Bonus: It works well with NUT (UPS monitor) so my NAS shuts down automagically on power outage. The UPS is connected to a different host :D Good luck! ---------------------------------------------------------------------- lilolalu · 2023-11-20 22:56 Unpopular opinion: don't use any of the Truenas, unRAID, freenas, Openmediavault, proxmox stuff. Sooner or later you will run into something those special interest systems can't do and it will feel annoyingly limiting. Choose a widely used vanilla Linux distro, Debian or Ubuntu, install everything as docker container. ---------------------------------------------------------------------- Sinister_Crayon · 2023-11-20 23:06 It's a homelab. I start from scratch about every 2 years just because I can. The only consistent element is the data on the system... the operating system, applications and frameworks often get a workout. Having said all that, my current setup is working exceedingly well for my use case and I don't know that I'm going to change it en-masse any time soon. Individual apps will change though I'm sure. ---------------------------------------------------------------------- Sinister_Crayon · 2023-11-20 23:10 My Plex / Jellyfin server is independent from the rest of my lab (yes, it runs both). It's a Dell Optiplex 3060 Micro with an i5-8500T that mounts my media library from the main lab using CephFS (since I use Ceph on the backend) but it has also been an NFS mount from a ZFS array in the past. There's no reason you can't break off demanding services to appropriate platforms. As it stands the 3060 has more than enough transcoding guts in the iGPU that it works a treat. ---------------------------------------------------------------------- GoobyFRS · 2023-11-20 23:18 I moved to TrueNAS Scale from Core. My worst decision ever. Core was more stable, faster, and the jails actually worked as expected without some shit k3s cluster nonsense. ---------------------------------------------------------------------- PikachuMayhem · 2023-11-20 23:20 Arch isn't declarative, use NixOS and you can have everything done in minutes. ---------------------------------------------------------------------- AnomalyNexus · 2023-11-20 23:24 Not much - it's been a pretty organic learning journey. Very much a crawl > walk > run thing. Can't necessarily jump straight to the end. ---------------------------------------------------------------------- Heas_Heartfire · 2023-11-20 23:29 I got a 4TB drive thinking it would be plenty for plex. Oh boi was I wrong. Specially when shows started to come in. A 12TB drive is on the way and I'll leave the 4TB one for immich and nextcloud alone. I don't have any issues with ubuntu server and I like portainer so I don't think I'd set things up differently. I should organize and backup the docker compose files though. ---------------------------------------------------------------------- RaiseRuntimeError · 2023-11-20 23:33 For the most part im with you except i use TrueNAS as my NAS and to host a few select services that i want to sit close to my data like Plex. Everything else gets hosted on Ubuntu servers with Docker and docker-compose files. ---------------------------------------------------------------------- PotDucky · 2023-11-20 23:39 I'd agree, but not sure how Proxmox fits in with the others? ---------------------------------------------------------------------- die9991 · 2023-11-20 23:40 Honestly i think i just want kubernetes as a way to learn it rather than actually add it to my homelab. ---------------------------------------------------------------------- RaiseRuntimeError · 2023-11-20 23:40 lol, i just went from Core to Scale. I was having stability issues with my Core server where it would lock up every now and then and NFS shares and the web portal would become unresponsive. Upgrading to Scale has been ok so far except a obliterated all of my NFS and dataset permissions and still cant get things right. ---------------------------------------------------------------------- 0h_WTF · 2023-11-20 23:46 I shoulda bought more ram for my truenas box, and prolly shoulda done my file structure a lil better but other than that, no regrets. ---------------------------------------------------------------------- dazchad · 2023-11-20 23:56 And that’s completely fine. I mean more in the context of people honestly thinking they need or would substantially benefit from it. ---------------------------------------------------------------------- schizovivek · 2023-11-20 23:57 Genuinely curious. Do you mind explaining why you are advising against pihole? ---------------------------------------------------------------------- nosliw_rm · 2023-11-20 23:58 Favorite reply yet, this should be higher up ---------------------------------------------------------------------- notdoreen · 2023-11-21 00:00 I use Docker-compose for everything instead of Docker run ---------------------------------------------------------------------- AudioOmen · 2023-11-21 00:06 Don't listen to others. Just do what you want and have fun. ---------------------------------------------------------------------- Dev-N-Danger · 2023-11-21 00:08 I have a bunch of crap deployed between two servers and a bunch of mix match hard drives. If I could start over I’d plan much better and keep everything organized. I’d run one server for all my dockers / apps. Then the other server for storage / backup ---------------------------------------------------------------------- DanGarion · 2023-11-21 00:15 This! Debian with Docker is all I want. I've tried all the others out and it just seems to complicate things more. ---------------------------------------------------------------------- Dogeek · 2023-11-21 00:19 I started from scratch recently. If I were to start again, I'd do my ansible playbook first. I installed so much stuff on that server, I don't recall everything, and now I am having trouble writing the recipes. Otherwise, i'd keep it as is. I like my kubernetes setup. I'd probably use helm more though. I'm stubborn in wanting to do everything with kustomize because helm is a hellscape, but I think using it to install operators makes a lot of sense, especially since you can declaratively use it with ansible. I'd also set up some CI/CD and GitOps early on. If I have another server I'll have to think about how to go about storage. It's not a problem right now, but in the future it might be I'd set up borg backup straight away. I still need to do it right now, and I don't have the drive to write a kube operator to do so. I need to though, for my peace of mind. ---------------------------------------------------------------------- broken_cogwheel · 2023-11-21 00:20 I've made a lot of "interesting" decisions and some things were silly, some things worked out, some things I needed to fix. I've been self-hosting for a long time. Before commodity virtualization (let alone containers)... As technology improves, one must adapt and that often involves rebuilding, migrating, retiring software. My only regret is the data I've lost from insufficient backups. Everything else is part of the fun, part of the experience. ---------------------------------------------------------------------- DanGarion · 2023-11-21 00:30 Same here, but I take it one step further I do it all from the Stacks section of Portainer with my compose files. ---------------------------------------------------------------------- purepersistence · 2023-11-21 00:30 Wikijs ---------------------------------------------------------------------- alnyland · 2023-11-21 00:32 Can you clone the system and give/sell them the current setup? I'm kinda joking but also kinda serious ---------------------------------------------------------------------- Aurailious · 2023-11-21 00:41 I'm going to go against the grain and say use K8s from the beginning, or maybe more broadly gitops. I now use a mix of nuc and pi with Talos Linux and deploy with argocd. My entire stack is run from git repos. For me thats really easy and makes running my lab and selfhosted parts a lot less work. Though the learning experience going from vm management to docker, to compose, to K8s, to gitops has been valuable. ---------------------------------------------------------------------- nosliw_rm · 2023-11-21 00:45 well go on then, what are you running ---------------------------------------------------------------------- bamhm182 · 2023-11-21 00:47 You could always just stop? "You're welcome for the free ride while it lasted, I'm cutting my energy usage down by 87%" ---------------------------------------------------------------------- notdoreen · 2023-11-21 01:07 Can you elaborate on that? ---------------------------------------------------------------------- Tharunx · 2023-11-21 01:11 Debian 12 with docker is what i use. Very stable for the past 2 years ---------------------------------------------------------------------- Tharunx · 2023-11-21 01:13 Do you use immich or nextcloud memories(nextcloud app) for backing up your main photo library? ---------------------------------------------------------------------- zfa · 2023-11-21 01:30 It lacks basic features that should be standard and is topologically inelegant. Sure, it was great when it was the only game in town but times have changed. I often compare it to rocking ddwrt on your router - it's fine if you have it in place already and are used to it, but if you're coming in fresh there are far better options. ---------------------------------------------------------------------- SirLoopy007 · 2023-11-21 01:32 Well you just caused me to delay that upgrade for a few more months again! ---------------------------------------------------------------------- ThereIsNoTri · 2023-11-21 01:34 So to summarize, first dedicate a room then install a subfloor, halon system, and power upgrades? ---------------------------------------------------------------------- SirLoopy007 · 2023-11-21 01:34 The joke answer is, take up a different hobby and avoid computers and servers at all cost! Personally though, probably not overthink everything and just jump in and try things more often. ---------------------------------------------------------------------- bw1235 · 2023-11-21 02:16 I'm happy with OpenMediaVault for my purposes, but I probably would've taken the time to research and leverage ZFS instead of EXT4. ---------------------------------------------------------------------- Sinister_Crayon · 2023-11-21 02:17 Oh jeez... Well basically a full on corporate class cloud. I have three servers operating as my primary. They have a Ceph cluster on them with each node having 5x 8TB drives and each with a 960GB SSD for high performance storage. Connected with 10G for front and backend networking. Each Ceph node hosts a VM that takes part in a Docker swarm... These are my app nodes. Having the swarm in VMs provides good isolation. Running on each node plus two other bare metal machines in the swarm I have a MariaDB Galera cluster that creates my core database... Each node has a full copy of the databases and if one restarts it catches up. Occasionally I need to reseed the cluster from one node but that's easy. Apps... Quite a few. I have my WordPress website and then Nextcloud both hosted on the swarm. There's a Docker Mailserver instance that hosts mail for me and my kids... Though I use it more than they do. I have Photoprism set up for photo management right now but also have an Immich instance I might migrate to. Hmm... What else? Oh yeah all the backend services are proxied through an NGINX cluster with HAProxy running on my pfSense firewall as an SSL offload and to provide redundancy and load balancing. Then there's all the rest; Jellyfin/Plex, PiHole (2 of), some of the *arr's... many I'm forgetting right now. I have a play instance of ERPNext as well and other stuff I put up as learning tools on and off. Oh yeah, then there's an UNRaid box that is mostly running backups and a few choice Docker containers like YouTubeDL-Material and a good place to prototype. Oh and last but not least there's a Synology DS418 that gets backups from the Ceph cluster as well that's on the second floor of my house while my lab is in the basement. Almost all the hardware is self built or corporate salvage. My switching is some Dell and Mikrotik. I have Unifi wifi. Anything else? Oh hell if I know... I'm on my phone and sipping a glass of wine but feel free to ask 😅 ---------------------------------------------------------------------- ProgrammerPlus · 2023-11-21 03:23 Proxmox is just debian with few bells and whistles. You can even install Proxmox on existing debian install. ---------------------------------------------------------------------- eve-collins · 2023-11-21 04:13 Thanks for this. I’ve been using Notion and I quite like it still, it selfhosted is an addiction so I’ll try ditching it in favor of BookStack. ---------------------------------------------------------------------- halpoins · 2023-11-21 04:13 I used to run Confluence self-hosted, but it was a pain in the ass and of course when your server is down, docs are down too. So I moved it all to Atlassian cloud. It’s free for what I need it for, just a homelabber keeping notes on everything. Decent iOS app. Doesn’t pester me too much to upgrade to a paid plan. When my reverse proxy is working I access it at confluence.mydomain.com but when it’s not, Atlassian provides you a decent alternative of yourname.atlassian.net/wiki. ---------------------------------------------------------------------- a_40oz_of_Mickeys · 2023-11-21 04:43 I started with OMV and a NUC connected to a 2 drive terramaster DAS. This was a mistake. Ended up building a box with room for 12 drives running debian with snapraid and mergerfs. Using duplicati to keep all my yaml and config folders backed up ---------------------------------------------------------------------- highedutechsup · 2023-11-21 04:44 Never buy into a platform, they tend to waste a lot of my time on changing things to upgrade versions that have no point except to change the whole platform, and the things you like to use get deprecated. Your vision is not the maintainers vision. ---------------------------------------------------------------------- Engineer_on_skis · 2023-11-21 04:45 Are there any quirks to getting your machine(s) up to the point you can install docker? Need to tweak GPU drives or jump through hoops to get your network card set up properly? What about the rest of your network? When my wife had our second child, my mom flew into town to help us with the newborn and the toddler. Before we came home from the hospital, she dusted our house. The DHCP for my network is handled by pihole running on a pi. That pi happened to have an Ethernet cable with a broken latch (that vale had since been retired). As my mom dusted, the cable came loose. The VPN I was isn't at the hospital went down. When 8 got home editing was down. It took me over 30 min to figure out what was wrong, since I was in dad mode, not IT mode, and was just panicking. If I had documentation reminding me that pihole was the key to everything, I might have started there. Do I have a good link to the switch? Nope; click. Hey! Look all my services are available again, and the internet works. If I had been the one to dust, I would've (and had many times) rendered to make she that cable was fully seated. ---------------------------------------------------------------------- New_d_pics · 2023-11-21 05:15 Create a new stack in Portainer, give it a name (I just use the app name), dump in your compose file, then just click to deploy. ---------------------------------------------------------------------- discoshanktank · 2023-11-21 05:42 Do you have a guide you fomllowed to get started? I’d love to try and go in that same direction ---------------------------------------------------------------------- lilolalu · 2023-11-21 06:01 That's the same for OpenMediaVault, still, you can easily break OMV by doing something OMV doesn't like on the underlying OS. ---------------------------------------------------------------------- ZeeroMX · 2023-11-21 06:04 I also migrated all my docker run containers to docker compose. I was tired of looking at history for the exact docker run command for anything. ---------------------------------------------------------------------- aamfk · 2023-11-21 06:19 I'd get my active directory and local dns situated before having a bunch of virtual machines setup ---------------------------------------------------------------------- isleepbad · 2023-11-21 06:49 I usually judge it by "if my server crapped out today, how long would it take me to get it back up to its current state?". If there is even one manual step I document it. Even if it's where to find the backup. ---------------------------------------------------------------------- isleepbad · 2023-11-21 06:53 I use Trilium ---------------------------------------------------------------------- servergeek82 · 2023-11-21 06:59 Started from a mini computer with 4gb ram and a USB nas. Barely supported 1 stream. Now I have a rack mount server. Overjoyed when I actually use 40gb of the 256gb of ram. Deployment is handled via gitea runners. Mirrored on a separate laptop in the fork of gitea. I get a notification when it completes. Do differently? .... I do have my new server I just got with 384gb of ram. Just waiting for me to find out. ---------------------------------------------------------------------- TheCaptain53 · 2023-11-21 07:01 Only change I'd make is to run Debian on my server over Ubuntu. I'd still run everything in Docker Compose rather than something else, or consider the use of something like k3s. The server setup to get it ready for hosting data was a bit complicated, so I liked someone's suggestion of putting everything in an Ansible playbook. I'd consider doing that. ---------------------------------------------------------------------- Cylian91460 · 2023-11-21 07:03 Yes and no, you need to redo the nix config file so it's probably slower. But yeah NixOS is would be better. ---------------------------------------------------------------------- lilolalu · 2023-11-21 07:12 I really enjoy the freedom of a vanilla linux distro. When i am moving machines, i dont even care what type if linux flavour the new host is running, I install docker & zfs, copy over my container config and persistent data, import my zfs pools, start containers and i am done. The task moving machines, which could take a day or two has become a matter of 30 minutes. ---------------------------------------------------------------------- Aurailious · 2023-11-21 07:23 With what I have now? I don't think I followed any particular guide. Most of what I did was read docs. I just saw a few things that people have done here and on /r/homelab, things mentioned like Talos and ArgoCD, and then dove right into it. https://www.talos.dev/v1.5/introduction/getting-started/ https://argo-cd.readthedocs.io/en/stable/getting_started/ https://helm.sh/docs/intro/quickstart/ The big thing for me moving from docker compose to k8 was how simple Talos was to use and get going. It handles a lot of bootstrapping and managing k8s itself. Then it became a challenge of trying to convert using compose to using helm, which is what I use through ArgoCD. Though its a lot harder to find helm charts then docker compose examples for most apps. Almost everything I have is through Artifact Hub, though I keep meaning to create my own templates. https://artifacthub.io/ There is this repo that I followed as well. I don't use everything exactly the same but a rough "sanity check" kind of way. I'm pretty sure onedr0p has also posted to /r/homelab. I would recommend using this as a way to understand what you might need. Everything a homelab/selfhosted setup running on k8s is here. Not really a guide, but lays it all out. I don't use discord but I think there is a good discord there as well if you have questions about it. https://github.com/onedr0p/home-ops ---------------------------------------------------------------------- The0neWhoKnock5 · 2023-11-21 07:42 Basically whatever you'd write in a docker-compose file, you can do the same in the Stacks section of Portainer. One thing to note though, use the short form of bind mount, otherwise when you're updating/pulling the new Container (in the Containers section), Portainer will complain that the mount is in use, which is only solvable by taking the stack down and restarting it. ---------------------------------------------------------------------- tobz619 · 2023-11-21 07:45 I aspire to be you :D slowly, but surely ---------------------------------------------------------------------- xerker · 2023-11-21 07:45 Probably wouldn't use Apache2. I want to use caddy and reverse proxy from there but I can't bring myself to do it. ---------------------------------------------------------------------- notdoreen · 2023-11-21 07:49 Awesome. Thanks for sharing. ---------------------------------------------------------------------- schizovivek · 2023-11-21 07:54 Actually that's what i was asking; what does it lack? I have pihole loaded and have read a few times folks suggest adguard so was just curious. I guess i'll look into the 3 apps you mentioned to see what they have to offer ---------------------------------------------------------------------- 7WuCF77S95mfQFvfwk7F · 2023-11-21 08:02 On the software side not much. On the hardware side I had a few fuck ups. Got a lenovo 720q 9400-t for free from work and used it as a single server. Wanted to upgrade storage and got a bigger motherboard and dropped a hardware raid controller in it. Soon realized hardware raid is shit and I need more PCIe slots and bought a ATX motherboard with enough lanes. Can't want for the moment I need more PCIe slots/lanes and buy an EPYC. -.-" ---------------------------------------------------------------------- FallenFromTheLadder · 2023-11-21 08:02 >Never offer Jellyfin to friends / family Tell them the system is getting too expensive to run and that you unfortunately need to shut it down unless they help you with the expenses. ---------------------------------------------------------------------- Heas_Heartfire · 2023-11-21 08:13 I use immich for that. Haven't tried nextcloud memories but I have no need to do so for now. ---------------------------------------------------------------------- Tharunx · 2023-11-21 08:18 Thanks for the info ---------------------------------------------------------------------- zfa · 2023-11-21 08:18 Well generally Pi-hole is just kind of hamstrung by its origins which were a few scripts and bootstrap GUI wrapped around dnsmasq. It's evolved a little (dnsmasq was forked, it even got an https support this year, lol) but it's topologically obtuse and tends to spray various files all over the os. This is why nearly everyone devotes a pi to it, or runs it in a docker container. To get modern features you generally have to install extra software like unbound, or stub resolvers, cloudflared etc. It is simply hamstrung by its not- insignificant technical debt Comparing with AdGuard Home, for example... AGH is a single self- updating binary available for pretty much every platform you can think of with a single sensible config file. It has an https interface, multi-user support, one click toggles of common blocks, supports more elegant, smaller blocklists. Has built-in support of DoH, DoT, DoQ, dnscrypt. Even acts as a DoH server etc. Can activate real-time blocking of emergent threats through an API if you want. Its acolytes still love it and will defend its poor featurese by saying you can tack on unbound, or cloudflared or ... or ... or ... to get parity with newer network-wide adblocking solns but the fact remains that you just wouldn't need to do that if you just started off with something else. It does have some upsides - cool Star Trek theme, for one. ---------------------------------------------------------------------- Frozen_Gecko · 2023-11-21 08:38 I would document everything I could if I started from scratch. I really regret not documenting from the beginning. Ex post documenting is just not as reliable. ---------------------------------------------------------------------- mcr1974 · 2023-11-21 09:04 why not kubernetes at this scale. ---------------------------------------------------------------------- mcr1974 · 2023-11-21 09:15 why talos vs k3s etc? ---------------------------------------------------------------------- dxrth · 2023-11-21 09:27 The next issue is when they do offer to help with the expenses, but you still just don't want to run those services lol ---------------------------------------------------------------------- schizovivek · 2023-11-21 09:28 Thanks for the detailed explanation. Pihole might just be enough for me but its good to learn whats others offer. Will definitely be checking out the apps you mentioned. ---------------------------------------------------------------------- FallenFromTheLadder · 2023-11-21 09:28 Just tell them a price that is higher than of a streaming service. They'll prefer to pay it, for sure. ---------------------------------------------------------------------- Aurailious · 2023-11-21 10:06 One of the big motivators I had was that I wanted to try out using some of these kinds of container/k8 specific OSs. I was originally going to use openSUSE's MicroOS with k3, but was looking at alternatives to compare that too. Fedora CoreOS with OKD was something I was going to look at too as I've generally preferred using Fedora/CentOS over the years. But with Redhat/IBM recently I was looking elsewhere. Talos just seemed a bit interesting as part of that group and I gave it a go to see what it was about. I ended up really liking how easy it was to configure and provision with a yaml file. It takes just 3 commands to go from booting the iso to having a functioning k8s cluster, though at that point just single node. Adding another node is simply booting and applying config with a single command. I swapped my nucs and pis over in an afternoon. It fit really well into the idea I wanted with controlling everything through code. I don't ssh and configure each different part of linux and k8 manually. Instead its all there in two files (separate control and worker config) that exists in my git repo. I can make a change, review my file, and when ready I can just *talosctl -n some-node -f config.yaml* and everything after is automated and I don't need to think about or do work to accomplish. Though I'm sure I can find a way to even remove running that command. I never looked that hard at MicroOS and K3 to know if it can do the same. I just know that Talos ended up working really well after I started using it. ---------------------------------------------------------------------- s3r3ng · 2023-11-21 10:11 What is wrong with a Synology NAS or TrueNAS? Plenty of apps for photos on or using such. I am falling in love with NextCloud for many things reasonably integrated together. Can definitely self-host that and share it or parts of it with others and collaborate with them if you like. ---------------------------------------------------------------------- mcr1974 · 2023-11-21 10:35 could you not achieve a very similar git-based setup throwing ansible into the picture? do you use ansible at all? ---------------------------------------------------------------------- Pingoui01s · 2023-11-21 10:38 I made the same conclusion when I tested all the NAS distros for a future NAS. I run Debian 12 with ZFS now and i'm happy. ---------------------------------------------------------------------- crusader-kenned · 2023-11-21 11:12 Do all my network configuration with ansible, terraform or something similar. And I wouldn’t document anything but instead automate everything. ---------------------------------------------------------------------- wub_wub · 2023-11-21 11:13 Take it one step further and pull the docker compose file from a git repo and use webhooks to notify portainer whenever a change is made, this allows you to easily add new containers, to have history of changes, and to quickly revert/remove things without ever even needing to open portainer. ---------------------------------------------------------------------- ArtificeAdam · 2023-11-21 11:18 Seconding WikiJS. Started with Dokuwiki, moved over to this because pretty. ---------------------------------------------------------------------- lemniskegg · 2023-11-21 11:21 Do you think running single-node k3s complicates things? I personally prefer that than plain docker-compose, especially installing things with helm and integrating it to my rancher server ---------------------------------------------------------------------- Heuristics · 2023-11-21 11:48 Currently doing that. Migrating from a raspberry pi 4. Here is the plan: Use the ASROCK N100DC-ITX motherboard+CPU for low power usage but enough power for a snappy web server. Add 32GB ram and 2tb m.2 ssd as well as 4+7tb external usb storage. Install windows 11 as root OS and use windows subsystem for linux with ubuntu for any webservices, install those using docker inside cosmos. This way the device can be used to play simple 2d games via steam yet be used for plex and web services such as bookstack and kavita. ---------------------------------------------------------------------- kelthuzad12 · 2023-11-21 12:25 Platform like what? ---------------------------------------------------------------------- ProgrammerPlus · 2023-11-21 12:30 OMV is like a little hobby project while Proxmox is a real enterprise grade virtualization platform. You can't compare both. I say anyone starting out should use Proxmox as LXCs give insane flexibility. Very easy to backup, etc. In fact what you said about breaking OS is true for plain vanilla OS as with virtuaization platforms, you dont mess with OS and everything will be done on VM or LXC. ---------------------------------------------------------------------- Aurailious · 2023-11-21 12:33 I have not used ansible. For what I described I think you could get a similar experience. But I did not need to learn anything to understand how to configure Talos. For my initial interaction with it I had gone from opening their docs to running a cluster in hours. That impression of low work and effort involved is a key reason of why I adopted it. Introducing ansible would require maintaining the ansible scripts and keeping a knowledge of ansible to troubleshoot problems. I have avoided using ansible in general since I don't particularly favor how it works. I would not want to use it in a gitops workflow, but this is definitely personal preference. I would like to add that what I described is Talos at its most basic. Ideally its possible to set it up to not need to run any commands at all. Plus its API opens a lot of possibilities too. I have a lot of new things that I can do and learn while not needing to worry too much about stuff I don't want to have to deal with anymore. ---------------------------------------------------------------------- lilolalu · 2023-11-21 12:34 If you take care of this yourself, it doesn't matter if the host is CentOS, Arch Linux or Manjaro. How long would it take you to move a complex Proxmox container setup to a non-Proxmox machine? ---------------------------------------------------------------------- mcr1974 · 2023-11-21 13:14 thank this is defo good insight. good luck with your exploration. would be good to get am update from you at one point / if anything changes. Will try and get a talos cluster running. ---------------------------------------------------------------------- KoppleForce · 2023-11-21 13:23 Can you post an example of some of your documentation? ---------------------------------------------------------------------- Sinister_Crayon · 2023-11-21 13:45 Well why not Docker Swarm? LOL Seriously, it mostly comes down to the fact that Docker was what I used first. I use Docker on a few other systems as well; both my PiHoles run it in a Docker container mostly just because I can and for simple isolation. I also run containers on my main PC for a few different things too, and unRAID uses Docker. As a result, having the same command set for all of my containerization just simplifies my life and though switching to Kubernetes wouldn't be terribly difficult I haven't yet found a really compelling reason to switch. The Docker Swarm is quite effective, "resource cheap" and easy to operate. I have Portainer deployed on it as well in order to have a nice GUI for managing the swarm and generally as I said it just works and gets out of my way. Also worth noting that Ceph as I have it deployed (cephadm) ALSO uses Docker for containerizing the various components. As a result, having that same command set again used to operate the components of Ceph adds even more consistency to my operations. Not that I usually need to do much with Docker on Ceph... mostly just watching logs, restarting a failed daemon and the like. That's not to say I won't switch. I might play with it when I have some free time by spinning up some more VM's and seeing if I find a compelling reason to shift over... but again it's not been a priority. I'll be honest that in the last couple of years my homelab has become less about being an infrastructure lab and more about being a place I can easily spin up and test or learn new applications or frameworks. I'm pretty happy with the way things function in that regard right now. ---------------------------------------------------------------------- ProgrammerPlus · 2023-11-21 13:45 You run docker in Proxmox LXC. If you want to move the app to non Proxmox machine, its just copy paste to another docker host. ---------------------------------------------------------------------- lilolalu · 2023-11-21 13:55 Eh cut and paste persistence storage to another host? I don't know. ---------------------------------------------------------------------- Sinister_Crayon · 2023-11-21 14:24 LOL... absolutely fair. It has taken me many years and MANY iterations to get to where I am today. I've always had a homelab and have been in the tech industry my entire career, so getting hardware was never really an issue. My last big refresh was switching from a monolithic ZFS array with a couple of VM hosts to the Ceph cluster I have now; two of the three nodes are self-built Supermicro mini-itx boards with AMD Epyc 3201 CPU's in Jonsbo N1 cases... the boards I purchased with the money I got after I was laid off from Dell and the cases I bought last year specifically for the big refresh. The third node is a corporate salvage Dell T440 (company was shutting down so I scored this system for pennies on the dollar) It's been quite effective. I started with 4TB drives (the old ZFS array) and moved to 8TB drives during the last 6 months just finishing that migration last month. I buy drives from eBay, usually enterprise- grade SATA drives that have a lot of hours on them but my philosophy has always been that drives primarily suffer infant mortality or basically live forever... so these things even with a few years of runtime might just be "broken in". Generally also these drives have a very low power-on count meaning they haven't been stressed all that much by power on and off events... my last set of 8TB drives actually ended up having one that had 4 years of run time in SMART and a power on count of 6 which amused me. This lab didn't happen overnight either. It has grown as my needs and demands have grown and while in total I have spent quite a bit on it, I haven't spent much compared to a lot of other hobbies... including some of my own LOL ---------------------------------------------------------------------- QuestioningMaeve · 2023-11-21 14:29 Documentation. Document how my drives are set up (like really... I don't remember how they are configured XD I only know I haven't ran out of space yet so everything is going the correct mount.) Keep a propper list of what process uses which port. Use containers from at the start. That's all I can think of atm. ---------------------------------------------------------------------- droans · 2023-11-21 14:31 Yeah I don't think I would have used Ubuntu if I'm starting from scratch. I've been planning on switching OS's for a while but I've been rather lazy about it. I've got my root and home on separate partitions so it should be as easy as it gets, but it'll still be work. If you're using Docker, another suggestion would probably be to put `/var/lib/docker` on a different partition. I followed some older advice which suggested a smaller root partition but I didn't account for that damned directory to take up so much room. I tried to increase my root partition size by shrinking home, but my home partition is XFS and it turns out that can't be shrunk. I set my root partition to be 200GB and it's already at 74% full. I tried to add a LocalAI container yesterday and it wouldn't work because it ate up the rest of my storage. ---------------------------------------------------------------------- Ursa_Solaris · 2023-11-21 14:49 > You say document everything, using techniques like infrastructure and configuration as code makes your setup self documenting. This documents the how and what, but not the why, which is often the most important thing to document. Careful observation can always determine the how and what, but why requires a lot larger picture and is much harder to piece together after the fact. Declarative code definitely makes the job easier, we all love our yaml, but you still need to add more on top to finish the job. ---------------------------------------------------------------------- ProgrammerPlus · 2023-11-21 15:31 At this point you are debating just for the sake of it. Using Virtualization to keep OS clean and services/apps separate is well established pattern at this point and there is a reason why Proxmox is so popular in this sub. Most of us used bare bones debian/ubuntu before moving to Proxmox or other virtualization platforms. I once messed up networking on my debian host when tweaking docker networks and lost connectivity to host. Such issues can be avoided using Proxmox. ---------------------------------------------------------------------- lilolalu · 2023-11-21 15:33 No I am not debating for the sake of it. A GUI heavy does-it-all system (yes, Proxmox falls under that category as well) usually has its own organizational logic which usually differs significantly from vanilla linux. Moving that logic to somewhere else takes a significant amount of work. ---------------------------------------------------------------------- tenekev · 2023-11-21 15:46 Bookstack is good but if you bork your setup bookstack goes down with it. I use Obsidian now. Incredibly customizable and good. It's a local markdown editor but you can sync the vaults between devices with something like Syncthing. ---------------------------------------------------------------------- MiteeThoR · 2023-11-21 16:20 I did start over recently - Dell R730 salvaged from work, 64 cores, 256GB RAM Took the guts and moved it to a Machinist X99 motherboard in a Rosewill server case so I could put in silent fans and have room for 15 drives. Proxmox hypervisor booting on 2x 2TB NVME drives. Bought 6 x 16TB drives and a HBA to run them, then did HBA passthru to a virtual machine running Truenas which mounts everything as a samba share Multiple other Ubuntu VM’s running docker compose. One VM runs utility / ARRS, 2nd VM runs Plex so that my media playback isn’t affected by utility Also have LXC running Adguard Home for DNS based adblock/malware protection for the entire house. Also have a Raspberry PI as a 2nd DNS server. Several Windows VM’s used for work (I connect to customer environments so I spin up a separate VM for each customer to keep their environments isolated. ---------------------------------------------------------------------- dazchad · 2023-11-21 16:38 It has been a while since I played with k3s, but I find docker compose pretty simple to use without extra dependencies. ---------------------------------------------------------------------- ProgrammerPlus · 2023-11-21 16:46 Thats the point you are missing. You dont touch Virtualization host after installing. You create a VM or LXC of OS your choice and do whatever you want to on that. ---------------------------------------------------------------------- kiwimonk · 2023-11-21 17:30 Definitely this. There's actually a cool new docker interface that the uptime-kuma guy is working on. You paste in your docker run command and it creates a compose file. https://github.com/louislam/dockge https://youtu.be/E805XcbTzgY?si=r6uFI2pvbJg1cXBe ---------------------------------------------------------------------- tech2but1 · 2023-11-21 17:58 My docker storage is an iSCSI drive on the NAS so I pre-thought of that! ---------------------------------------------------------------------- tech2but1 · 2023-11-21 18:56 I made some notes on ```dpkg``` when I was troubleshooting something. Found bits here and there but added them all on one page for future reference, save an hour of Googling again! First of all you should check if this package is correctly installed in your system and being listed by ```dpkg``` tool: ```dpkg -l '*urserver*'``` It should have an option ```ii``` in the first column of the output - that means 'installed ok installed'. If you'd like to remove the package itself (without the configuration files), you'll have to run: ```dpkg -r urserver``` If you'd like to delete (purge) the package completely (with configuration files), you'll have to run: ```dpkg -P urserver``` You may check if the package has been removed successfully - simply run again: ```dpkg -l urserver``` If the package has been removed without configuration files, you'll see the ```rc``` status near the package name, otherwise, if you have purged the package completely, the output will be empty. And then when I installed openHAB I had a few things that needed tweaking so just documented what I did and why. Official: https://www.openhab.org/docs/installation/armbian.html If that doesn't work: https://www.themakersphere.com/install-openhab- le-potato-home-automation/ Add user to openhab group ```sudo usermod -a -G openhab username``` Give user write permissions on ```/etc/openhab/html``` for uploading files ```setfacl -R -m u:username:rwx /etc/openhab/html``` (```sudo apt install acl``` if command not found) Default Armbian on Le Potato installed Java 11. 17 is needed. Check versions ```java -version``` ```javac -version``` ```sudo apt install openjdk-17-jre openjdk-17-jdk``` Then switch version ```sudo update-alternatives –config java``` Log in/set admin, not sure what is set here when restored so do minimal setup at least. Check addons, if duplicates show delete ```.kar``` file in addons folder I also make use of interwiki links so the above for instance has addons linked to the openHAB addons page that goes into detail about installing them as there's a couple of different ways. Nothing too complex, all in simple text, literally just notes. Some have screenshots, some are copied from guides on the web which have a habit of disappearing. ---------------------------------------------------------------------- uoxou_xoxo · 2023-11-21 20:44 My best decision ever: use NixOS from start I don't have to deal with documents,proxmox,kurbenetes using NixOS. I regret nothing. ---------------------------------------------------------------------- ManSpeaksInMic · 2023-11-22 11:45 My documentation is disaster recovery. Over the years, especially back when I had less experience / less sophisticated setups, I completely lost my home server twice. Usually through my own damn fault. 😛 Rebuilding my server from effectively scratch is painful; especially remembering or rediscovering the gotchas to implement when things don't go fully smooth out of the box. (Pihole in docker, e.g.; or when services need to share docker networks.) The document is here in case things go catastrophically wrong and I have to start over. Nothing in there is rocket science, but with the doc at hand I could rebuild the whole machine in a few hours. (Modulo how long it takes to condense the backup back out of the cloud.) Of course people may argue I should just put it all in a script, or ansible, or devops flavour of choice. But my machine doesn't break often enough to do that. 😛 ---------------------------------------------------------------------- ManSpeaksInMic · 2023-11-22 13:45 This is the way. Don't forget the EMC farady cage, multi-lock steel fire door and encrypted key card access.