[HN Gopher] My self-hosting infrastructure, fully automated
___________________________________________________________________
My self-hosting infrastructure, fully automated
Author : rmbryan
Score : 638 points
Date : 2022-01-21 22:52 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| walrus01 wrote:
| my main objection to this is choosing host machines that don't
| meet the criteria of anything like serious server-grade hardware.
|
| if this is for a home lab where any one of the services run on it
| are not actually going to affect you if it goes belly up? or the
| whole host machine? sure, okay, but that's self hosting a home
| lab, not self-hosting actual infrastructure...
|
| clearly the hardware shown in the image is meant to be small, not
| noisy, and not take up a lot of space, and operate in somebody's
| house.
|
| but the people I know who _seriously_ self host all their stuff
| have it on something like a few old Dell R620 1U servers with 1+1
| dual hot swap power supplies, RAID-1 for the operating system
| boot, RAID-1 or RAID-6 (or something like mdadm raid6) for
| storage drives (again all hot swap capable), etc.
|
| note that I am not talking about a lot of money here, you can buy
| the hardware I described in the preceding paragraph for $300 on
| eBay and then add your own choice of SSDs.
|
| and not in a house, but in something like a small to medium sized
| ISP's colocation environment, with UPS, generator backed power,
| etc. also attached to network equipment and a DIA circuit that's
| considerably more serious than a tp-link unmanaged switch
| attached to a residential internet service.
| Spooky23 wrote:
| All overkill. If you want something more reliable, get some
| late model HP or Dell thin clients and put Debian on them.
| Usually you can pick them up for <$75.
|
| I setup something like this to control some farm equipment when
| I was in college 20 years ago. The farmers' son called me a few
| months ago because they were having an issue with the
| equipment. Turned out oily dust coated the thing and got inside
| - I had him pop the device open (old Wyse box running Slackware
| iirc) clean up the goop and hit it with rubbing alcohol. He
| dried it and everything is working again.
| surfsvammel wrote:
| How much energy does this consume?
|
| I have done some home lab going on over the years, and find that
| trying to do more on less W has been the most fun.
| khuedoan wrote:
| (Author here) Yep that's my goal too, hence the small form
| factor PCs. It costs me around 2-5$/month depending on how much
| I play with it.
| quocanh wrote:
| I love this!
|
| Do you experiment with your tech stack? Swapping things in and
| out to see which apps are best?
| khuedoan wrote:
| (Repo owner here) Yes, you can see the visualized history of
| that here
| https://www.reddit.com/r/homelab/comments/s9bfht/the_evoluti...
|
| I think the OP discovered my homelab through that post.
| erulabs wrote:
| Awesome! hajimari is neat, haven't seen that before. I'm building
| a startup that's trying to put a good chunk of this in a box for
| folks to buy and plug-n-play self-hosting (as much as that is
| possible). Since you're running k3s already, you might checkout
| our templates at https://kubesail.com/templates for more home-
| hosting apps :)
| khuedoan wrote:
| (Repo owner here) The templates look cool, are those Helm
| charts underneath?
|
| Also I believe the tag filter is malfunctioning:
| https://kubesail.com/templates-by-tag/Media
| erulabs wrote:
| Ack, will look at that bug! No, the templates are currently
| plain YAML with a handle-bars style variable replacement (and
| a couple other neat tricks). We do support Helm charts tho, I
| just haven't had time to add them to the "Templates" view.
| There are loads of good Helm charts here:
| https://kubesail.com/helm/k8s-at-home
| nyellin wrote:
| To everyone saying that Kubernetes is unnecessary, try
| implementing autoscaling, service discovery, secrets management,
| and autohealing in a vendor independent way without it.
|
| Of course none of that is necessary for a self hosted home lab,
| but neither is gitops.
|
| This is a very nice example of how to set stuff up properly.
|
| OP, I would love to see Robusta (https://robusta.dev) as part of
| this too. It's definitely in line with your vision of automating
| everything, as it let's you automate the response to alerts and
| other events in your cluster. (Disclaimer: I'm one of the
| maintainers)
| dijit wrote:
| Oh. I've actually done that. It's not as hard or complicated as
| you claim.
|
| This was before kubernetes, arguably learning kubernetes is
| harder than implementing auto discovery and auto healing
| because you have to learn a lot about cluster semantics, the
| various network providers, service accounts, role based access
| control, integrations with secrets manager or the very
| disparate implementations of custom operators which may operate
| in your environment.
|
| Kubernetes is a platform, I don't doubt it solves a lot of
| problems. But it's a complicated wee beastie, and I say that as
| a person who did it the easy way and used GKE on Google Cloud
| nyellin wrote:
| I don't know how you implemented it, but the big innovation
| of kubernetes was doing this with declarative inputs and
| using control loops for reconciliation instead of edge
| triggered changes.
|
| It might not matter on a small scale, but on a large scale
| that's what makes it robust. It's also not trivial to
| implement a distributed version of this in house at scale. At
| the very least you need something like etcd at the core
| dijit wrote:
| The way we did it is that we registered to a centralised
| "name" service which was an in memory database.
|
| Failure to register to the name service leads to
| registration towards the next name service in the list, we
| ran three- so there were two spares, the library would then
| keep trying to register to the first name service, this
| would be our reconciliation loop.
|
| When you request a service you request "traits" through the
| name service.
|
| Not "trivial" but even though it was implemented in C++ on
| windows it was much less complicated than kubernetes
| discovery systems, which there are many ways to integrate
| (env-var, dns, api) and some should or must be disabled for
| security reasons (eg. The environment variables that
| display all services) or can change depending on RBAC and
| service account.
|
| We deployed about 300,000 or so services with this
| infrastructure and it's been running in production since
| 2015.
|
| (Original product release was supposed to be 2013 but due
| to the client changes it was delayed, not due to the
| backend. This is Video Games unfortunately).
|
| I'm not like, "hard" on these opinions, but people talk
| about kubernetes as if it's the only way of solving these
| problems, when it can be the case that you have to learn a
| lot about kubernetes before you can actually do what you
| need to do; if you already understand distributed systems
| then it's a bigger effort to understand kubernetes than
| would be to implement a distributed system that solves your
| needs from scratch
| nyellin wrote:
| Very cool and thanks for sharing.
| [deleted]
| akkartik wrote:
| The question I have whenever I see something like this is what
| the upgrade process looks like. How much time do you spend per
| month upgrading dependencies, how many mailing lists you're
| subscribed to, etc. When the log4j thing blew up, how long did it
| take you to gain confidence that you were no longer impacted?
| khuedoan wrote:
| (Repo onwer here) The upgrade process can mostly be automated
| (for example using Dependabot). I haven't configured it yet,
| but that's on my TODO list.
| akkartik wrote:
| <3
|
| You're fighting the good fight, keep it up.
| akkartik wrote:
| What languages does it depend on so far? Do you try to
| minimize the number of language eco-systems you rely on?
| khuedoan wrote:
| Yes, I aim to keep the amount of languages and tools I use
| to a minimum:
|
| - For simple scripts, use POSIX sh
|
| - For more complex scripts, use Python
|
| - For the same sort of task, use only one tool (e.g., only
| use Ansible for configuration management, don't mix in
| Puppet or Chef)
| akkartik wrote:
| Awesome!
|
| I was thinking more about the user-facing apps you use.
| Scanning https://github.com/khuedoan/homelab, Gitea is
| Go, Grafana is Javascript (approximately), Element is
| Javascript, Dendrite is Go, Vault is Go. K8s itself is
| Go. Are you planning to run your own email server?
|
| Now I find myself wondering what app choices exist if one
| were to try to constrain oneself entirely to operating in
| the Go eco-system, with Javascript only for web clients.
| That would be quite parsimonious.
| khuedoan wrote:
| Ah, for user-facing apps I prefer Go or Rust for their
| performance, although it's not a hard requirement.
| zenlot wrote:
| Doesn't really matter, as long as it's fun. Some people play
| games, so people enjoy re-configuring their setup (I do enjoy
| it too). It's not a business / prod env setting.
| akkartik wrote:
| Yeah, that's totally fine. I ask in case someone has thought
| about it, so I can learn from them.
|
| Doing just the fun parts of self-hosting will not get you to
| _infrastructure_.
| throwaway894345 wrote:
| Probably no worse than the alternatives. Just about everything
| is going to be easy to upgrade except possibly the Kubernetes
| masters (and even that isn't so bad if you can spare a bit of
| downtime).
| akkartik wrote:
| For sure no worse than the alternatives. I also don't care
| about downtime. But I'm looking for something that minimizes
| the overheads over _not self-hosting_.
|
| I'd love to prove Moxie Marlinspike wrong that "people don't
| want to run their own servers, and never will."
| (https://moxie.org/2022/01/07/web3-first-impressions.html)
| This is the key bottleneck in getting people to run their own
| servers.
| throwaway894345 wrote:
| For sure. There's not much of an economical argument for
| self-hosting--if you're doing it, it's almost implied that
| you're doing it for fun. Although having run a small (but
| really scaleable c/o k8s) homelab for <$2/month, I'm not
| sure I'd save much effort versus using some PaaS or cloud
| provider now that I know what I'm doing. Like I'd need a
| real load balancer and a few other things so we're talking
| < ~$100/month to productionize.
| akkartik wrote:
| I still believe :) I'm looking not for an economic
| argument but for a _strategic_ one. I think[1] a self-
| hosted setup with minimal dependencies can be more
| resilient than a conventional one, whether with a vendor
| or self-hosted.
|
| https://sandstorm.io got a lot right. I wish they'd paid
| more attention to upgrade burdens.
|
| [1] https://github.com/akkartik/mu
| throwaway894345 wrote:
| Fine, take my upvotes. :)
|
| In all seriousness, that's an interesting perspective
| that I hadn't considered.
| cyfex wrote:
| I always found the hurdle with self-hosting to be maintaining,
| not the initial setup. Things like upgrading, in order to keep
| getting security fixes, and verifying everything works after the
| upgrade, are what has taken the most effort and time in the past
| for me.
|
| This looks like a great setup by the author, but difficult to
| maintain in the long run without significant time investment.
| input_sh wrote:
| Docker and docker-compose make maintaining stuff dead simple to
| me. In like 95% of the cases it's just a matter of changing the
| version in a YAML file and running one command, and in the
| remaining 5% of the time I'm not concerned at all with some
| downtime because I'm (mostly) the only user.
| goodpoint wrote:
| Containers are not good for security updates.
|
| docker also adds a big attack surface.
| BrandoElFollito wrote:
| This. I used to host directly on my server and discovered
| docker a few years ago.
|
| It was a relief. Coupled with watchtower the updates are
| automatic except for the two services I really rely upon.
|
| I used to be on Ubuntu for ages and moved to Arch a few days
| ago - my server only runs docker today.
|
| I still want to keep access to a shell do it is Arch and not
| Rancher or something (I did not research much the bare metal
| hypervisors).
|
| My maintenance is minutes every month if everything works
| fine, up to a max of an hour when Home Assistant broke things
| once two years ago.
|
| DRP from bare metal is an hour.
|
| Adding a service is a few minutes.
| toshk wrote:
| What do you use for serving docker-compose?
| g_p wrote:
| Docker-compose doesn't need "served" per-se - it lets you
| run some containers, and handles some lightweight
| orchestration around them, so that you have groups of
| containers for a service (i.e. a Wordpress service
| compromises a mariadb container and a Wordpress container).
| The compose file handles port bindings/mappings.
|
| If you then want to put a reverse proxy in front of the
| docker containers (you almost invariably will), then you
| can look at different options like caddy, Traefik, nginx
| etc. I, for one, like to be old-fashioned and have my
| docker containers' ports bound to localhost, then manually
| maintain my own "outside of docker" instance of nginx as a
| reverse proxy that uses these as upstreams. That's not the
| most "container-first" way of working, but it worked for
| me. Caddy can do similar. Traefik is more integrated with
| docker and the docker ecosystem, but that might do what you
| need better.
| hdjjhhvvhga wrote:
| True, it's not easy. In the past I used to run everything on
| Debian stable for that reason. And after every two years or so,
| I'd book two days for the upgrade, with some time offline.
|
| The problem was, at some point Debian stable got so distant
| from modern infrastructure you had to patch it from the
| beginning with newer versions, so reluctantly I switched to
| testing (that was years ago). I was surprised to find out
| things were working just fine.
|
| The problem is today everybody distributes software in their
| own way. Especially web software - native packages are more and
| more rare. So automatic updates that work are indeed a
| challenge.
| k8sToGo wrote:
| Automating it away (Ansible and Docker) is a good way to reduce
| overhead. I barley invest any time in maintaining my
| infrastructure.
| AdrianoKF wrote:
| I recently wanted to give my current RPi4 home server a GitOps
| makeover, migrating services (Home Assistant, Nextcloud and
| others) to k3s in the process.
|
| What has been an obstacle is the availability of officially
| maintained Docker images for some of the components I've been
| wanting to use - afaict neither Argo CD nor Rook have official
| armv7/aarch64 images (though it seems Argo will release one
| soon).
|
| Until then, I'll hold off on that pet project until I get my
| hands on a reasonably priced x86 SFF PC (the ThinkCentre M700
| Tiny from TFA looks interesting!).
| yebyen wrote:
| FWIW, FluxCD has full armv7 and aarch64 support (it definitely
| works on a Raspberry Pi 4, the earlier models with 1GB of RAM
| too, although depending on the size of your git repos that may
| not be enough memory, 4GB-8GB certainly is for Flux.)
|
| How many Rpi4 servers are we talking? I have never run Rook
| before (though I ran Ceph, when it was called deis-store way
| back before it was cool) and I always remember that you needed
| at least 3-5 servers with at least one dedicated disk for OSD
| per each to make it work well, if at all!
|
| I'm looking at synology-csi right now for my homelab, and
| wondering if there is hope that I can stop using local-path-
| storage this way, it looks a little bit outdated, so I'm afraid
| that may keep me from running the latest version of Kubernetes
| on it... unless I hack the support into it myself :gasp: surely
| that's gonna void the warranty
| 3np wrote:
| For various reasons I started building all container images
| myself. In most cases when arm64 images aren't provided, they
| will build just fine anyway.
|
| Running you own registry is super simple.
|
| https://docs.docker.com/registry/deploying/
| iamgopal wrote:
| Is there similar repo for SAAS apps ? Not too fancy, just Django
| and a db.
| simonw wrote:
| This is very cool!
|
| Have you considered something like Tailscale so you can securely
| access it from outside your home? I've been thinking about
| spinning up my own home server that way, seeing as Tailscale
| makes it easy to securely access it from my phone when I'm out
| and about.
| omnicognate wrote:
| I haven't looked into Tailscale but I do that with plain
| Wireguard and it's awesome.
| g_p wrote:
| It's not quite the same as a VPN, but in addition to Tailscale
| and some of the alternatives suggested in other comments, it
| may be worth looking at using HTTPS client certificates.
|
| Done right, you get a pretty simple second line of defence -
| you can validate the client has an appropriate certificate
| (running your own CA is pretty straightforward for small-scale
| home use or for a small group of users). Without such a
| certificate, users can't access the web service.
|
| If your goal around a VPN is a second line of defence against
| the application's own authentication logic failing, client
| certificate authentication might be worth a look. If your
| threat model needs to cover a major issue in your web server,
| you might still want to stick with a VPN-based setup.
|
| (You can of course do both, and bind services to an internal-
| only IP that you can only reach via a VPN, then have
| certificate auth on that too if you so desire)
| pydry wrote:
| The headache I find with client side certs is that it's
| annoyingly common for client side apps to have their own CA
| store which you have to track down and install your CA.crt
| in.
| mlangenberg wrote:
| I found the Tailscale Android app to take quite a big hit on my
| battery when used as an always-on VPN, so I use cloudflared on
| a VPS to create a Google authenticated tunnel to home assistant
| (over Tailscale).
| kinduff wrote:
| I use Tailscale to access a jumpbox I have in my home server,
| but it could also be installed in all the nodes to access them
| individually.
|
| I really like the service, works like a charm. Sometimes I use
| one of my servers as an exit node when I'm out as a VPN.
| chaxor wrote:
| Isn't headscale a clearly better option, since it removes the
| need to trust and depend on external sources? It the same
| software essentially, but if we're talking about self hosting,
| headscale is just inherently better, since it actually is self
| hosted.
| egberts1 wrote:
| !Dos Mios! 129 Go dependencies just for a Headscale control
| server?!
|
| I think I will wait until a C variant appears after they've
| settled.
| mcescalante wrote:
| I just stumbled on a self hosted project similar to tailscale:
| https://github.com/gravitl/netmaker. There is also
| https://github.com/juanfont/headscale which is an open sourced
| implementation of the Tailscale coordination server. Tailscale
| "just works" and makes things incredibly easy for personal use,
| but it's awesome that there are also similar self-hosted
| options out there
| khuedoan wrote:
| (Repo owner here) I'm planning to build my own router with
| OpenWRT and install Wireguard on it, however due to the chip
| shortage I can't get the hardware for a reasonable price yet.
| Netmaker looks amazing, I need to play with it.
| afeiszli wrote:
| Netmaker repo owner here. Let us know if you have any
| questions and we can help out!
| Datagenerator wrote:
| For small use cases at home you might want to try GL.iNet
| GL-MT300N-V2 ( generally available within days ). Or
| virtual for free, ymmv
| zrail wrote:
| I'm using Tailscale on my home network. Right now my two most
| interesting use cases are 1) I have it installed on my Home
| Assistant VM so I can access it wherever I am without having to
| futz with internal vs external URLs, and 2) I have a few
| Raspberry Pi's scattered around the house running zwavejs2mqtt
| which doesn't have any authentication built in (i.e. anyone on
| my wifi could theoretically mess with my zwave network). I told
| it to listen on each Pi's Tailscale address and use ACLs to
| lock it down so just Home Assistant can talk to the websocket
| API.
| rhizome wrote:
| Honest question: why not SSH?
| omnicognate wrote:
| The difference vs SSH for me (using plain Wireguard but
| achieving the same thing) is that I can set up my phone with
| always-on VPN to home and then wherever I am I'm effectively
| on my home network. I can access all my home-hosted services
| in my browser, securely access IOT things that can't talk to
| the open internet, use my home-hosted DNS rather than my
| mobile provider's, etc. I do use ssh over that connection
| (with Termux) to talk to servers, ofc.
| peterbraden wrote:
| They solve different problems. Tailscale basically allows you
| to ignore any NAT topologies separating you from your
| machines. You can have a pseudo local network of your
| machines behind a VPN, allowing SSH to any machine with zero
| routing issues. And it's incredibly seamless. I'm a big fan.
| rhizome wrote:
| But in this model wouldn't you control the NAT config?
| You're already running servers, so outside access isn't an
| issue. I dunno, OP is running that "Cloudflare tunnel"
| thing for exposed ports so maybe there's something I don't
| know, or that the internet connection is still some
| consumer thing that prevents or forbids running servers.
|
| Actually, now that I look closer they aren't running their
| own email, so maybe this really is primarily a mostly-
| internal-access project. I do like those SFFs tho!
| smasher164 wrote:
| This is how I currently manage my RPi NAS on NixOS:
| https://github.com/smasher164/nas.
|
| Exposing Home Assistant over it allows me to do things like
| turn my lights off when I'm away from home, without exposing
| stuff to the public internet. https://www.blog.akhil.cc/shelly-
| dimmer
| hahajk wrote:
| Before I clicked I thought "man, a single config.nix for his
| whole system? this is going to be big..." But I've
| continually been surprised how little configuration you
| actually need to do in nixos. I've been planning on
| configuring Home assistant on an RPi myself, so this will
| help. Thanks for sharing!
| tksb wrote:
| I was just looking into doing something similar after
| realizing how little my RPi3B+ running only HomeAssistant was
| actually doing. I'd love to have PiHole and Shelly among
| others so maybe this route on a RPi4B is the way to go.
| Thanks for this!
| Orou wrote:
| Are there any major benefits to using Tailscale over vanilla
| Wireguard?
| hobofan wrote:
| UX. Compare the quick start of Wireguard[0] and Tailscale[1].
|
| [0]: https://tailscale.com/kb/1017/install/
|
| [1]: https://www.wireguard.com/quickstart/
| Macha wrote:
| In their benefits:
|
| > Host a private server for you and your peers. Use it to
| play Minecraft together or chat together on Discord.
|
| I don't think the documentation writer knows how discord
| works. There's no private/self-hosted option
| [deleted]
| 0xCMP wrote:
| NAT busting is a plus. Avoid needing DDNS and opening a port.
| Also easier setup for accessing other stuff on the network
| via that computer vs up/down rules with a standard wg config.
|
| For a small setup thats the big thing, but for anything a
| little more it does key rotation, handles ips for you, and
| offers "magic dns" which makes all the devices get a nice DNS
| address when tailscale is on.
| aborsy wrote:
| NAT busting is a minus. If apps can create holes in your
| network and "bust your NAT," your network is not well
| secured; you should disable UPnP and filter egress.
|
| Also, you gain simplicity but give up to coordination
| servers, that can be a privacy and security problem.
| [deleted]
| rwbhn wrote:
| Tailscale isn't using UPnP. See
| https://tailscale.com/blog/how-nat-traversal-works/
| istjohn wrote:
| That was an interesting read. Turns out they do use UPnP,
| but it's just one of many tools they turn to when other
| techniques fail.
| trulyme wrote:
| Sidenote: this is a fantastic writeup.
| tlrobinson wrote:
| On the flip side, the main downside I found was my travel
| router supports vanilla WireGuard but not Tailscale.
| divbzero wrote:
| I was happy to see that this project describes self-hosting in
| the fullest sense, from hardware on up.
| mikesabbagh wrote:
| it amazes me that hacker news run on a single node.
|
| All this work is nice and beautiful, the problem will come when
| you try to update different components.
| christophilus wrote:
| > it amazes me that hacker news run on a single node.
|
| It amazes me how much of the internet doesn't run on a single
| node, when it probably could.
| jaimex2 wrote:
| Why would anyone run a simple elegant site when you can get
| buried in js dependencies and plumbing stacks for months?
| gigel82 wrote:
| This needs a companion guide about how to set up the host
| machine. Which Linux distro to choose, how to set it up / harden
| it, nftables / firewall, public key login, etc.
|
| If anyone has one handy, I'd appreciate a link.
| khuedoan wrote:
| (Repo owner here) I've already automated that with Ansible,
| including Linux installation (Rocky Linux). There's no
| hardening yet but that's on the roadmap.
| Art9681 wrote:
| Several people in this thread stating they are working on
| solutions to assist others learning modern cloud/devops stacks.
| I'll throw my name in the hat too. I have a single node home
| lab running enterprise cloud/devops software stack. I have been
| working on a project to deploy on-prem cloud (OpenStack) in
| disconnected environment. I've had to learn how to build and
| host all of the infrastructure required to do this without the
| luxury of internet access. I realized the same thing can be
| simulated very cheap in a modern computer with enough cores and
| memory to run the stack as a single bare metal host. My build
| cost $1300 2 years ago but the knowledge gained has paid me
| back 5 figures worth of raises in my career in 16 months.
|
| Having my own personal CloudBox has allowed me to experiment
| and fail fast. I am ahead in experience and knowledge than the
| rest of my team as a result. I have a tool they do not. I
| realize it would be better if we all had a tool like this.
|
| So that's the pitch, a single node "cloud in a box', the
| CloudBox for IT students or professionals to learn any aspect
| of IT.
|
| Now I just need lots more time to actually turn my prototype
| into a product.
| indigodaddy wrote:
| Doesn't the ansible portion do all that? Believe he uses Rocky
| Linux.
| dmayle wrote:
| You are unlikely to find what you're looking for. I'm going to
| dig into this article a bit, but what you're talking about is
| inherently hard, and delivers tremendous value to businesses.
| There are a lot of people doing it for money, and someone
| builds something that starts to actually work, they get
| acquired, or they take their project and make it enterprise-y
| (because that's where the funding is).
|
| There's flatcar and k3os and fedora coreos and talos and
| lokomotive. There are maybe a dozen others as well, but those
| are the ones I know something about.
|
| The real problem is that the orchestration of PXE boot, DHCP,
| name services, server registration, cluster bootstrapping,
| while simultaneously building a light distribution that makes
| the right calls on persistence, security, etc. is just *really
| hard*.
|
| I took at a stab at it myself (failed startup) and have a
| custom distribution based on alpine, but the amount of work to
| go from there to everything is so large that it's tough to take
| on if you're small (and there is the constant desire to go
| enterprise because of the money)
| gigel82 wrote:
| Thanks, some of this went over my head and sounds way too
| complicated.
|
| I'd be satisfied with a home-user-oriented manual tutorial,
| like install Debian with these packages, a nftables setup
| that firewalls everything but these 3 ports, how to setup
| auto-update, turn off root & password-only logins, and
| general things to be reasonably secure; as well as tips for
| on-going maintenance and so on.
| mrslave wrote:
| I've been eyeing off OpenStack for sometime for a similar use
| case, but I must admit it is an uninformed opinion. What am I
| missing between the two solutions (OpenStack vs this specific
| custom solution)?
| unixhero wrote:
| Extremely impressive
| candiddevmike wrote:
| I enjoy reading about homelabs that aren't using kubernetes or
| some other scheduler and how they're doing it/what the full stack
| looks like. This is just another "how I installed kubernetes"
| thing only without any real scale behind it. There are other ways
| to deploy and run things, and it takes people toying around with
| alternatives for something unique to spring up.
| nullwarp wrote:
| I've used everything under the sun and for homelab use IMO
| nothing beats using docker-compose and salt.
|
| 1. Unless you have crazy dynamic scaling needs, there's
| literally no point in using k8s or nomad 2. Unless you want
| RBAC and fine tuned ACL, theres literally no point in using k8s
| or Nomad
|
| I hate this idea that you need a Ceph cluster and a 3 node HA
| cluster to have a home lab.
|
| Not saying don't do it because you want to play/test/learn any
| of those platforms, just that if you are going to run a server
| service maybe consider if you are doing it for learning a task
| orchestration service or just want to run XYZ reliably.
| sgarland wrote:
| I was settled on Docker-Compose and Ansible for a long time,
| and while it was indeed quite easy to use once everything was
| tweaked correctly, what finally got me to move to k8s was HA.
| Having a single node with nginx, DNS resolution and Unifi
| controller running on it meant if I ever took the node down
| down for something, everything broke. Plex being down because
| I wanted to play with something unrelated wasn't ideal.
|
| Agree that Ceph is ridiculously overkill though, I tried.
| Longhorn is much easier and perfectly fine for distributed
| storage.
| Macha wrote:
| Unifi controller isn't a dependency for your network to
| continue working. Config is distributed to the devices
| themselves, the controller is just a UI and collector of
| data.
| moontear wrote:
| I also run everything with docker stack(s) currently. Not
| using salt and haven't heard of it. Why would I want salt and
| not ansible or something similar?
| mafro wrote:
| Salt (https://saltproject.io/) was an almost-winner around
| the time that Ansible blew up. It's pretty irrelevant
| nowadays since terraform and kubernetes - but still it's a
| very very good tool.
|
| I'd suggest if you already know Ansible then don't change.
| If you have no idea which is which then try them both and
| let us know :)
| dijit wrote:
| I used saltstack at a decent scale.
|
| The reasons that I used it were because the agent (called
| "minion") initiates a connection towards the master, which
| can be handy if you're behind NAT- but it was interesting
| to us because incoming connections are easier to manage at
| scale than outgoing ones.
|
| Another reason was that the windows support was much more
| mature (though not perfect) and our environment was mostly
| Windows servers.
|
| That said, installing the minion agent was easy, much
| easier than enabling winrm.
|
| If you have sunk significant time into ansible I wouldn't
| recommend switching, but it's definitely not dead as per
| the sibling comment, I personally found it more enjoyable
| and easier to work with once understood the DSL pattern and
| added a few custom modules, it's very simple underneath.
| thejosh wrote:
| I really like nomad for a single node homelab, it's super
| easy to maintain and deploy stuff.
| Axsuul wrote:
| I run mine using HashiStack (Nomad, Vault, Consul), highly
| recommend going that route over Kubernetes which I have also
| used.
| endigma wrote:
| What do you use for storage?
| Axsuul wrote:
| If it's images, files then something like S3. If it's a
| database then I mount it locally on the host.
| dijit wrote:
| What do you do if the host goes down?
| Axsuul wrote:
| Nothing happens to the data -- it's still intact when the
| host comes back up. Happy to clarify further though if
| that's not what you meant.
| dijit wrote:
| Ah, so your systems are down until you fix a node, and if
| the drive(s) fail you have a dead node you're stuck
| restoring backups.
|
| I guess that's fine, I'm just used to having labs to test
| how I can avoid single instance failures. :)
| movedx wrote:
| I've come from bare metal, to virtualisation, to Cloud. I
| then went from AutoScaling Groups and AMIs to K8s and Docker
| Images. Recently I went from K8s to Nomad+Consul and Docker
| Images.
|
| Now I'm building a small platform to host a business I've
| started and I'm going back to ASGs with simple EC2 Instances.
|
| There's really no need for the features Nomad/K8s offer at
| almost any scale outside of a few big companies.
| Axsuul wrote:
| It depends. Orchestrators like Nomad can give you a lot of
| leverage as a small team. It's not very difficult to manage
| and can give you lots of redundancy, control, and a path to
| scalability while utilizing best practices so you don't
| have to think about it. There are lots of ways to do this
| and not every app needs this but as you begin to get
| traction, you either have a choice of outsourcing your
| devops or bringing it in-house which happens way before you
| are even a big company. Nomad solves the problem of the
| latter.
|
| Also, I'm curious why you didn't just end up sticking with
| Nomad now that you know it.
| Macha wrote:
| My setup I just have ansible deploy to a couple of hosts
| running Arch. Generally I just use OS packages and have ansible
| deploy appropriate configs then for some software where that's
| more hassle I use podman rootless containers. Such software
| includes e.g. Graylog, which wants particular versions of
| Elasticsearch and mongo, or postgres which requires a manual
| data migration for annual version upgrades
|
| k8s and nomad are on my "maybe eventually later" list though.
| depingus wrote:
| I'm currently building out my single-node "homelab" with coreos
| + podman + systemd. Most of the config is pulled from 1
| ignition file during coreos installation. Its been a learning
| experience and but also very lightweight.
| marginalia_nu wrote:
| My "homelab" is just debian stable, with a moratorium against
| containerization. Systemd keeps the services going. Updates are
| manual. It's a bit incredible how clean and easy such a set-up
| can be. Like there's almost nothing to say. It's extremely
| reliable and just keeps ticking.
|
| I actually did use to have a bunch of virtualization and
| kubernetes and crap, but got rid of it because it ate literally
| half the systems' resources, in a death by a thousand
| containers-type of way. There was also a lot of just jank,
| stuff was constantly breaking and I was always trying to piece
| together why from half a dozen out-of-date tutorials. Felt like
| herding cats.
| ryandrake wrote:
| Yea I think I missed the whole web-complexity bus and am
| probably out of touch, but I still don't get the use case for
| docker and containers and kubernets and and orchestration and
| all that stuff, just for a simple home setup. I serve a tiny
| web site, email, backups, a NAS and a few other internet
| services for my family, and my "stack" is vanilla Debian
| Stable.
|
| Maybe I don't know what I don't know, but my setup works for
| me and I don't really have any problems maintaining it so I
| figure why add all the complexity?
|
| It always feels weird to see threads and threads of people
| talking about dozens of software programs I've never even
| heard of, let alone used. Maybe I'm living in the past but to
| me a "stack" is: OS, server, database, application. Like
| LAMP. Wonder when this changed!
|
| It makes me curious about what kinds of stuff people do in
| their home networks that I never even considered doing.
| sgt wrote:
| It doesn't make sense, but often people who set up K8s at
| home do it for educational reasons or just having a
| playground so they are more capable at administering
| systems like that at work.
| iso1210 wrote:
| Sure, using K8S in your learning envornment because
| you're managing 800 different services in your dayjob is
| fine. Personally I have my learning environment as part
| of my day job though.
|
| At home I just want things to work, hence a nice simple
| LAMP
| sgt wrote:
| Agreed. I don't even want Kubernetes at work.
| naasking wrote:
| > but I still don't get the use case for docker and
| containers and kubernets and and orchestration
|
| For docker there's a simple motivating case: some services
| are difficult to configure securely with minimal
| permissions, and having a standard docker image provided by
| people who know what they're doing would be a big net win
| for security on the internet. There are a lot of poorly
| configured and insecure http server's out there. Think
| about how many vulnerable http servers are running on
| routers.
| Macha wrote:
| I want multiple conflicting versions of Elasticsearch on
| the same host. My OS packages are built to support a single
| deployment. I could manually setup users, config files,
| systemd services, etc. for the second deployment, or I
| could just run them in containers, which is the lower
| effort option
| marginalia_nu wrote:
| > I want multiple conflicting versions of Elasticsearch
| on the same host
|
| Why do you actually want this? Even having one ES is a
| bit of a smell that you're probably over-engineering
| things.
| Macha wrote:
| Dependencies of software I didn't write (e.g. Graylog)
| marginalia_nu wrote:
| This is what I'm getting at. That is an enormous overkill
| for a single server, and exactly the problem with
| kubernetes: It drives so much complexity that you kind of
| need this type of heavy-duty enterprise grade solutions
| even in an area where you really shouldn't need it.
|
| If it's just a single server, you're fine with logrotate
| and grep.
| sgarland wrote:
| I like practicing enterprise at home; it makes me feel
| more comfortable using them at work, where they're
| needed.
| Macha wrote:
| So first of all: I do have to clarify that I just use
| bare podman containers deployed by ansible and not a
| while k8s setup, so I'm defending containers and not k8s.
| But anyway:
|
| Who said graylog is only logging on that single server?
| That server is by far the biggest and hence why I run
| multiple services with big dependencies on it, but I have
| that server, 2 cloud servers (one running my personal web
| page and other publically accessible services, one
| running a couple of game servers) + a couple of Raspberry
| Pis.
|
| Even if I didn't have all that, a web interface to
| configure it to email on ssh login or send me a mobile
| notification that my offsite backups are still happening
| and/or not happening for devices or similar items would
| still be a nice feature, even if you personally feel it's
| too complex or you could write a cron job for each of
| them.
| ClumsyPilot wrote:
| "This is just another "how I installed kubernetes" thing
| only without any real scale behind it."
|
| It is just a learning project, for me K8S didn't 'click'
| untill i tried to configure it myself on a few machines.
|
| There is something about doing it om physical real things
| that aids learning, like you could read all chemistry
| textbooks in the world but untill you actually try it
| yourself it's not quite the same
|
| I think it's important to keep you 'precious
| files/services' and you experiments separate.
| sgarland wrote:
| I've transitioned from a single server running services on
| baremetal Debian, to running services with Docker on Debian,
| to running services with Docker on Debian VMs under Proxmox
| (aka Debian with a repo), to running services on a three-node
| Kubernetes cluster using k3os (I was using Talos, but it
| doesn't have Longhorn support yet, and Rook/Ceph was a
| nightmare).
|
| The baremetal server was the easiest to set up, obviously,
| but the hardest to maintain. Where did I put that random
| config? Wait, why did this break when I upgraded X? Switching
| to Docker (and later Compose) wasn't that difficult, and made
| maintaining services much easier. Going to k8s has been
| challenging, mostly because of how I'm doing it - a
| controlplane and worker node on each physical node in
| Proxmox. Writing the YAML is, well YAML.
|
| I'm mostly looking forward to not having to keep my VMs
| updated. I automated a lot of that (VMs are Packer templates)
| with Ansible, but it's still annoying. Upgrading k8s nodes is
| a lot easier, IMO.
| oceanplexian wrote:
| Yeah I'm with you. I run a rack at home with servers running a
| few applications orchestrated with a bit of Puppet, it's 100x
| more reliable than K8s, I'm not having to do weird hacks to
| make persistent storage work, edit dozens of YAML files and run
| a whole host of dodgy control plane services.
|
| Sadly, I leaned this the hard way by setting up a 4-node k8s
| cluster and realizing bare-metal support is abysmal and the
| maintainers simply don't care about you if you're not running a
| cluster on AWS or GCP. Unfortunately my day job still involves
| k8s and people give me strange looks when I suggest metal so
| substantially less complex, cheaper, more performant, and
| easier to orchestrate and maintain.
| movedx wrote:
| I'm building my own education platform around getting people
| into IT and towards a career in Cloud and DevOps.
|
| A few months back I built a Nomad based cluster with Consul
| and was like: wow! This is super cool... but it's also
| expensive and complex.
|
| I'm going to be building out the platform with an ALB, two
| EC2 Instances in an AutoScaling Group based on an AMI I bake
| with Ansible and Packer. I'll use Terraform for the entire
| infrastructure.
|
| The release pipeline for content updates (it's an
| MkDocs/Material based book with videos blended in) will be me
| updating the AMI and firing off a `terraform apply`.
|
| I won't be using Docker. I won't require or need an
| orchestration engine.
|
| It's a static website delivered via a custom web server
| written in Go (as I need custom "Have you paid to view this
| content?" logic.) That's about as complex as it gets for now.
|
| I'm actually looking forward to implementing it and watching
| as something so simple hopefully goes on to allow me to build
| and scale a business.
| throwaway894345 wrote:
| Kubernetes is just young. The story for bare metal is
| improving all the time. You just have to pick the right
| components for things like storage and load balancing, which
| is tedious but no more so than picking the right combination
| of tools to use to build your own Kubernetes replacement. In
| time even this pain will go away once someone builds a decent
| bare metal distro with all of this stuff available out of the
| box (k3s is trending in that direction).
| throwaway894345 wrote:
| Personally I don't use Kubernetes for the scalability, but
| because I haven't seen anything else remotely maintainable (and
| yes, Kubernetes still has lots of room for improvement).
|
| I don't want to build a good chunk of Kubernetes myself just to
| host my software. It's nice to be able to Google for solutions
| to my problems or to ask on various forums/chat rooms (or in
| many cases, not have the problems in the first place because
| someone else committed the fixes to Kubernetes long ago).
| Lamad123 wrote:
| I unaccidentally remembered Gilfoyle's famed "what do I do?!"
| jaimex2 wrote:
| I miss LAMP :(
| rambambram wrote:
| It's still there, also waiting for you my friend.
| midasuni wrote:
| I use LAMP. WHy would I miss it?
| kstenerud wrote:
| This is definitely cool, but it also highlights a huge problem we
| have with software nowadays.
|
| There are 19 stacks in this repository. 19 pieces of software
| that require their own maintenance, JUST TO RUN YOUR
| APPLICATIONS! The amount of extra work required just to host the
| software that views your pictures, plays your videos, and allows
| chat with people is absolutely insane.
|
| I'm not talking about this particular repo; it's just indicative
| of how complicated things have become that you must do the
| equivalent of building and maintaining your own operating system
| just to get even simple things done. And I belive that it's
| unnecessarily complicated (once again, not this repo, but rather
| the state of affairs in general). We're at an inflection point in
| the industry, and haven't come out the other side yet.
| rapnie wrote:
| > There are 19 stacks in this repository. 19 pieces of software
| that require their own maintenance, JUST TO RUN YOUR
| APPLICATIONS!
|
| There's tools like Yunohost [0] that can save a lot of work and
| are focused on making the subsequent maintenance burden easier
| too. There's an application catalogue [1] with a lot of popular
| self-hosted apps to choose from and new ones being packaged all
| the time. I haven't used it myself yet, but hear very good
| experiences from people that have. Especially those that lack
| the more sysadmin tech skills and just wanna click-install &
| go.
|
| There's also Freedombox [2]. I'd be interested to know about
| other such tools and your experience with them.
|
| [0] https://yunohost.org
|
| [1] https://yunohost.org/en/apps
|
| [2] https://freedombox.org/
| usrme wrote:
| Thank you so much for name-dropping YunoHost! The demo alone
| sold me on finally going the self-host route for things such
| as ebooks, music, and photos.
| [deleted]
| nyellin wrote:
| It's not just running an app. He's building a setup that
| supports FANG-level infrastructure with autoscaling, self
| healing, etc.
|
| Maybe it's unnecessary for tiny companies, but if you're
| dealing with infrastructure at scale then the complexity is
| unavoidable. The only question is whether you want to invent it
| in house or use a robust system that someone else built
| properly.
|
| I elaborated more in my comment below
| tluyben2 wrote:
| I could post my fully automated self hosting infrastructure,
| but it would be boring. Few bash ssh scripts and LEPP stack.
| None of this complexity and yet 10+ years uptime. Interesting
| nonetheless but think i'll pass on this. I have done these
| setups at cloudproviders for clients; that is definitely the
| risk averse version; if something goes wrong every month with
| AWS, it's just 'ok'; if something goes wrong every 5 years with
| your non-cloud, it's panic & heads need to roll (even though it
| costs a fraction).
| eloeffler wrote:
| > I could post my fully automated self hosting
| infrastructure, but it would be boring.
|
| Actually, I would find that not boring :)
| rawoke083600 wrote:
| Yup ! This I've used bash+ssh for the last 10 years for my
| solo founder and commercial products. Is sexy or even nice ?
| Absolutely not, but damn its simple and most Linux semi
| professionals can understand it with a glance.
|
| Bash as infrastructure+maintenance is almost like SQL for
| data.
|
| Everyone agrees is not fantastic, many have tried to improve
| and yet SQL + BASH is still around chunking along doing their
| unsexy things just good enough :)
|
| Never underestimate software that is 'just good enough' :)
| legends2k wrote:
| Isn't it how the general software philosophy is? A composition
| of decoupled, do-one-thing-well pieces working together to
| achieve a larger task?
| kstenerud wrote:
| Yes, but after the initial monoliths of mainframes, we had a
| golden era of operating systems, where all of the plumbing
| and infrastructure to run your applications was maintained by
| someone else (Microsoft, Apple, Commodore, Atari, SGI, Sun,
| Palm, Redhat, etc).
|
| Now we've come full circle back to the bad old days where you
| need an entire team of dedicated people and arcane knowledge
| just to run your application software again.
|
| I suspect that this will continue until we reach a point
| where the big players out of necessity come to an agreement
| for a kind of "distributed POSIX" (I mean in spirit, not
| actual POSIX). These are exciting times living on the edge of
| a paradigm shift, but also frustrating!
| midasuni wrote:
| Apt install your lamp stack
|
| Deploy your files
|
| Cronjob to apt upgrade every night
|
| Why does that need a full team? Or more than 10 minutes
| every few years?
| chaxor wrote:
| Ummm I'm sorry, but I don't see how this is _practically_
| simple. _Theoretically_ , maybe - [and I agree with the
| idea that there is _potentially_ (hopefully) a paradigm
| shift in initial play now, with the thrust being
| multifaceted including companies ' lack of upholding user
| privacy/selling you as a product, slowly increasing
| service charges, and lack of customer customization.]
|
| I think tis thrust towards self hosting should make the
| construction of these systems closer towards what you
| describe, but it isnt really that simple yet.
|
| For instance, if you want to self host even a decent all-
| in-one calendar/notes/etc system like nextcloud, it isn't
| just `sudo pacman -S nextcloud` and youre done.
|
| There is an enormous consideration of how to construct
| the 1) network architecture, 2) encryption of disks, 3)
| secure/encrypted, incremental, and full filesystem
| offsite backup, 4) security and encryption of network, 5)
| secure interplay between self hosted and non self hosted
| data, etc.
|
| An example of (1) may be that, much if the discussion
| around self hosting states that one should not open their
| LAN to the internet with port forwarding (as it's
| inviting you to be hacked), and you should rather point
| your registered domain to a VPN such that you must login
| with any remote devices to access your server _(just to
| point out - likely here you would need to use a dynamic
| dns service to get access to your system, which is also
| another hassle for newcomers)_.
|
| Further on (1), this involves ensuring that the choice of
| VPN is adequate (many older technologies have known
| security flaws, like oracle padding attacks, etc.) - and
| finding a combination of routers with correct firmware to
| allow for VPN servers, (also VPN clients on top of that
| if you care about not having your isp sell you to
| everyone) the correct hardware and network architecture
| that can properly mitigate the encryption of traffic on
| your network (e.g. can a raspberry pi really handle all
| of your traffic and maintain 1Gbps file download/uploads
| with vpn encryption?).
|
| I recognize that much of this is not really possible to
| `sudo apt-get` on one system with recommended hardware,
| but the more FOSS out there that can get closer to
| mitigating all of this, and to have well packaged
| tutorials on how to safely and properly self host (e.g.
| complete google service replacement) while maintaining
| security and 10Gbps speeds, the better off we will be.
|
| This really is what we should be driving towards.
| Companies can certainly offer their servers for customer
| use, but I think it should be expected that companies
| have the code that is used to store that customer data
| fully as FOSS, for auditing and to allow for self
| hosting. There's still money to be made for companies in
| helping people self host everything for themselves - we
| just have to push for it.
|
| One decent enabler of this is companies that make
| software for data backups. This is another one of the
| enumerations above, because it is quite difficult to find
| a solution that has quantum secure encryption, excellent
| compression, incremental (only pushing diff like git)
| capabilities, and is reasonably simple to use. AFAIK,
| Duplicati is the only option that seems to be able to hit
| all of these well (hopefully someone can correct me).
|
| Anyway, this was a meandering long way to say - self
| hosting is complicated, but i do really hope we can
| change that.
| midasuni wrote:
| I've been self hosting for over 20 years on apt based
| systems, aside from changing from telnet/ftp to ssh,
| adding in letsencrypt, very little has changed. You can
| still collocate your hardware, but it's cheaper to use a
| generic VPS. You can host from home, but even a pi will
| probably cost more in electricity use than a $3/month
| vps.
|
| Backups are far easier now as you use a VM provider like
| linode, digital ocean, even lightsail, which will
| schedule snapshots. If you run your own hardware then
| mysqldump and restore to your backup server is a small
| shellscript and cronjob. Disk encryption is handled by
| your OS.
|
| LAMP was trivial in 2002, it still is. You can use nginx
| or postgres instead of apache and mysql, but it's broadly
| the same.
|
| If you want to make your site complex and dependent on
| thousands of JavaScript libraries and frameworks which
| change every year or two, that's fine, but you don't need
| to, it's a choice, one which adds complexity. If you want
| five nines or absolute guarentee of not failing, you need
| to think more about replication than just a nightly
| snapshot, but that's not a problem solved with thinks
| like kubernetes.
|
| If you want to scale to millions of concurrent users
| pulling terabytes, sure, don't self host from your DSL on
| a pi. If you want to serve a personal site for hosting
| bits of stuff, it's not hard.
| 867-5309 wrote:
| one does not simply "deploy"
| midasuni wrote:
| tar -xvf of your git repository
| xupybd wrote:
| Okay but what happens when you have a breaking change
| from one of your updates?
| midasuni wrote:
| From a standard system upgrade? Never happened to me in
| 20 years. Between OS versions sure, once every few years
| spend 20 minutes building from scratch
|
| If it did happen, same as if your machine caught fire.
| restore from your backup
| xupybd wrote:
| For me it's when a library my app uses gets an update
| then my app fails.
| emteycz wrote:
| No need to update the library. Or keep both versions.
| rambambram wrote:
| I want to visibly second this (with a comment, not only
| an upvote). I dubbed the LAMP stack the CHAMP stack for
| some time now (CH standing for CSS and HTML).
| theK wrote:
| That's actually very insightful. The OS era you describe
| was mostly local with not so many things happening in
| online interactions and way less devices collaborating to
| create for example your images folder.
|
| Now that we mostly have the server-client model ingrained
| in every online activity there is an obvious complexity
| centralisation on the server side.
|
| Might be interesting to see how web3 and other
| decentralised web movements might factor in to this!
| INTPenis wrote:
| Most of that is kubernetes, which is completely unnecessary for
| self hosting purposes. It's mostly for learning, or for the
| convenience of hosting multiple different services, trying many
| different containerized applications.
|
| If your only goal is to serve Nextcloud, Plex or whatever to
| your family, you can get away with much less than that.
| dgellow wrote:
| You don't need all of this for a home server, the author is
| just having fun.
|
| Systemd units and a bit of sh is enough if you just want your
| applications.
| tiarafawn wrote:
| I agree that the number of different technologies here is
| shocking, but you get a lot more value as a result than "just
| to run your applications". Most notably that includes CI/CD,
| scaling, and redundancy, all with IaC. These things used to
| require a lot more administration and development effort.
|
| Plugging together all these different tools has become so much
| work though that in many organizations the platform team(s) who
| are mainly occupied with doing just that take up a lot of
| engineering resources.
|
| In my opinion the next evolutionary step would be for all of
| this to be bundled and abstracted away. Funnily enough, we
| pretty much have that product already with autoscaled
| "serverless" cloud services, GitHub Actions/Azure DevOps
| pipelines etc.
|
| The biggest problem is probably that things like AWS Lambda,
| SQS, and DDB lack versatility and user friendliness. If we get
| some improvements on that front, many organizations might opt
| for that instead of dealing with their own K8s deployments.
| Even better would be if we had something like a stripped-down
| version of OpenStack, just focussed on doing "serverless"
| right, and rebuilt from the ground up.
| ekianjo wrote:
| You don't need Kubernetes to do self-hosting. Completely
| overkill.
| dspillett wrote:
| Overkill probably, but useful practise for when working on
| something of a scale where it will make a significant
| difference. And they might just like playing with these things.
| A home lab does not have to be DayJob practical!
| status_quo69 wrote:
| Some people just like to tinker and try other things even if
| it's overkill. I have a k8s cluster on DO to host shitty side
| projects, it's pretty nice to just `kubectl apply -f
| manifest.yml` and be done with it, especially since I'll
| probably wind up tearing everything I just set up anyway.
| throwaway894345 wrote:
| I fully agree that setting up Kubernetes is a lot of work, but
| I think people forget how much work it is to set up anything
| _maintainable_ for more than a few services running on a
| handful of services.
|
| The nice thing about Kubernetes is that it's probably not much
| harder (if at all) than the alternative (assuming you already
| know which tools to use to compose your bespoke platform) _and_
| you can easily find online resources and support (never mind
| all of the problems you _don't_ run into because they're
| already solved in Kubernetes).
| walrus01 wrote:
| I would rather choose something like xgp-ng as a hypervisor for
| a home lab environment such a this, instead of a custom one-off
| kubernetes environment.
|
| Or just a total DIY xen or KVM hypervisor on debian or centos
| approach. But with considerably more RAM in the host machine
| than 16GB (like 64-128GB).
|
| the kubernetes sort of makes sense if the purpose is also for
| the person who owns/operates it to test kubernetes things,
| learn kubernetes, or use it as a home lab of some larger
| similar environment that they have elsewhere.
| moondev wrote:
| > Or just a total DIY xen or KVM hypervisor on debian or
| centos approach.
|
| Or just install kubevirt on the existing cluster and manage
| kvm virtual machines with k8s
|
| https://github.com/kubevirt/kubevirt
| the_duke wrote:
| You don't need it.
|
| But Kubernetes exists for a reason and makes a lot of things
| easier.
|
| If you are familiar with the ecosystem, k3s is a great
| foundation for self-hosted setups.
|
| Especially thanks to the many helm charts and operators that
| exist.
| z3t4 wrote:
| Talking about overkill, he also use 4 physical machines each
| with a 4 core CPU. But its not about what you need, its about
| what you want to learn. So if you want to learn about
| cluster/orchestrating i think its warranted.
| comprev wrote:
| I use something very bare bones as the host OS, say Alpine, and
| install Docker with a handful of other tools for basic sysadmin
| tasks (curl, etc.).
|
| For a single server deployment docker-compose is very useful.
| qbasic_forever wrote:
| Very cool, this is like a perfect little kubernetes development
| environment. The combo of tekton + gitea + k8s is really nice and
| just a shame it takes bodging all the lego pieces together
| manually right now. I wish there were a one click exe install
| like Docker for Desktop that gave people an entire environment
| like this on their desktop. Like a little private github but so
| much more powerful.
|
| If you're looking for some nice stuff to develop on an
| environment like this, check out VS Code and Google's cloud code
| addon:
| https://marketplace.visualstudio.com/items?itemName=GoogleCl...
| It's not totally tied to their GCE cloud offerings and is
| actually just a VS Code integration of their Skaffold tool (a
| fantastic inner dev loop tool for k8s). It works great with
| minikube or any other k8s cluster, presumably this one too. You
| can very quickly get up and running with code in your IDE now
| running and debugging on your k8s environment.
| khuedoan wrote:
| (Repo owner here) Glad someone mentioned it, I do have one:
| https://homelab.khuedoan.com/try_on_a_vm
|
| Although it's still incomplete and not _one click_ yet, that 's
| the direction I'm heading in: anyone can try my homelab on
| their PC, and if they want to use it on their real hardware,
| they can continue with the full install.
| turtlebits wrote:
| This is cool, but are you doing this as learning experience? For
| a homelab it seems severely overkill. I understand it all being
| self-hosted, but it could be drastically simpler if you adopt a
| few SaaS products (using free-tier)
| yosito wrote:
| I have mixed feelings. There is some over-complication, but I
| wouldn't call infrastructure without backups complete.
| citizenpaul wrote:
| I've been burned too mamy times. Once a product realizes that
| x% of its free tier uses only feature X they suddenly put that
| feature behind a paywall. Now I"ve got to research and setup an
| alternative or pay 20$ for some minor convenice I set it up
| for.
|
| Most free tiers requires a card on file. I shudder to think at
| having to check dozens or hundreds of such services regularly
| to make sure they have not sent out an email about how we are
| now charging you. Then I don't notice for 6 months and find out
| I spend $150 on something I could have setup myself in less
| than an hour that will never charge me.
|
| I have had hundreds of such products subscribed at work. At
| least every year or so an entire day gets burnt updating credit
| card info from logging in and looking up the bespoke way every
| single site requires the updates to be done for payments.
|
| The services that suddenly shut their doors and again require
| me to research and setup an alternative.
|
| Services get acquired all the time then switch to pay only and
| they have your CC on file already for easy billing.
| khuedoan wrote:
| (Repo owner here) Yes, I'm mostly doing this as a learning
| exercise; there's still a lot of work to be done before I can
| rely on it to host my services.
|
| > For a homelab it seems severely overkill
|
| Isn't that the point of homelab? ;)
| pdonis wrote:
| _> it could be drastically simpler if you adopt a few SaaS
| products (using free-tier)_
|
| For some people (including me), the risk involved in any SaaS
| product suddenly either dropping or imposing unworkable
| restrictions on free tier is high enough to make the extra work
| involved in self-hosting worth it. (Granted, my current self-
| hosting setup is a lot simpler than the one described in this
| article, but even if mine were more complex I would still say
| the same thing.)
| b33j0r wrote:
| Personally I dig it. I recently did something similar, and felt
| dirty not automating everything, every step of the way.
|
| I knew I'd want to change the LAN architecture and the
| services, and everything; but it was definitely intended as a
| learning experience for me.
|
| Kinda looking forward to doing it over again with my
| new/refreshed skillset. Automated everything.
| nodesocket wrote:
| What do you use to provision Kubernetes persistent volumes on
| bare metal? I'm looking at open-ebs (https://openebs.io/).
|
| Also, when you bump the image tag in a git commit for a given
| helm chart, how does that get deployed? Is it automatic, or do
| you manually run helm upgrade commands?
| khuedoan wrote:
| I'm currently using Longhorn for storage, but if I find some
| reasonably priced HDDs, I may add or switch to Rook.
|
| When you make a change in git, it is automatically deployed
| without the need for human intervention.
| nodesocket wrote:
| Can you elaborate a bit more on how the automatic helm
| upgrades are performed?
| extinctpotato wrote:
| This is a very good example of how you can save yourself the
| mental hurdle of remembering how to configure something etc.
|
| I can only suspect how much time, trial and error this must've
| taken. This is my main issue with IaC. The concept really lends
| itself to any kind of modern infra, however I'm really put off by
| the sheer amount of time it takes me to whip out a bunch of
| Ansible playbooks and helper scripts, and on top of that make
| sure that all of them are idempotent.
|
| Maybe I'm doing something wrong and this should be easy?
| pishpash wrote:
| Today's stack is way too complex, and fragile. The entire stack
| relies on the goodwill of other projects to continue to be
| maintained/not have breaking changes. Even with the initial
| setup cost s(h)aved there will be a continuous maintenance
| burden. What you get in abstraction and automation you pay back
| in babysitting the additional connecting interfaces.
| christophilus wrote:
| > The entire stack relies on the goodwill of other projects
| to continue to be maintained/not have breaking changes
|
| I agree with the sentiment of your comment, but when was this
| ever not the case, other than the days when you built your
| own OS and tooling from scratch?
| khuedoan wrote:
| (Repo owner here) You're not wrong; it's still not _easy_ for
| me even though I do this in my day job.
| artdigital wrote:
| It's a good habit to always use tools like terraform for cloud
| or Ansible/Salt/Puppet for machines instead of directly doing
| something.
|
| Especially cloud setups that just run containers are relatively
| easy to get idempotent with terraform
| danuker wrote:
| No. It never gets a habit. It is torture, and you have to
| look up the commands every time you do it, because you do it
| infrequently enough not to learn by heart.
| xyzzy_plugh wrote:
| Huh? Compared to... running commands infrequently to
| administer servers? IaC is strictly and comically better.
| necovek wrote:
| Commands you use every day in the CLI? In your editor?
|
| I can trivially enter Emacs and modify Apache config when
| it breaks backwards compat between distribution upgrades,
| but when it's infrastructure as code...
| midasuni wrote:
| I can't think of a single time I've had an Apache config
| break with upgrades - other than having to recompile the
| webservice proxy module to change the case of the Upgrade
| header for a device that doesn't work with the standards
| and is case sensitive
|
| (Major upgrades every couple of years break the binary
| compatibility)
| necovek wrote:
| I am pretty confident it broke for me between 12.04 and
| 14.04 Ubuntu releases, and I am pretty sure both were
| already in the 2.x series.
|
| Most likely the change in NameVirtualHost behaviour as
| listed at
| https://httpd.apache.org/docs/2.4/upgrading.html, but
| there are a bunch of other changes too.
| midasuni wrote:
| Oh sure every few years when you upgrade to a new OS, but
| not a normal security upgrade.
| cube00 wrote:
| Projects are making this easier. For example moving from large
| mono config files to using conf.d/ directories where you can
| drop in extra files and manage each one independently is great
| for IaC.
|
| While it's challenging up front I do enjoy being able to
| freshly install the latest version of Fedora, run my playbook
| and be more or less up and running.
|
| It feels cleaner and more reliable (at least until this week
| when a broken selinux policy made it to stable) rather then
| trying to upgrade packages across major release versions in
| place.
|
| <hat material="tinfoil"> If I've somehow acquired secret hidden
| malware or someone has opened up something to come back in
| later that's also going to get flushed out at least once every
| six months.
| earleybird wrote:
| Pretty sure you're not doing anything wrong - well, if you are
| then I am too :-)
|
| What's often overlooked (I believe) is that when you're doing
| this work in your day job you've for existing infra to support
| your infra along with other folks you can rely on to for help.
|
| With home infra, you first must invent the universe (apologies
| to Carl). Having built 3 variations of home infra (Swarm on
| x64/Beelink, K3s on Pi, K3s on x64/Odroid) I've gained a strong
| admiration for anyone who take this on irregardless of there
| relative success.
|
| What I've learnt over time is to add as little accidental
| complexity as is possible which I think is what you're getting
| at. One incarnation of the Pi K3s was provisioned by Ansible
| (on it's own stand alone Pi that the cluster would netboot
| from). Was Ansible better that imaging the usb drives manually
| and running a small script on each of the nodes? - probably a
| wash. I did however learn a bit of Ansible.
| gorgoiler wrote:
| I wonder how the fan noise is on those M700s. These little Lenovo
| machines are very pleasant. My only wish with my M93 is that the
| temperature would stay below 50'C and keep it 100% silent.
| throwaway894345 wrote:
| This is great. Would love to see something like this packaged as
| a "homelab cluster distro"!
| JamesAdir wrote:
| For people who self host with like setups or even simpler - how
| you guarantee network access? where I live internet might be
| shaky at times, and for that I can't access my home setup while
| away. How do you deal with that?
| Macha wrote:
| Regarding my server connection: I can't help you there, my ISP
| has only had one incident of downtime I've noticed in the seven
| years of being with them.
|
| Regarding my client connection while away from home: If my
| client doesn't have a connection, cloud services don't work any
| better. But a lot of apps (e.g. bitwarden, some jellyfin
| clients, etc) are smart enough to work offline to some extent
| via operating on cashed resources.
| killingtime74 wrote:
| I use ZeroTier. Works well with flaky internet
| 616c wrote:
| Pretty sweet. I have been called a devops person by others around
| me (and I am hesitant to proudly identify as one) because this
| homelab is more impressive and modular than a lot of stuff I have
| seen colleagues and I put together for professional stuff.
|
| Well done. I was not aware of the Cloudflare solution. Is this
| something someone can use, _with_ their Cloudflare Access
| offering, for personal dev/lab envs without breaking the bank?
| altano wrote:
| Cloudflare Tunnels (formerly Argo Tunnels) are a free
| complement to Cloudflare Access. It's an easy way to expose
| internal servers to the public internet, which you can then
| lock down with Cloudflare Access.
| LilBytes wrote:
| I set up Tailscale a little over a week ago, it boggled my mind
| how easy it was.
|
| I'm using it for personal use at the moment and I'm considering
| changing to a paid user to my friends and family can access
| Emby over the internet.
|
| I did have a few dramas getting in to work for my LxC
| environments but nothing a quick Google resolved for me.
| 616c wrote:
| I've been thinking about it but I wanted to set it up on a
| BSD VPN appliance, and I was not sure that was easy for
| pfSense in an easy installable way, then expose some NUC
| systems.
|
| Are you just running it on a system behind a
| firewall/router/NAT-ed network device or on a terminating
| device itself?
|
| But, as usual, far behind on my personal projects ...
| LilBytes wrote:
| Nope. It runs on every client that I want to remotely
| access.
|
| This includes a Proxmox host, a few LxC environments, home
| PC, work laptop and on my phone.
|
| I did have some issues getting it running on WSL2 but as
| long as I can get to the Windows host running it (and I
| can), that's fine for me.
|
| I didn't need to change anything on my network equipment
| which is why I was so amazed at the ease of it's use. I do
| have a static IPv4 at home but from what I understood that
| had absolutely no bearing on my ease of installation.
|
| No port forwarding, no changes to my Meraki MX gateway, no
| nothing. It was essentially plug and play.
|
| Edit: looks like BSD is supported too. Though I don't have
| a BSD environment to test it on.
|
| https://tailscale.com/kb/1074/connect-to-your-nas/?q=Bsd
| Pufferbo wrote:
| Why not use something like ZeroTier? I've seen a lot of
| people talking about Tailscale; I've been using ZeroTier to
| connect to my local NextCloud and it works great.
| zrail wrote:
| Fwiw you probably don't need to pay for tailscale for that.
| You can just have them set up their own single user tailscale
| accounts and then share your Emby node with their account.
|
| https://tailscale.com/kb/1084/sharing/
| endofreach wrote:
| Now all we need is robots hooked into a Ci/CD pipeline that turn
| on computers and initiate the process. And a few more years to
| automate computer production, let the computers find bugs &
| optimize computers, and we have a full circle of life.
|
| Good first version, i am excited for the beta!
| khuedoan wrote:
| Repo owner here, I just created this account, I'm a long time HN
| lurker.
|
| I was surprised to find this on Hacker News, I wanted to wait
| until the stable release before posting on HN, but thank you for
| posting :)
|
| This project is still in alpha stage, but please feel free to
| critique; I'd appreciate it.
|
| Edit 1: After reading some of the comments, I want to clarify a
| few things:
|
| - Because it is currently in the _alpha_ stage, I do not host
| anything important on it.
|
| - This is also my learning environment, I use Kubernetes in my
| day job.
|
| - Yes it is overkill ;)
|
| Edit 2: Wording
| sandGorgon wrote:
| do you use k3s in production as well ?
| khuedoan wrote:
| Is production == my day job? If so we don't, we use managed
| Kubernetes (we used to manage our own clusters but they will
| be decommissioned soon).
| nicbou wrote:
| This is a beautiful writeup. I find these posts really
| interesting to read. Thanks for sharing!
| vimsee wrote:
| This looks very interesting. Already saved your github repo for
| later use. Also, overkill; what is that?
| hunta2097 wrote:
| Your stack looks a _lot_ like my lab setup!!
|
| Mine is based around VMs using K3OS but I have also been
| looking at thinkstations and other cheap Intel-T CPU
| workstations.
|
| How do you find disk performance and reliability?
| khuedoan wrote:
| The performance is pretty good, but I didn't get a chance to
| test the reliability because I keep nuking the cluster to try
| new things :P
| jmakov wrote:
| Thanks for the project! Once deployed, how do you keep
| everything up to date?
| khuedoan wrote:
| I'm currently bumping the versions manually (via a commit),
| but I plan to automate that with system upgrade controller
| [1] and Dependabot [2] (or similar)
|
| [1]: https://github.com/rancher/system-upgrade-controller
|
| [2]: https://github.com/dependabot
| whalesalad wrote:
| Bravo, nice work! I am certainly going to use this as a model for
| my personal setup. Right now I have a few different hosts running
| different hypervisors, but would like to consolidate on kube and
| an iaas tool.
| goodpoint wrote:
| Deploying services is 1% of the work. Maintaining them, fixing
| bugs and bottlenecks is the real works.
|
| And you cannot do when you deploy tools worth millions of lines
| of code.
|
| Complexity matters. Those popular products make sense only if you
| have a 20 engineers is your team or you don't care about
| reliability.
| Gigachad wrote:
| This is downvoted but I think it is true, self hosted services
| don't get nearly as much love as they need. It takes a huge
| amount of effort, monitoring, and knowledge on the specific
| tools to be able to host them securely and detect intrusions. I
| attempted self hosting for years but decided it ultimately
| wasn't worth it and that I have very little ability to prevent
| against attacks or detect that they had happened.
| goodpoint wrote:
| Not my point. Self-host plenty of stuff but choose *simple*
| tools, not behemots.
|
| Also it's important to use software packaged in distros like
| Debian in order to receive security updates for many years.
| spaniard89277 wrote:
| Maybe they want to practice. I installed proxmox and I'm pretty
| much set up, but my needs are pretty huble, just a couple of
| VMs and maybe containers in the near future.
___________________________________________________________________
(page generated 2022-01-22 23:01 UTC)