[HN Gopher] What scripts have you built to stand up a new server?
___________________________________________________________________
What scripts have you built to stand up a new server?
Setting up a new Linux server can take a long time, and it's very
repetitive. I always forget the workflow to set user passwords,
harden ssh with my ed_25519 keys, etc. Show us your scripts!
Author : JP_Watts
Score : 20 points
Date : 2024-10-30 19:52 UTC (3 hours ago)
| bigfatkitten wrote:
| This is what cloud-init[1], Puppet[2] et al a for. I haven't done
| this with shell scripts in at least 15 years.
|
| [1] https://github.com/canonical/cloud-init
|
| [2] https://www.puppet.com/
| PaulHoule wrote:
| I've written Java programs that write a bash script that runs
| under cloud init. It's a lot less code than Puppet, it "just
| works" and you never hear that "you can't get from there" or
| spend three days to find out how to configure something you
| could configure in three minutes if Puppet wasn't involved.
| bigfatkitten wrote:
| I can't say Puppet has ever gotten in my way, but I've been
| using it for a very long time. I will admit it has a steep
| learning curve.
| politelemon wrote:
| Have a look at ansible too, you can have what are called
| 'runbooks' to set up your server the way you want.
|
| https://docs.ansible.com/ansible/latest/getting_started/intr...
| JP_Watts wrote:
| cloud-init looks great, I'll try it for new bare metal
| installs. How do you get it to work in Docker containers?
| thrw42A8N wrote:
| You don't use cloudinit with Docker containers. You'd use
| cloudinit to set up the Docker/K8s host on EC2 or similar
| IaaS.
|
| Software for Docker containers is built like this:
| https://12factor.net/
| bravetraveler wrote:
| On mobile at the moment, can't share - but I have a great library
| of Ansible roles.
|
| Strongly recommended for those looking for inspiration. Give _"
| ansible-doc --list"_ a spin! This shows modules, it can filter
| for existing roles as well.
| JP_Watts wrote:
| Thanks! I haven't looked at Ansible before
| talldayo wrote:
| I have a NixOS config in GitHub that will configure all of my
| servers to be identical when you symlink it to /etc/nixos/ and
| run nixos-rebuild --switch
| erulabs wrote:
| This will be an unpopular answer, but none. Karpenter for AWS,
| Rackspace spot, or microk8s up... Stock Debian + Kubernetes means
| I never have to configure a host system again. Even raid these
| days is one line of config: "instanceStorePolicy: RAID0".
|
| My Raspberry Pis at home are maybe the only exception, in which
| case I don't need to harden anything as they're isolated on my
| network and run trivial things like a webcam.
|
| I've been a Linux sysadmin my entire professional life, and I'll
| never go back to VIMing conf files ever again. It's wonderful.
|
| That said, if you do have to do this, Packer and Ansible still
| exist.
| akira2501 wrote:
| I built a shell and dialog based monstrosity 10 years ago to help
| take stage3 gentoo installs to a full desktop install.
|
| Here's half of it.
|
| https://pastebin.com/ayZ2j9dQ
| eternityforest wrote:
| https://github.com/EternityForest/KaithemAutomation/tree/mas...
|
| These are all meant for RasPi embedded controls, so they don't
| handle a lot of security related things that aren't relevant for
| just a Pi on a private network without ports open.
|
| I set the password with the flasher utility, then have my app
| server just use Linux authentication so I have fewer things to
| mess with and more that can be done with standard tools.
|
| Unfortunately MQTT can't do that and the PKI model is hard to set
| up fully automatically, but almost all routers have guest
| networks and such, so relying on WPA3 is fine for non-critical
| stuff.
|
| If I need remote access, I use Zrok.io and avoid having to manage
| certs myself.
| PLG88 wrote:
| Love zrok.io, I work on its parent, OpenZiti. It makes me
| wonder; OpenZiti makes PKI much simpler while providing the
| secure overlay, we even used our SDKs to demonstrate zero trust
| overlay networking built into MQTT -
| https://github.com/ekoby/mqziti... could that be useful for
| your use case??
| neilv wrote:
| For one startup, the MVP included deploying some Linux-based
| overseas factory stations, under initially impossible conditions.
|
| For reasons that were really good in context, the software
| aspects of our station builds were based on a set of step-by-step
| written manual instructions, to configure the Linux distro to
| "bootstrapping" state for running the huge configuration shell
| script.
|
| The huge shell script was written to be idempotent. So it could
| do both the initial heavy configuration _and_ then the same or
| later versions of it re-run to make any engineering-change
| adjustments. Including re-run safely remotely, after the machine
| was deployed on the other side of the globe, in an active
| production line. If you 're comfortable with Bash or Python, a
| script like this can be easier than shoehorning the problem into
| the structure of some declarative tool.
|
| (The really good reasons would sound crazy out of context, and
| you wouldn't think it worked, but we did ultimately have perfect
| software uptime. And, yes, there were plans (before the Covid
| chill on our customers and VC) to do a lot of things differently
| in our next-gen station, including putting most of it in a
| container.)
|
| Besides bespoke scripts (Bash, Python, Make), I've also used
| declarative off-the-shelf tools that do a lot of heavy lifting,
| including doing things with Terraform. (For example, Terraform to
| allocate a set of AWS EC2, storage, and networking, to simulate a
| real metal server environment, for developing some infrastructure
| software.) But, even with those tools, there's always some kind
| of documentation of it, and often also bespoke scripts.
|
| Regarding remembering all the steps and getting them right... One
| of the tricks to documentation for things like "pets" servers is
| to have a canonical internal wiki into which pretty much all
| useful info that's not otherwise in Git goes. For example, if you
| go to the AWS Console to add or change something, have the right
| wiki page open, and update it, including copy&pastes, as you go.
| (Also, cross-link that with any issue-tracking for the task.) If
| you keep wiki access low-friction and high-value, it should save
| you at least as much time as it costs, and be even more valuable
| to others. (Put loosely, I've seen situations in which a project
| or a person's job hinges on whether or not one key sentence was
| captured in a wiki.)
|
| Sometimes, this documentation can later be turned into a script.
|
| Following these lightweight doc conventions, even my personal
| laptop has a wiki page on how to reconstruct its configuration,
| and it's kept up-to-date for years.
| sarcasmatwork wrote:
| Always had ansible playbooks run against the server..
|
| Here is a basic example:
|
| https://www.digitalocean.com/community/tutorials/how-to-use-...
| godelski wrote:
| This is probably an unpopular opinion, but I have a bunch of
| install scripts that install some programs from source. Even some
| basic things like vim. The reason being that there's some
| customization I want to do. A lot is that I often can't trust the
| package manger. For simple things like idk if I'll get python3
| (or even python) support in vim to the fact that Ubuntu 20 had
| fd-find and batcat while Ubuntu 22 has fd and bat. The other side
| of this is that I don't always have full control over the machine
| so I'll just install things into
| `"${HOME%/}"/.local/{bin,builds,include,lib,share}`. There's
| rarely "one-size-fits-all" solutions, but if we know what
| decisions we will want to make, we can leverage that.
|
| This does also end up having a multiplying effect where if I want
| to use ansible I can call these scripts directly (which I find
| often easier than using ansible itself...). Then usually I can
| have the system set itself up, or at least 90% of the way and
| while it doesn't save wall time, it saves _my_ time. I can also
| make specific options for the distro at hand or when I have
| certain constraints (it is easy to probe for quickly and that can
| be put in a common file that can be sourced by other scripts).
| The scripting method is helpful because you 're just doing the
| same design pattern as when programming: containerizing
| functions, creating flexibility, modularity, and readability (I
| also highly suggest putting notes in these scripts. Not for
| others, for you. The more you automate the quicker you'll forget
| the awesome tricks you found, but you'll be more likely to
| remember where to revisit). Because like you said "it's very
| repetitive." When you see that, then you know there's a great
| opportunity to leverage your programming skills.
|
| I purposefully try to make these scripts require minimal tooling.
| My main ones are `curl`, `grep`, and `sed`. So I can generally
| rely on having those on a fresh system. This is really all you
| need (though you should use these to grab things like `make`).
|
| Pro tip: make a template maker. I have one for a github source.
| While I can normally just source my common file, there is some
| benefit for having everything self contained. You won't be able
| to write the whole script this way but you definitely can get all
| the boiler plate out of the way which is at least 80% of it, and
| hey, maybe you could get an LLM to do another 10% for you. Though
| it I haven't found one that's really that good at bash. (I
| suspect that this is primarily due to the metric ton of shit bash
| scripts, since the average one is beyond terrible). Idk why bash
| scripting is a "lost art" but it is not that hard (for the love
| of god, use functions).
|
| I also suggest writing some systemd and cron skeletons. These can
| save a lot of time and really help as you find mistakes or if you
| want to add extra system hardening. I do find that common
| implementations do not have the containerization I want (since
| you're mentioning hardening). You can't always trust the
| ArchWiki, but it is usually mostly right. An example might be
| with Fail2Ban, where I don't like to put logs in
| /var/log/fail2ban/ instead of /var/log/fail2ban.log{,.{1..N}}
|
| I'd share but I don't want to dox myself here but I'm happy to
| share some bash tips or other quick hacks.
|
| (to be clear, not everything is or should be installed from
| source. You don't have infinite time and I don't used Gentoo).
| kelnos wrote:
| > _A lot is that I often can 't trust the package manger._
|
| If you can't trust the package manager, how can you trust the
| install media?
| godelski wrote:
| I think you're misunderstanding. Read the sentence after the
| quoted one for clarification.
| thorvaldsson wrote:
| I use Ansible to setup my servers, and also all my workstations.
|
| The first step is to run a bootstrap script, then run a tailor
| made playbook for each situation.
|
| Those specific playbooks haven't been made public, but I wrote
| something[0] last year about how to setup an Android development
| environment using Ansible, and as part of that shared my
| bootstrap script.
|
| [0]: https://hth.is/2023/01/02/android-ansible/
| kelnos wrote:
| I recently wrote something like this, as I was replacing my home
| media center/server box with new hardware, and wanted to start
| fresh instead of imaging the drive and copying it over.
|
| At first I considered that it would be a good opportunity to
| learn something like Ansible, but after looking at the getting
| started docs, I realized I didn't feel like taking the time. So,
| a bash script!
|
| I wrote a script that assumed I'd just installed a Debian base
| image. The script installs a couple needed things with apt-get,
| adds some extra config to /etc/apt, and then does a full upgrade,
| as well as installing other needed packages.
|
| Next it creates some daemon users & groups, copies in a file
| system overlay (mostly config files), and then has a list of
| things to download and install to /opt (stuff that isn't
| available from a Debian repository). There are a few things I run
| on it that come in Docker containers, so those are set up (with
| systemd service files) as well.
|
| Finally it installs duplicity and restores app data that gets
| backed up nightly to S3 on my original media server box, and sets
| things up so those backups will happen on the new box as well.
|
| Ultimately I'm not thrilled with it: some of it is fragile (like
| 'sed' that changes existing config files), and I of course made
| mistakes as I was writing it, so I had to also take care that the
| script was idempotent and could handle being run multiple times
| on the same box without erroring or re-doing what had already
| been done. I imagine/assume something like Ansible (or Chef or
| Puppet or whatever) would handle this for me.
|
| But it worked, and was fairly low-effort compared to learning a
| new provisioning system, so it's fine. Maybe I'll learn Ansible
| another day.
| chrisgoman wrote:
| Strongly do NOT recommend what we are doing
|
| - https://github.com/chrisgo/fabric-example
|
| - https://github.com/chrisgo/fabric2-example
| r14c wrote:
| nixos-anywhere and some flakes is all i need! sometimes for new
| setups I'll run nixos-anywhere and write the config directly on
| the machine, but that's usually just for sketching out ideas.
| I'll reprovision with a module after I've worked out the kinks.
| doublerabbit wrote:
| Not ready yet, but a TCL powered agent that manages *BSD jails.
|
| I currently have a working design of my host, which has a jail,
| that hosts a jail within that jail which then hosts another jail
| which then hosts two more jails of dns and www.
|
| or, otherwise:
|
| Host -> Infrastructure -> NetServices -> Netbox which hosts DNS
| and WWW
|
| This agent sits on all jails and reports back stats such as
| networking with auto throttling. Because jails are just a zfs
| zvol and config away it's has the ability to create too.
|
| All sitting cosy within a TK GUI. On the roadmap is to plug it in
| to NaviServer and with essence of vanilla JavaScript turn it in
| to a web frontend.
___________________________________________________________________
(page generated 2024-10-30 23:01 UTC)