Post AbcIZVBcyzmrtsoP5s by BoydStephenSmithJr@hachyderm.io
(DIR) More posts by BoydStephenSmithJr@hachyderm.io
(DIR) Post #AbcH5v9rkybroAJp2m by mjg59@nondeterministic.computer
2023-11-09T02:39:12Z
1 likes, 2 repeats
Today I got to tell my students that in the bad old days we used to write scripts that just SSHed into every machine and ran sed on config files but today we had puppet and I regret to inform you that based on their facial expressions we apparently still write scripts that just SSH into every machine and run sed on config files
(DIR) Post #AbcHPxDLozN7ydV8uO by otakup0pe@sfba.social
2023-11-09T02:42:57Z
0 likes, 0 repeats
@mjg59 For my latest project, I wrote a bunch of shell scripts that SSH into the remote hosts and run shell scripts and sed on configuration files. Feels good. Liberating.Don't get me wrong, I still love ansible. But there's just something... Je ne sais quoi....
(DIR) Post #AbcHa5hX7ef5C6S78q by baljemmett@mastodon.online
2023-11-09T02:43:59Z
0 likes, 0 repeats
@mjg59 Ah, but, now we have that *as a service*. So it must be better, right?
(DIR) Post #AbcHuuSc6soiPgOvce by unixbhaskar@mastodon.social
2023-11-09T02:48:42Z
0 likes, 0 repeats
@mjg59 :)
(DIR) Post #AbcI4vCSDRsVbOcI9Q by fivetonsflax@tilde.zone
2023-11-09T02:49:36Z
0 likes, 0 repeats
@mjg59 people love to hate on Puppet, but I have seen it used to do remarkable things across a fleet larger than any the haters have managed
(DIR) Post #AbcIFBPConyil7TSKG by soleblaze@infosec.exchange
2023-11-09T02:49:41Z
0 likes, 0 repeats
@mjg59 ah yes. The good old days when I nuked everyone's fstab file by putting '>' instead of '>>' in a script. Luckily I was able to reconstruct all of them with mtab.. after I calmed down.
(DIR) Post #AbcIONiOQC0hXgppMu by Anarcat@kolektiva.social
2023-11-09T02:49:46Z
0 likes, 0 repeats
@mjg59 now come on, we're in the future, i write Python code that uploads itself onto remote servers and runs shell scripts that sed config files, please
(DIR) Post #AbcIZVBcyzmrtsoP5s by BoydStephenSmithJr@hachyderm.io
2023-11-09T02:51:34Z
0 likes, 0 repeats
@mjg59 The automation engineer on our team has provided me an ansible script which does a lot, but not everything so I follow that with an ssh+sed script. :)
(DIR) Post #AbcIiu6TiB4nmjt1VY by mjg59@nondeterministic.computer
2023-11-09T02:57:05Z
0 likes, 0 repeats
@fivetonsflax I could enable security config on over 100,000 systems with reasonable confidence it would work and even 20 years ago I would not have thought that was a good idea with SSH
(DIR) Post #AbcIyuUu0y2hdkwXIm by swelljoe@mas.to
2023-11-09T03:00:11Z
0 likes, 0 repeats
@mjg59 after having used puppet pretty heavily for a couple years, I think I'd prefer ssh+shell scripts. (I mostly don't use ssh+scripts, I mostly use config packages distributed via apt for stuff that's consistent across all devices of a type and a custom config service for the stuff that isn't. Puppet was just too heavy and fragile for our use case. Also complicated. Our field ops folks had a lot of fights with it, because they didn't understand what it controlled and how to change it.)
(DIR) Post #AbcJAH5mLdqQtzbvWK by mjg59@nondeterministic.computer
2023-11-09T03:02:05Z
0 likes, 0 repeats
@swelljoe I think this is an argument against the idea that DevOps is a way to get rid of sysadmins because you still very much need people who understand this shit works
(DIR) Post #AbcJLJUZlW6JWyTO8O by adx@hachyderm.io
2023-11-09T03:03:33Z
0 likes, 0 repeats
@mjg59 I had a finally tuned cfengine setup that used rsh.
(DIR) Post #AbcK5Ff0LOfsy3Fvn6 by swelljoe@mas.to
2023-11-09T03:12:53Z
0 likes, 0 repeats
@mjg59 I believe Google invented "DevOps" because they didn't know how to hire good sysadmins but knew how to hire software engineers back in the beginning. And, the dirty secret in DevOps is that any DevOps engineer over a certain age is almost certainly also a sysadmin. Perl/shell scripting skills translate pretty easily to Python. And, config files is config files, even if they're in yaml or json.
(DIR) Post #AbcKFRWh57YAfIEDa4 by paul_ipv6@infosec.exchange
2023-11-09T03:13:54Z
0 likes, 0 repeats
@mjg59 i remember when i thought that expect and tcl/tk were just the $#!t...of course, i'd hoped we would stop using rancid on routers before the start of this millenium and found rancid running less than 5 years ago on a large network...
(DIR) Post #AbcMQuoUjVKOai9Wts by wollman@mastodon.social
2023-11-09T03:39:10Z
0 likes, 0 repeats
@mjg59 @fivetonsflax We use puppet *and* ansible *and* shell scripts. (The installer is a dodgy shell script, ansible is used for provisioning special-purpose systems after they get installed, and puppet is used for site-wide policy like users, logging, filesystems, and data-driven web server config.)
(DIR) Post #AbcOGjrgmg1reCvE8W by djmitche@mastodon.social
2023-11-09T03:59:49Z
0 likes, 0 repeats
@mjg59 and let's be honest, Puppet isn't all that much better.
(DIR) Post #AbcVtmHcQUxG7Wc3Ie by SadKitten@mastodon.social
2023-11-09T05:23:56Z
0 likes, 0 repeats
@mjg59 we are calling it Rundeck and ansible now but it's still ssh and sed
(DIR) Post #AbcZJyxKERNgodJ7om by purpleidea@mastodon.social
2023-11-09T06:03:42Z
0 likes, 0 repeats
@mjg59 I've been trying to build a new mechanism for doing this, patches, support, etc, are welcome!https://github.com/purpleidea/mgmt/
(DIR) Post #Abcb2Y4X8DuWQGUfpo by sushee@fosstodon.org
2023-11-09T06:22:54Z
0 likes, 0 repeats
@mjg59 pdsh and sed to do stuff ON the computer - curl and jq to get stuff FROM the computer. and your ci/cd is shell scripts in a fancy workflow trenchcoat.
(DIR) Post #Abcnd9ywrMghsC6bSK by sophie@mastodon.catgirl.cloud
2023-11-09T08:44:00Z
0 likes, 0 repeats
@mjg59 Well we also have NixOS now. That way we can SSH into our machines and execute a Perl script to symlink generated config files into place.
(DIR) Post #Abcnn8AKT4zp8SBPzk by kkarhan@mstdn.social
2023-11-09T08:44:06Z
0 likes, 0 repeats
@mjg59 isn't #Puppet doing exactly that?Basically all #Orchestration systems that don't rely on a client to regularly pull configs does that...
(DIR) Post #AbcpVkRvagcUZBFo7k by GabrielKerneis@oc.todon.fr
2023-11-09T09:04:40Z
0 likes, 0 repeats
@mjg59 During my first oncall rotation at Google (9 years ago), I learned how to roll out a release to N datacenters by opening N terminals and running `borgcfg` in each of them — carefully picking the order manually, based on the worldmap, so that I didn't push to two continents at the same time. (Yes, one of my first proposals to the team was to setup some kind of automation.)
(DIR) Post #AbcxMkBxygbAnM0PDs by spacehobo@teh.entar.net
2023-11-09T10:32:23Z
0 likes, 0 repeats
@mjg59 All of my "Teach SysAdmins To Use Puppet, Not Just How To Use It" materials started with a careful viewing and inspection of the Sorcerer's Apprentice segment of Disney's Fantasia.
(DIR) Post #Abd5nerOhz8Vi8jGpE by cigitalgem@sigmoid.social
2023-11-09T12:07:10Z
0 likes, 0 repeats
@mjg59 ssh is new
(DIR) Post #Abd7ea5FIYlQCN0R6m by markwalker@fosstodon.org
2023-11-09T12:27:43Z
0 likes, 0 repeats
@mjg59 not that long ago really. The days of local settings on servers so secrets weren't in VCS. When you'd need to create a new AMI after making changes so the system would still scale.I do not miss those days!
(DIR) Post #Abd9TPwypj3LUewcVs by olbohlen@norden.social
2023-11-09T12:48:32Z
0 likes, 0 repeats
@mjg59 but we embed the shell script in ruby!
(DIR) Post #AbdGI1eVX12wd1oqTA by dalias@hachyderm.io
2023-11-09T14:03:44Z
0 likes, 0 repeats
@mjg59 @fivetonsflax One person or even org having access to do that on 100,000 systems is something severely wrong with the world.
(DIR) Post #AbdKFvctMgf1B0Jpcu by grumpybozo@toad.social
2023-11-09T14:49:09Z
0 likes, 0 repeats
@mjg59 Hey, yeah, I remember truncating /etc/passwd on 221 Solaris boxes in 30 seconds with one of those scripts… It was not intentional.
(DIR) Post #AbdRe5gcJRzL0SWuAq by malin@dice.camp
2023-11-09T16:10:57Z
0 likes, 0 repeats
@mjg59 if the scripts are idempotent and get tested, isn't that okay?
(DIR) Post #AbdSc3bXBN5pvMkLRI by mjg59@nondeterministic.computer
2023-11-09T16:23:02Z
0 likes, 0 repeats
@malin does it deal correctly with machine-specific config? What happens if a machine is down when you run it? What happens if someone manually reverts the change?
(DIR) Post #AbdTn0YkwXMHcXEUYi by bassplayer@mas.to
2023-11-09T16:35:43Z
0 likes, 0 repeats
@mjg59 I remember a conversation at work where someone said: the best part of ansible is it's not puppet or chef 😀
(DIR) Post #AbdUmDLrzqyOxXOHqq by malin@dice.camp
2023-11-09T16:46:52Z
0 likes, 0 repeats
@mjg59 idk, more bash happens probably.
(DIR) Post #AbddrlbuwwXwFLcSki by dentaku@fnordon.de
2023-11-09T18:28:59Z
0 likes, 0 repeats
@mjg59 @mutax I deploy a container as a DaemonSet to k8s that mounts the host filesystem and runs sed on config files.
(DIR) Post #Abl8PSzBm0mvGP43Lk by TerrorBite@meow.social
2023-11-13T09:13:57Z
0 likes, 0 repeats
@mjg59 Ansible is just SSHing into well-organised things and running sed on config files with an added check to see if you've already done it