Is it possible to automate OS upgrades? reddit / freebsd ---------------------------------------------------------------------- TheNinjaGeek · 2021-01-29 14:01 I'm looking for possible ways to automate the OS upgrades for FreeBSD, whether it be minor or major revisions.Basically, I help manage +100 FreeBSD servers and last year it took us almost the entire year to upgrade to 11.4-RELEASE, this includes planning and scheduling change control with in the allow change windows.Now we have until September to do it all over again to get them all up to 12.2-RELEASE. I've been testing using Ansible to make it easier for the engineers with a command simular to the following: - name: FreeBSD freebsd-update fetch and install shell: freebsd-update --not-running-from-cron upgrade -r 12.2-RELEASE; freebsd-update --not-running-from-cron install ​ Any assistance would be greatly appreciated, even if it's just saying that it can't be done. ---------------------------------------------------------------------- daemonpenguin · 2021-01-29 14:26 I guess my first question would be: where is the bottleneck? You said it took almost an entire year to upgrade 100 or so servers. Why? I mean was it because someone felt each server had to be done one at a time, were there problems, were the configurations so different each one had its own unique problems, does each server run a different application that had to be tested? Knowing what caused things to go so slowly would help a lot. I mean, if the configs are mostly the same in theory you could upgrade all the servers remotely at once (after proper testing and backups were made). ---------------------------------------------------------------------- TheNinjaGeek · 2021-01-29 14:42 The main reasons for it taking such a long time is that although the config is pretty much the same across the servers, depending on their function, different groups of servers belonged to different customers. Essentially with a small team of engineers, roughly 3 to 4 of them, they would only be able to perform upgrades for a single a customer each day, within the allowed change window and with as little down time as possible. Some of the upgrades would then have to be scheduled over more than a single change window. Due to the OS upgrades being manual and very time consuming, I've been tasked to try and automate the process more, which would mean we could possibly do more upgrades at a time, or so it's believed, if all they need to do is click a button (so to speak) and check up on it incase something goes wrong. ---------------------------------------------------------------------- seanrees · 2021-01-29 15:31 Have you considered this from a turnup angle vs the upgrade angle? In such a setup you would achieve an "upgrade" by reinstalling the OS to a newer version. One approach would be a netboot to an installer. Combining the paths would allow you to reuse the infrastructure for both use cases, which would help out with machine upgrades, adds, or replacements. ---------------------------------------------------------------------- daemonpenguin · 2021-01-29 16:12 I guess I'm still not clear on why your team is doing things the way they are. Why can the team only upgrade one customer a day? Why does the team think the upgrade is a lengthy process when it should only take a couple of minutes? I often upgrade three or more servers at a time using remote access tools and the whole process, including making snapshots and downloading the patches, is probably less than 20 minutes. Even only doing 10 at a time your team should be able to blow through 100 in five hours, spread out over your change windows. That's without any automation so what is holding them back? At a minimum you could probably set up a script that basically does "create snapshot, download next version, send e-mail asking admin to manually perform upgrade/patch". Then all they need to do is confirm the new config files look good, reboot, and run pkg upgrade. ---------------------------------------------------------------------- FUZxxl · 2021-01-29 16:24 Please don't use `--not-running-from-cron` when batch-updating servers. The entire point of the default behaviour is to avoid overloading the FreeBSD servers when such batch updates occur. ---------------------------------------------------------------------- TheNinjaGeek · 2021-01-29 19:33 Thanks for the heads up, although we push the updates through a proxy which caches them, I probably need to just increase the cache so it doesn't try and hit them too hard. ---------------------------------------------------------------------- FUZxxl · 2021-01-29 19:35 Ah yes, if you have a proxy then it should be fine. ---------------------------------------------------------------------- TheNinjaGeek · 2021-01-29 19:55 One thing I probably forgot to mention is that 90% of the servers are physical tin and due to the current architecture the main servers at customers are not able to be rebuilt so easily, mostly because some of the main components can not use shared storage for hosting the stored data and due to the high I/O nature of the systems, using jails or switching to poorly managed vms, on the client's side, is not an option. One of the reasons we do one customer at a time, is mostly due to the fact that most customers have multiple servers and to prevent burn out, we try and only schedule a customer at a time, with an allowed change window of only 2 hours and depending on the amount of servers that need to be upgraded, each engineer will tackle a set amount of servers during that time and it needs to be done in a staggered approach to have a minimal amount of downtime, taking into account that sometimes we need to do a postgres upgrade as well and we need to re-install, using pkg-static upgrade -fy, of all packages on the system (thank god for poudriere so that I can maintain the packages). I understand that this is probably the incorrect way of doing things and things will hopefully improve over time, but this is basically the reason for my post, to get ideas on ways to improve stuff and do it the right way. With regards to why things take so long, we are in a 3rd world country, not every server has gigabit or higher connectivity to be able to download patches, OS upgrades and packages in a short amount of time and as the servers are already sometimes under high load, by design I might add, merges and checks that freebsd-update performs can take some time. What I've put forward so far to the team is to perform the upgrade fetch, freebsd-update upgrade -r , before hand as they have always done to save time, and then we can use the automation tools available to complete the rest of the upgrade process. ​ TL/DR: * Physical tin, can't do stuff that are done with jails and vms. * Don't always have fast internet for downloads, but using a proxy for most to speed it up. * Instructed team to rather do the fetch before hand as it doesn't change the OS. * Use automation to complete the upgrade during change window. ​ I thank everyone who has given there opinion so far and I'm always open to new suggestions, especially since I feel the way we've been doing things for the last few years need to change. Another question would probably be, how do people manage hundreds of freebsd servers? ---------------------------------------------------------------------- pacmanlives · 2021-01-29 20:02 I would add an Ansible when statement for an OS version check to see if it even needs to be applied. The best part of Ansible is its item potent and I probably switch out to command instead of shell ---------------------------------------------------------------------- unitrunker2 · 2021-01-29 20:57 I like your comment "do the fetch before hand" will likely help since that can be done outside the change window. ---------------------------------------------------------------------- TheNinjaGeek · 2021-01-29 21:15 I've actually added this using when:    \- freeBSD\_version != ansible\_distribution\_version along with other checks of course ---------------------------------------------------------------------- whackamaul · 2021-01-30 08:58 RemindMe! 36 hours "check for updates" ---------------------------------------------------------------------- RemindMeBot · 2021-01-30 08:58 I will be messaging you in 1 day on [**2021-01-31 20:58:01 UTC**](http ://www.wolframalpha.com/input/?i=2021-01- 31%2020:58:01%20UTC%20To%20Local%20Time) to remind you of [**this link **](https://np.reddit.com/r/freebsd/comments/l7t7zz/is_it_possible_to_ automate_os_upgrades/glcnylo/?context=3) [**CLICK THIS LINK**](https://np.reddit.com/message/compose/?to=Remind MeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2Ff reebsd%2Fcomments%2Fl7t7zz%2Fis_it_possible_to_automate_os_upgrades%2F glcnylo%2F%5D%0A%0ARemindMe%21%202021-01-31%2020%3A58%3A01%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)] (https://np.reddit.com/message/compose/?to=RemindMeBot&subject=Delete% 20Comment&message=Delete%21%20l7t7zz) ***** |[^(Info)](https://np.reddit.com/r/RemindMeBot/comments/e1bko7/remindm ebot_info_v21/)|[^(Custom)](https://np.reddit.com/message/compose/?to= RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside%2 0square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Your Reminders)](https://np.reddit.com/message/compose/?to=RemindMeBot&subj ect=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](https: //np.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot%20Fe edback)| |-|-|-|-| ---------------------------------------------------------------------- vermaden · 2021-01-30 18:25 You commands are: # freebsd-update --not-running-from-cron upgrade -r 12.2-RELEASE # freebsd-update --not-running-from-cron install This will fetch the files/patches needed to upgrade your system and update/install only new kernel. The full procedure is: # freebsd-update --not-running-from-cron upgrade -r 12.2-RELEASE # freebsd-update install # shutdown -r now // now you are booting 12.2 kernel with 12.1 userspace # freebsd-update install // now 12.2 userspace is installed # pkg upg # freebsd-update install # shutdown -r now The second 'install' installs the world or userspace (same as `make installworld`). There is also sometimes third 'install' that will remove unneeded files. You need two reboots for full upgrade process. Hope that helps. ---------------------------------------------------------------------- grahamperrin · 2021-01-31 16:43 If there's manual intervention – e.g. requirements to use `vi` and/or respond **y** or **n** with regard to things appearing reasonable – then it's not entirely automated. I might make the same observation re: some of the other commentary; I'm not sure what degree of automation is required (and I don't know Ansible, and so on). ---------------------------------------------------------------------- grahamperrin · 2021-01-31 17:02 > … Another question would probably be, how do people manage hundreds of freebsd servers? Thought-provoking https://markmail.org/message/ze2hzr2r34hapqwv NFS as part of a solution. Like, build once, mount everywhere. I don't imagine this being suitable for your (geographically dispersed?) team, but it did fall under the heading of *Updating multiple computers*. ---------------------------------------------------------------------- grahamperrin · 2021-01-31 17:11 ## PkgBase ### Officially https://lists.freebsd.org/pipermail/freebsd-pkgbase/ https://wiki.freebsd.org/PkgBase * maybe quite outdated (it mentions a 2016 CFT but not [a 2019 call](https://lists.freebsd.org/pipermail/freebsd- stable/2019-April/090963.html), and so on) ### Unofficially https://alpha.pkgbase.live/ Quoting the creator: > … i think the main advantage of PkgBase is the speed of which updates can be installed on a system. > > … > > i do indeed like PkgBase for the space compactness and the speed of updates, … ---------------------------------------------------------------------- vermaden · 2021-02-01 09:16 These should help: - https://serverfault.com/questions/508334/how-do-i-non-interactively- update-freebsd-system-and-ports - https://forums.freebsd.org/threads/can-freebsd-update-upgrade-be- run-non-interactively.57752/ - https://forums.freebsd.org/threads/non-interactively-update-freebsd- system.47583/