How do you keep your services up to date? reddit / selfhosted ---------------------------------------------------------------------- Theon · 2023-10-08 17:55 I regularly see people here running literal dozens of services, and there's no way you'd go around and update them manually one by one. On the other hand, keeping them at whatever version... doesn't seem like a good idea, to say the least. I use [watchtower](https://containrrr.dev/watchtower/) but I'm not too happy with it - not all docker repos have semver, and pinning to `latest` can also bring unpleasant surprises. Is there some kind of update dashboard or update scanner, that I'd give a list of URLs (or container names) and it'd give me an overview of current and latest versions, or ping me when there's an update? Or do you all use some different approach entirely? Thanks! ---------------------------------------------------------------------- CrispyBegs · 2023-10-08 18:07 i use watchtower *and* I let it auto-update stuff. on the rare occasions it messes something up (only twice so far) i treat it as an opporunity to learn how to fix things. ---------------------------------------------------------------------- ben-ba · 2023-10-08 18:20 Diun with mail notifications ---------------------------------------------------------------------- ElevenNotes · 2023-10-08 18:23 You have to do it manually because you should read the changelog of every major and even minor version due to changes that might break your current system (your integration of that app). Otherwise you will some day find out the hard way. ---------------------------------------------------------------------- Special-Swordfish · 2023-10-08 18:24 Keel(.sh) works a charm on kubernetes. And on the rare occasion an upgrade breaks something, it is easily restored using a yaml definition from a kube-dump cronjob. ---------------------------------------------------------------------- fenty17 · 2023-10-08 18:29 Yeah I’m still manual too. Reading changelogs is also sensible in case there are new features you might miss during normal operation of your apps. Not tried watchtower but just don’t like the idea of important things breaking without my knowledge/input. ---------------------------------------------------------------------- clintkev251 · 2023-10-08 18:33 I use Renovate which has been working really great for me. Opens up pull requests against my git repo and I can review and merge them as I see fit. Combined with FluxCD, it means that all I need to do is spend a few minutes every few days reviewing PRs and once I merge them, changes are automatically mirrored to my cluster ---------------------------------------------------------------------- richneptune · 2023-10-08 18:36 Unsure why you've been downvoted. Not all of us are running mission critical services where we're trying to get 99.99999% uptime, things unexpectedly breaking and learning to fix the resultant issues is absolutely fine. ---------------------------------------------------------------------- EndlessHiway · 2023-10-08 18:37 I don't. Took me forever to get things running. Never touching anything again. /s ---------------------------------------------------------------------- lilolalu · 2023-10-08 18:44 On my essential containers I have set watchtower to only send notifications over matrix-chat when new updates exist but don't auto update. On the not so important containers I let it do auto updates. ---------------------------------------------------------------------- WiseCookie69 · 2023-10-08 18:47 ArgoCD, Git, Renovate ---------------------------------------------------------------------- Nagashitw · 2023-10-08 18:58 I second renovate. Have the same setup and love it. ---------------------------------------------------------------------- lesigh · 2023-10-08 19:00 Or you can keep daily backups and auto update and pray things don't break 😅 ---------------------------------------------------------------------- lesigh · 2023-10-08 19:00 This. Ain't no one got time to manually update 30+ containers ---------------------------------------------------------------------- haaiiychii · 2023-10-08 19:10 This is what I've done. Some stuff that I'd rather do manually I just left the watchtower tag so it leaves it alone. Has worked well for me for an couple of years now. ---------------------------------------------------------------------- CrispyBegs · 2023-10-08 19:21 That’s it. It’s just tinkering for me, nothing life-threatening. lol @ being downvoted for that. Sorry everyone! ---------------------------------------------------------------------- FreebirdLegend07 · 2023-10-08 19:23 Or if it's in a deployment it'll just auto revert back to the last successful deployment. Keel is amazing I just need a way to handle the webhooks better than using ntfy. ---------------------------------------------------------------------- adamshand · 2023-10-08 19:24 I get a weekly email from newreleases.io. If there are any updates I want or security issues, I'll update that day. Otherwise, I update everything manually once a month or so. Tip for anyone interested in newreleases.io, if you use Github to track each project (instead of Docker Hub etc) then you get the release notes for each update in the email. ---------------------------------------------------------------------- Wartz · 2023-10-08 19:27 I learned Ansible and some python/shell scripting. ---------------------------------------------------------------------- code-lover · 2023-10-08 19:34 You might also want to take a look at this interesting project: [https://github.com/fmartinou/whats-up- docker](https://github.com/fmartinou/whats-up-docker) It has auto-update feature like WatchTower. But looks like the manual update feature, through its web UI, is currently in progress which can be a great help for doing the major updates. ---------------------------------------------------------------------- Spuxilet · 2023-10-08 19:54 Do not ever update containers automatically. To search for updates i use whatsupdocker it is a great service. You can write regex to check exactly that image what you like. It's configured using labels so it's very easy to setup. Mine looks like this https://imgur.com/mSljIFS ---------------------------------------------------------------------- MegaVolti · 2023-10-08 20:13 I use btrfs, if an update goes wrong I can simply roll back. Which means automated updates have essentially no downside - well except for some potential downtime till I roll back, but since updates going wrong are extraordinarily rare, the time invested in rolling an update back is negligible compared to the time it would take me to do all updates manually. As for those automated updates, I used to use Watchtower, but I found it to be ultimately pointless. A simple systemd (or cron) script calling `docker compose pull` and `docker compose up -d` is sufficient and achieves the same thing. I have that wrapped in a small shell script that simply emails me the output, it runs once a week. ---------------------------------------------------------------------- phillibl · 2023-10-08 20:18 I set WatchTower to notify only, no reason for it to auto update ---------------------------------------------------------------------- mimikater · 2023-10-08 20:30 Docker compose pull, docker compose up. ---------------------------------------------------------------------- opensrcdev · 2023-10-08 20:39 I've started using Docker Compose to handle this. * Define all services, volumes, ports in a **docker-compose.yml** file * Periodically run **docker-compose pull** * Periodically run **docker-compose up --detach** That should keep your container images up-to-date and keep your data in-place during the upgrades. In fact, I'm running this process my VMs today! 🙂 ---------------------------------------------------------------------- smarzzz · 2023-10-08 20:39 I do run mission critical workloads that need 6 nines uptime, and we still auto update; we’ll let canary deployments and livelinesschecks stop the continuation of rollouts. In my homelab, I auto update in a less fancy way; just by updating ---------------------------------------------------------------------- FLeiXiuS · 2023-10-08 20:46 Are you using k8s? ---------------------------------------------------------------------- Feeling-Crew-1478 · 2023-10-08 20:54 I copy and paste lines to update my containers. Easy and not automated. I take a snapshot of the VM first. ---------------------------------------------------------------------- gruffogre · 2023-10-08 21:00 Run proxmox containers and use the script on ttecks page ---------------------------------------------------------------------- prime_1996 · 2023-10-08 21:03 Same here, I actually use Ansible to run apt and docker compose update, works great no issue so far ---------------------------------------------------------------------- prime_1996 · 2023-10-08 21:06 Currently using Ansible to update all my docker nodes with apt, and also the docker compose stacks. Works great. I do have one node setup with watchtower and auto updates, works fine but does on a container level not docker compose stacks level, hence I think the Ansible approach works better for me. ---------------------------------------------------------------------- lmm7425 · 2023-10-08 21:15 I use renovatebot with my K8s cluster and with my docker compose stack. Flux merges my renovate PRs automatically and I have a script that runs on my docker compose host the constantly merges once I approve the PRs in gitea. ---------------------------------------------------------------------- ftrava · 2023-10-08 21:16 It also has telegram notifications which i find super useful because with watchtower i never know when a container is update (while im writing this i am not sure if watchtower has notifications as well…) ---------------------------------------------------------------------- ftrava · 2023-10-08 21:18 Holy shot this is huge, i used to subscribe to rss feeds of release notes of every repo i was interested in ---------------------------------------------------------------------- Anejey · 2023-10-08 21:19 Uhh... as of right now, I don't. I have setup things via LXC, Docker, and whatnot, and I'm just happy that it works right now. Doing updates is a to-do thing. Since most of my services are via Tteck scripts, I'm hoping to create something that updates everything at once, but it's a work in progress. ---------------------------------------------------------------------- lilolalu · 2023-10-08 21:21 Watchtower has both of that features. You can flag containers to skip auto updates and just notify about a new version, also via Telegram if you like that. https://containrrr.dev/watchtower/notifications/ ---------------------------------------------------------------------- ftrava · 2023-10-08 21:27 Do you automatically add new containers to uptime kuma or do you have to manually add them? ---------------------------------------------------------------------- opensrcdev · 2023-10-08 21:35 Right now I manually add HTTP endpoints or IP addresses that I want to monitor. ---------------------------------------------------------------------- Fade_to_Blah · 2023-10-08 21:50 I uh, have dozens of services that I update by hand. Honestly it’s just a click and not a big deal. No way would I blindly allow something to update automatically so I prefer this. Most services are not updated that often so it’s not that much work ---------------------------------------------------------------------- syfr · 2023-10-08 21:51 Kubernetes with image pull policy set to always. Image tag is set to latest so all I have to do is ‘Kubctl rollout restart all deployments -n homelab’ Or ‘Kubectl rollout restart deployment sonarr -n homelab’ It’s all taken care of automatically but obviously this only works with the image tag latest. There are other commands like rollout upgrade where you can specify an image version if you want to pick the image you want to go to ---------------------------------------------------------------------- adamshand · 2023-10-08 21:54 Yeah same! Very convenient! ---------------------------------------------------------------------- Nagashitw · 2023-10-08 22:02 Why not? ---------------------------------------------------------------------- Sekhen · 2023-10-08 22:04 Dozens? Those are rookie numbers. 29 is closer to the truth. The solution is crontab and scripts. Lots of scripts. ---------------------------------------------------------------------- cavilesphoto · 2023-10-08 22:07 https://github.com/mag37/dockcheck Dockcheck. I do it manually and everything (-ish...) dockerized. I used watchtower but I don't like it to autoupdate without my supervision ---------------------------------------------------------------------- LostLetterbox · 2023-10-08 22:18 I don't have too many containers, I'm the only one using them and I'm mostly doing it for learning. Ive been using podman/systemd to run the containers (including using the podman auto update function), for pinned versions I check every now and then... I'm running a single(ish) database and been dragging my heels upgrading to 16 (time has been elsewhere). So far still waiting for my first dumpster fire (it'll be one of those learning experiences I'm after). ---------------------------------------------------------------------- Flux_nzl · 2023-10-08 22:28 I use [Dockcheck Web](https://github.com/Palleri/dockcheck-web) to understand if I need to do any updates. It's just sitting as a bookmark at the bottom of my [Homepage](https://github.com/gethomepage/homepage) dashboard. I'll check it every week or so, and when there are some updates I'll start my Watchtower container with --runonce to do it's thang. Then I spotcheck a few services like Nextcloud and Bookstack (or things running with DBs) to see if they've broken. It's a great compromise between limiting effort and still sitting at my computer in case everything breaks. Then if anything does break I roll back my Docker VM with Proxmox. :D ---------------------------------------------------------------------- ButCaptainThatsMYRum · 2023-10-08 22:30 Checkpoint. Run scripts, keep an eye out for errors. Check services, confirm good or address issues. Remove checkpoint. ---------------------------------------------------------------------- kriibby · 2023-10-08 22:49 Updating containers automatically is fine. It's why tags exist ---------------------------------------------------------------------- Cheeze_It · 2023-10-08 23:07 Depends on your uptime needs... ---------------------------------------------------------------------- schklom · 2023-10-08 23:12 I use `stable` whenever I can instead of `latest`, and then it is a simple cronjob ``` docker compose down --remove-orphans docker volume prune --force docker compose pull --ignore-pull-failures docker compose up -d docker image prune --all --force ``` `docker compose down` is not necessary but I have had issues without it once in a while. The whole script takes around 2 minutes, so it's not like the downtime is large. ---------------------------------------------------------------------- returnofblank · 2023-10-08 23:30 docker compose ---------------------------------------------------------------------- Ejz9 · 2023-10-09 00:22 What’s Up Docker sounds like what you could be looking for. It can be automated like watchtower but I find watchtower is better. Otherwise it’ll show when a service has an update but this can apply to nightly builds etc but that depends what you like. Latest isn’t terrible though and most of the tools I use recognize that updating to latest is the easiest and probably most preferred but I’ve yet to have anything break. If it does anyways, just have to fix it I guess. Nature of how computers and software work just leaves managing computers to be a always hands on job which I’d prefer to automate everything I can and from there manually fix as needed. Nothings perfect so… 😄 ---------------------------------------------------------------------- A2251 · 2023-10-09 00:33 Is there a good writeup on how to set this up ---------------------------------------------------------------------- Mundane-Garbage1003 · 2023-10-09 00:51 Seems like there is a lot of this in some of these spaces with doing things the “right” way unnecessarily. Yes some of things are risky, but people seem to forget that “I understand the risk, and am willing to accept that in exchange for some other benefit (convenience/cost/whatever)” is a perfectly valid stance to take. ---------------------------------------------------------------------- applesoff · 2023-10-09 01:03 You can set watchtower to notify without updating so important stuff doesn't go down. I do that with gluten. No other services require me to monitor their updates ---------------------------------------------------------------------- thekrautboy · 2023-10-09 01:47 Fyi you could do the pull first and then down etc to reduce the downtime. ---------------------------------------------------------------------- Vlinux · 2023-10-09 01:56 I usually just ssh into each of my servers and run "docker-compose pull; docker-compose up -d" since all of my services are defined in docker-compose.yml files. I do have an Ansible playbook which does the same thing that I use sometimes. ---------------------------------------------------------------------- kilaire · 2023-10-09 02:20 No, it doesn’t. Pinning is good, but you should autoupdate and roll those changes out programmatically. And since you have high uptime needs, your automated testing should catch anything huge. ---------------------------------------------------------------------- schklom · 2023-10-09 02:23 Good point, I'll do that from now on. Thanks :) ---------------------------------------------------------------------- kilaire · 2023-10-09 02:29 Hardcore “old school” operations believe that the safest way is to research each update and choose to upgrade (or not) based on the upgrade. They’re not wrong…but also, an unpatched system running out of date software is about as bad as you can get. And manually reading everything takes time and energy. It’s generally better to have tests for the important things, and update frequently (and automatically) in a visible way…and make sure things don’t break in an integration / non-prod environment. But…we’re talking home lab here in most cases - breaking is a nuisance. We’re not running hospitals or airlines. So yolo, update often, and fix it when it breaks. ---------------------------------------------------------------------- GWBrooks · 2023-10-09 02:32 Preach. ---------------------------------------------------------------------- kilaire · 2023-10-09 02:33 Latest tag is not always a good idea. A lot of CI scripts will tag something as latest when it’s the latest build…of an old version of the software. And if you don’t use image pull always (I know you said you are) - there’s a good chance you’ll hit inconsistent cache issues that are just hard to troubleshoot. /shrug But for homelab…this is probably just fine! ---------------------------------------------------------------------- thekrautboy · 2023-10-09 02:50 Just for Docker images: [diun](https://github.com/crazy-max/diun) on each Docker host, notifications for available updates through Pushover/ntfy/gotify. Then depending on the service, i check the release notes of the update. When i actually want to run a update, i either do it manually for just that one service (dc pull && dc down && dc up -d). Or i use the nice [dockcheck script](https://github.com/mag37/dockcheck) (dockcheck.sh -a -p -s) to update every container on a host. ---------------------------------------------------------------------- Cheeze_It · 2023-10-09 02:53 I need 100% uptime. I also (currently) do not have the infrastructure to make sure stuff doesn't go down, I am begrudgingly forced to not update in an automated fashion. Things have to 100% work. There can be nothing that breaks unfortunately. ---------------------------------------------------------------------- kilaire · 2023-10-09 02:57 That sounds like a really bad situation to be in. I’m sorry. ---------------------------------------------------------------------- Cheeze_It · 2023-10-09 03:01 Such is life my good sir/madam. There's times where uptime is king, even if it's a little less than optimal. ---------------------------------------------------------------------- instant_dreams · 2023-10-09 03:32 Diun with Telegram notifications ---------------------------------------------------------------------- javellin · 2023-10-09 04:13 I like ouroboros. Sends me a push notification when a container gets updated. ---------------------------------------------------------------------- acid_etched · 2023-10-09 04:18 Whenever unraid tells me there’s a new one I click update. Hopefully I don’t break anything this way but it’s not the end of the world if I do. ---------------------------------------------------------------------- FishSpoof · 2023-10-09 04:24 For things that only run in my internal network, I just update when the need arises. Maybe there's a new feature I want or a bug has been fixed. Most externally accessed services I do manually once a week or two. ---------------------------------------------------------------------- Charming-Molasses-22 · 2023-10-09 04:24 This is the way. ---------------------------------------------------------------------- FishSpoof · 2023-10-09 04:25 docker compose pull Done! ---------------------------------------------------------------------- syfr · 2023-10-09 04:57 Exactly my point, it works for me as I have the technical know how to fix a problem when it comes up, or to revert > But for homelab…this is probably just fine! Yep business critical we dont do that policy for exactly the reasons you stated above. ---------------------------------------------------------------------- ftrava · 2023-10-09 04:59 Why don’t you use docker monitor? ---------------------------------------------------------------------- bufandatl · 2023-10-09 05:04 I have an Ansible playbook that runs in a CI/CD pipeline once a week. First updates the hosts then checks if a reboot is required and if so reboots. If only system services need a restart those are restarted. And then it runs the usual playbook I use to deploy it just has all extra_vars set that allow to pull the latest images of the containers I run. ---------------------------------------------------------------------- veyper · 2023-10-09 05:21 I'm primarily built utilizing docker compose on individual LXCs hosted in proxmox. I am currently using watchtower to email me when updates are available, then rely on Ansible playbook(s) for updating depending upon the use-case. I've been less than impressed with watchtower ux in this setup, but it is working and pretty much flawlessly thus far. ---------------------------------------------------------------------- servergeek82 · 2023-10-09 05:25 Rookie numbers.... 42 after removing a few ---------------------------------------------------------------------- servergeek82 · 2023-10-09 05:29 Gitea actions that runs weekly. But have it written in Jenkins and ansible too ---------------------------------------------------------------------- xardoniak · 2023-10-09 05:39 I previously used Watchtower but when rebuilding my infrastructure I moved to GitOps stacks (compose files) in Portainer. I version tag my compose files in GitHub, which Portainer checks for changes every 5 minutes. The Renovate bot watches the compose files and creates a pull request with the updated version number. I can review the pull request for information about the change and approve / deny it. I still use Watchtower for Portainer and Portainer edge agents. ---------------------------------------------------------------------- smarzzz · 2023-10-09 05:40 Homelab no. The LCMcycles on k8s are a bit too intense. There no need and value for me running that at home ---------------------------------------------------------------------- sowhatidoit · 2023-10-09 05:53 Can you elaborate on the Ansible part? Can Ansible be used to update containers as well? ---------------------------------------------------------------------- sowhatidoit · 2023-10-09 05:55 How do you use Ansible to run container updates? By using the :latest tag in the compose file? ---------------------------------------------------------------------- sowhatidoit · 2023-10-09 06:00 This! I've been learning Ansible as it seems like a 2 for 1 type of deal. Auto update + documentation (IaC). You could potentially get all new hardware and be up and running with very low touch. ---------------------------------------------------------------------- thelittlewhite · 2023-10-09 06:12 Dockcheck will chech if there are available updates for your containers: https://github.com/mag37/dockcheck/releases You can use a specific version of the image instead of the "latest". You can also prevent a container from being updated by watchtower by using a label. I use a combination of all 3. Most containers are updated automatically, except databases for which I use a specific version. For the essential services I disable the auto update and check manually with dockcheck. ---------------------------------------------------------------------- PovilasID · 2023-10-09 06:15 I like it because I can get an update informed if there is an update and then update it if I think it needs it and to check if there are no breaking changes. ---------------------------------------------------------------------- Rocketstone · 2023-10-09 06:43 Well I guess I’m one of those people who literally manually updates all their services. Why? Because automating it is too much work and I’m too lazy 😂 ---------------------------------------------------------------------- cip43r · 2023-10-09 06:44 I don't ---------------------------------------------------------------------- michalsrb · 2023-10-09 06:45 zypper up, update it with the rest of the system? Why is everyone so keen on sticking everything in containers? ---------------------------------------------------------------------- averyrisu · 2023-10-09 07:04 I pin my docker containers to latest. anything not in docker its sudo apt-get update & sudo apt-get upgrade ---------------------------------------------------------------------- Shane75776 · 2023-10-09 07:32 I use unraid, every so often like maybe once every couple weeks I press a button that updates all the containers that it sees have updates. Its simple, and I don't have to think about it. I've never had a problem with any container updating but if I ever did, since it's docker it takes all of a minute to roll back to the previous version. It doesn't have to be complicated or time consuming. ---------------------------------------------------------------------- FragoulisNaval · 2023-10-09 07:46 I use watchtower to update and pull down any new images. I don’t update the stacks that use databases unless the database has also an update for it. Services that run as standalone, I update them every week. I use the docket compose files as they are from their creators, with no issues so far, except Nextcloud which is to be expected 😂 If anything goes wrong, I restore my backup from duplicaty which runs daily ---------------------------------------------------------------------- reciprocaldiscomfort · 2023-10-09 07:59 Diun and I check the logs cause I haven't set up any of the notifications. ---------------------------------------------------------------------- moontear · 2023-10-09 07:59 Have been doing this since years, also with :latest and stuff is fine. There are some things that break - DB updates sometimes need a little fixup afterwards or some apps introduce breaking changes that you are Sometimes surprised by this way, but in the end it is all fixable afterwards. If it isn’t (didn’t happen so far ever), I would just roll back to the last version using my backups (you have backups, right?). ---------------------------------------------------------------------- Dusterthefirst · 2023-10-09 08:25 How do you integrate your git repository to your cluster? ---------------------------------------------------------------------- EmberMouse · 2023-10-09 08:27 TeamCity here; CD is the way to go. ---------------------------------------------------------------------- MDSExpro · 2023-10-09 08:35 > and pinning to latest can also bring unpleasant surprises. Way less surprises than getting data leak / ransoware through unpatched hole. I'm having :lastest everywhere + daily "dnf update -y && reboot" in crontab on k8 nodes for 2 years now and nothing broke except Nextcloud, but Nextcloud brakes whenever you update it anyway, so it doesn't count. Just add automatic daily snapshots and rollback of something breaks. ---------------------------------------------------------------------- anon108 · 2023-10-09 08:57 I look at portainer a lot. So whenever I don't see a green mark next to the container, I use dockcheck to update them. ---------------------------------------------------------------------- viviolay · 2023-10-09 09:03 Unraid with an auto update community extension. But I also have one that does multiple backups too. It’s set to check and update all containers at 2:30am. ---------------------------------------------------------------------- jogai-san · 2023-10-09 09:11 With cosmos, see /r/selfhosted/comments/172ycag/cosmos_0100_all_in_one _secure_reverseproxy/ ---------------------------------------------------------------------- daedric · 2023-10-09 09:28 Diun with Matrix notifications ---------------------------------------------------------------------- Elle221LL · 2023-10-09 10:02 watchtower + duplicati duplicati - Run script to shutdown all services that you want backup off, then backup with duplicati the config folder ---------------------------------------------------------------------- kneticz · 2023-10-09 10:33 cronjob and a docker build every week 🤷‍♂️ ---------------------------------------------------------------------- SpongederpSquarefap · 2023-10-09 10:48 Watchtower and cron jobs Cron jobs handle my compose stacks by doing a compose down, pull and up Then I have some other stacks (Zabbix and Immich) where I just do a compose pull and update manually every few months ---------------------------------------------------------------------- nothing_slash_actor · 2023-10-09 10:49 I second renovate. I use it with portainer repositories and git hub actions. Reading release notes is essential because sometimes app do breaking changes that are time consuming to reverse. Now I get an email for a new release, click one button to merge and portainer updates the compose stack. ---------------------------------------------------------------------- Wartz · 2023-10-09 10:50 It can be used to automate actions on a container host, there is a community module for it. https://docs.ansible.com/ansible/latest/collections/community/docker/d ocker_container_module.html It's also great for configuration management of new virtual machines, hypervisor config, anything really. ---------------------------------------------------------------------- fuken33 · 2023-10-09 11:24 That's the neat part... You don't ---------------------------------------------------------------------- djgizmo · 2023-10-09 11:27 Make daily backups of services config / data folder and auto update. Saves so much time. ---------------------------------------------------------------------- MrSliff84 · 2023-10-09 11:33 On general docker hosts: Renovate or watchtower. On unraid: Auto update applications Plugin. ---------------------------------------------------------------------- purepersistence · 2023-10-09 11:55 I update Most things with watchtower to latest. A few things I update only to a major release I'm on. That includes my nginx proxy manager and wiki site. I don't want the proxy manager breaking because that will break everything it proxies to. >i treat it as an opportunity to learn how to fix things. Some times I do similarly but I sure don't want to be forced to. In my case all my docker containers run on a linux VM. I take scheduled snapshots of each VM a few times a day. If watchtower breaks something I can just restore that VM from a snapshot. Of course I can't just stop there because watchtower will do the same thing later that day if I let it. But this is still an easy way to get things working again in like two minutes. ---------------------------------------------------------------------- Theon · 2023-10-09 12:01 > 29 That's about 2.5 dozens ;) >scripts What do they do exactly, check a RSS or a docker hub - or also the "grunt work" like taking backups, reinstalling (if necessary), etc? ---------------------------------------------------------------------- Theon · 2023-10-09 12:03 Thanks, this looks great! Overkill for many for sure, but I think I've just reached the point where this is really enticing. Any pointers or guides you'd recommend? ---------------------------------------------------------------------- Theon · 2023-10-09 12:07 Very cool! Unfortunately, the very first service I tried to add (Forgejo) isn't supported, since it's on Codeberg.org including docker, definitely will explore though, thanks ---------------------------------------------------------------------- sun_in_the_winter · 2023-10-09 12:15 Watchtower has telegram notifications and works perfectly. ---------------------------------------------------------------------- prime_1996 · 2023-10-09 12:30 Yeap, I use docker compose, and each service has its own folder. So I get a list of folders dynamically and loop through them, doing docker compose pull the compose up. ---------------------------------------------------------------------- prime_1996 · 2023-10-09 12:32 Ansible semaphore is a great tool to run Ansible playbooks. ---------------------------------------------------------------------- penguinmatt · 2023-10-09 12:34 Whats Up Docker ---------------------------------------------------------------------- Sekhen · 2023-10-09 12:59 * Running apt functions. (apt update && apt upgrade). * Rebooting when there is a new kernel. * Running backups. * Making sure the server has an IP. * Updating software settings (pihole -up). * Copying updated config files. To name a few.... ---------------------------------------------------------------------- jstevewhite · 2023-10-09 13:22 Yep. I run "all these services" to learn, not because I can't find an external site that hosts a pastebin or image sharing app. I do it to 'dogfood' services. Of course I use a lot of the skillset in my day job so it's "continuing education", as they say. ---------------------------------------------------------------------- clintkev251 · 2023-10-09 14:06 No guide that I know of which covers everything, but the individual components aren't too hard to set up. FluxCD is really the backbone of it all in my case and they provide some pretty good documentation around getting started, the biggest pre-requisite is Kubernetes, and I think that probably rules it out for most people. [https://fluxcd.io/flux/installation/](https://fluxcd.io/flux/installa tion/) If you're not running Kubernetes, I'm pretty sure Portainer (and probably other options as well) has a GitOps component which could be used to integrate a Git repo with your service deployment, though I don't have any personal experience with this. Once you have some kind of CD in place, Renovate is super easy. They have a hosted app so you can basically just add it to your Github repo and it will start working. [https://docs.renovatebot.com/getting- started/running/](https://docs.renovatebot.com/getting-started/use- cases/) You can also self host the runner if you want as well ---------------------------------------------------------------------- clintkev251 · 2023-10-09 14:10 That's what FluxCD does. Basically Flux ensures that the state of your cluster always matches the state of your repo. So if I want to stand up a new application, all I do is write the manifests, commit and push them to my repo, and Flux deploys them to the cluster and Kubernetes handles standing them up. Flux also comes with a bunch of CRDs that make managing stuff like Helm fully declarative and much easier ---------------------------------------------------------------------- Theon · 2023-10-09 14:16 Awesome, thanks! This does help a lot. > If you're not running Kubernetes, I'm pretty sure Portainer (and probably other options as well) has a GitOps component which could be used to integrate a Git repo with your service deployment, though I don't have any personal experience with this. Yes it does, I finally understand why would you want to use it :) (Also you guessed right I'm not using k8s (yet?)) This may not apply to you at all then, but is my understanding right in that you end up having a git repo **per** stack? As in, if I'm running 10 stacks (with 1-3 services each), I'll probably need to set up 10 repos? ---------------------------------------------------------------------- Double_A_92 · 2023-10-09 14:36 Why did I get a notification for this just now? Fucking reddit... ---------------------------------------------------------------------- NOAM7778 · 2023-10-09 14:36 On my k8s cluster, I pin all images to a specific version/tag (trying to avoid latest as much as possible), and I watch the github release pages on a Change-Detection instance running in the cluster (only for important apps I want to keep updated regularly). Besides that I update pfSense, trueNAS and Proxmox whenever I have time to deal with potential issues and downtime - they are pretty mature projects with really stable releases that IMHO don't have to be udated so often (UNLESS there is a security vulnerability) ---------------------------------------------------------------------- froli · 2023-10-09 14:42 I use Watchtower to only fetch updates and notify. I have the RSS feeds of the Github release page of the services I host to my RSS aggregator. When I get notified, I go check my changelog feed on Miniflux and then manually apply the update. Also, I keep each project in their own docker compose stack. Being diligent in reading the changelogs allows me to mostly use `:latest` for everything without causing issues. Only for some DB containers I might pin to a major version. ---------------------------------------------------------------------- froli · 2023-10-09 14:50 Pinning DBs to a major version would probably take care of this. For example `postgres:13`. You'll get all the 13.x.x.x updates but it wont upgrade to 14 and beyond until you change the tag. ---------------------------------------------------------------------- chrishas35 · 2023-10-09 14:53 I'd be interested in the docker automation. I have something rather basic that I'm not terribly happy with, so would love to see how others are accomplishing it. ---------------------------------------------------------------------- chrishas35 · 2023-10-09 14:56 >portainer updates the compose stack. portainer will automatically update? I've missed this then. I've cobbled together some scripts, that I'm not entirely happy with and would love something better. ---------------------------------------------------------------------- computerhero1337 · 2023-10-09 15:15 I have RSS feeds from the GitHub release pages of the services and do upgrades manually. ---------------------------------------------------------------------- Fatali · 2023-10-09 15:29 Renovate eith ArgoCD here, works great Many are set to automerge on patch/minor with Argo set to auto sync Major apps, or apps I don't trust the releases of are manual merge/sync ---------------------------------------------------------------------- nothing_slash_actor · 2023-10-09 16:52 Yes, you can use the GitOps update feature and use webhooks. As soon as you update your compose file you can send a request to the webhook and it pulls and redeploys the stack. ---------------------------------------------------------------------- moontear · 2023-10-09 20:40 Very good point! The „problem“ I have with pinning that I then pretty much never get updates. I just found a container that wasn’t updated in a year because I used a specific version and boy there were a million updates. ---------------------------------------------------------------------- speediegq · 2023-10-09 20:51 It is 2023 and as such I let package managers do the job. ---------------------------------------------------------------------- opensrcdev · 2023-10-09 22:09 Hmmm, I'm not sure what that is. I have more to monitor than just containers though. I use Uptime Kuma for ICMP (ping) and HTTPS monitors. ---------------------------------------------------------------------- zpool_scrub_aquarium · 2023-10-09 22:33 Watchtower, every 30 minutes. I don't have any databases, and I have two internet facing services (Filebrowser and Plex). Thus, something breaking is preferable to getting a critical update too late. ---------------------------------------------------------------------- CodexHere · 2023-10-10 03:53 Honestly I really only ever update when HomeAssistant has an update 😅 But I wrote a suite of scripts to help me manage my Docker Compose environments: https://github.com/hangouthere/multi-docker-compose ---------------------------------------------------------------------- ftrava · 2023-10-10 05:14 https://github.com/louislam/uptime-kuma/wiki/How-to-Monitor-Docker- Containers ---------------------------------------------------------------------- OnAQuestForDankCatsA · 2023-10-10 11:02 Same, renovate that auto merge with argocd with apps that auto sync ---------------------------------------------------------------------- lmm7425 · 2023-10-10 13:57 At first, I was using Drone to SSH into my Docker host and run a script to update my containers on every PR merge in Gitea. That setup is below. https://loganmarchione.com/2023/04/continuous-delivery-with-docker- compose-and-drone/ Then, I realized that Drone was basically running "docker compose up -d" on, so I wrote a small bash script to run that via crontab. Much simpler and less moving parts. https://github.com/loganmarchione/dccd ---------------------------------------------------------------------- lmm7425 · 2023-10-10 13:57 I just wrote another comment with more details. https://reddit.com/r/selfhosted/comments/1735dyj/how_do_you_keep_your_ services_up_to_date/k49w1p6/ ---------------------------------------------------------------------- chrishas35 · 2023-10-10 14:19 Thanks for sharing! Your alternatives listing harbormaster was timely, as I even started to use it until I came to dislike the structure it imposes. ---------------------------------------------------------------------- FLeiXiuS · 2023-10-11 04:57 I meant at whatever workload that demands six 9's. ---------------------------------------------------------------------- smarzzz · 2023-10-11 05:42 Absolutely ---------------------------------------------------------------------- kon_dev · 2023-12-31 12:58 Sounds impossible and/or exaggerated, sry... How do you handle OS patches which require reboots or hardware failures? Replacing a failing disk in a RAID is not that hard, but other things will fail eventually as well. I mean you would need every software and hardware component multiple times to ensure redundancy, also most likely multiple independent ISPs if network access is part of your 100% claim. Unless you are running a mainframe or spending incredible efforts in building redundancy in every spot, 100% is extremely unlikely. How do you even define and monitor uptime? Not even cloud providers or services like amazon S3 claims 100% availability: https://aws.amazon.com/s3/sla/ They also exclude outages "caused by factors outside of our reasonable control, including any force majeure event or Internet access or related problems beyond the demarcation point of Amazon S3" from their SLA statements.