Are there any concerns/risks to using Watchtower? reddit / selfhosted ---------------------------------------------------------------------- notdoreen · 2023-11-15 05:12 Someone recommended it for keeping my containers up to date automatically. I checked out the [repo](https://github.com/containrrr/watchtower) and it seems too good to be true. It just updates your containers when a new image is available and everything just works out of the box? I'm a bit scared of just leaving it alone in case it might break something. Does anyone here use it and can recommend it? Any horror stories? ---------------------------------------------------------------------- LeftBus3319 · 2023-11-15 05:19 The only issue I've ever had are when apps make a breaking change and I didn't keep up with it until the app stopped working upon Watchtower automatically updating it. *(So, not a watchtower problem)* That being said, setup something like Uptime Kuma for each service you run, and then use ntfy to get push notifications when downtime is detected, then you get notified when your app is broken, and then go fix it. ---------------------------------------------------------------------- shadoodled · 2023-11-15 05:26 > I’m a bit scared of just leaving it alone in case it might break something. yes, this is the primary concern because new version could have breaking changes. you do have options like configuring it to notify only. or you can use version specific docker image tags - example `:3.0` instead of `:latest` or you can use [diun](https://crazymax.dev/diun) instead which is built to just notify of updates and pull the updated images yourself. ---------------------------------------------------------------------- notdoreen · 2023-11-15 05:32 Thank you ---------------------------------------------------------------------- notdoreen · 2023-11-15 05:32 Thank you ---------------------------------------------------------------------- shahmeers · 2023-11-15 05:32 Watchtower keeps killing my containers for some reason. It stops the running container and fails to properly bring up the replacement container. Doesn't happen for all containers, but even one is too many IMO. I eventually replaced it with a simple Github Action that runs docker compose pull && docker compose up -d && docker system prune -A. The action is scheduled to run every day at 4AM. Unrelated but these Watchtower random failures finally pushed me to using Docker Swarm instead of Compose. Swarm is far superior to Compose as it has an actual daemon that continuously checks to see if the state of your system matches with what you specified in your compose file (and fixes things if it doesn't), whereas Compose simply gets things up and running. Swarm does have key limitations, for example you can't route a container's traffic through another container (e.g. routing qBittorrent traffic through a VPN container), but there are workarounds. ---------------------------------------------------------------------- davidht0 · 2023-11-15 05:58 I've been using watchtower for more than a year on all my containers and no issues so far. I have read many warnings against automating the updates, but it has never broken anything in my case. I'm talking about 3 VMs (on Proxmox) and 2 Synology boxes. 5 instances of watchtower keeping a total of 84 containers updated. Nonetheless I try to play it on the safe side and make daily backups in case something breaks. I've had a couple of containers breaking (nothing related to watchtower, AFAIK) and I have recovered easily restoring the latest backup. ---------------------------------------------------------------------- zoredache · 2023-11-15 06:01 There are risk, that a newer version of an image will accidentally, break things, apply breaking changes and so on. Good, frequent, tested backups, could be a mitigation to this. If an image breaks, you just restore your data from the backup, and pull the older image. I use the `klausmeyer/docker-registry-browser`, and that recently broke, but it just needed me to provide an additional configuration variable. I use `advplyr/audiobookshelf`, which upgraded to a different database engine and schema a couple months ago. For some small subset of people (including me) the migration to the new database didn't go well. But I had a backup from 6 hours before the update, so restoring and then using the older image until the fixes were released was easy. Even with the occasional issues I prefer letting watchtower automatically update most of my images for my home. I don't really want to spend my time manually applying updates when 98% of the time it will be fine. ---------------------------------------------------------------------- matz0r81 · 2023-11-15 06:53 I have been using it for years and never had a single issue. I use it on a private server only I use, so even if an update fails in the future it would not be the end of the world. I would not use it in an company environment I guess where a lot depends on everything running smoothly. ---------------------------------------------------------------------- Lanten101 · 2023-11-15 07:00 Bee using it for a while no issues, only tim3 I had to manually revert the update was when plex broke transcoding.. You can add a flag to delete old images as well.. otherwise they pile up and takes lot of space ---------------------------------------------------------------------- notdoreen · 2023-11-15 07:08 Thank you. What do you use for container backups and restoring? ---------------------------------------------------------------------- Old-Satisfaction-564 · 2023-11-15 07:16 I prefer to be there when container ar updated so that I can promptly fix anything that breaks. I have 2 watchtower instances in a docker-compose, the first container 'watchtower-monitor' uses **command: --monitor-only** and warns me over gotify about the availability of updates but does not modify anything, the second 'watchtower-once' uses **command: --run-once** and it is usually inactive since it performs all updates once and than exits. When i am ready to update everything I just **docker-compose start watchtower-once** container to start the updates. ---------------------------------------------------------------------- ChickenMcRibs · 2023-11-15 07:17 What system are you using to backup your containers? ---------------------------------------------------------------------- gohankr · 2023-11-15 07:17 If you want highly available system, then you should perform updates with a custom made script, where you can control update issues. Otherwise watchtower is good. ---------------------------------------------------------------------- rafipiccolo · 2023-11-15 07:20 Bind mounts + rsync + mysqldump of each database ---------------------------------------------------------------------- zfa · 2023-11-15 07:37 Normally fine but if you want to be more careful about what is being pushed to your server you can use something like [diun](https://github.com/crazy-max/diun) to get notifications and run updates manually. Personally I love [dockcheck](https://github.com/mag37/dockcheck), which I think is by a guy on the sub. I tend to just run that every now and again and be done with it unless I am notified of a perssing update, although I do still have a couple of things I don't care too much about just auto update with watchtower. ---------------------------------------------------------------------- AnderssonPeter · 2023-11-15 07:48 If you have a script like this any chance you could open source it? ---------------------------------------------------------------------- zoredache · 2023-11-15 07:49 My primary 'backup', or easy recovery method is that I use ZFS, and take snapshots via sanoid frequently. I have a mydumper jump making backups of my mariadb server. I use syncoid to doing sends to external storage. So most things can just be fixed by copying the files from an older snapshot. I also have a completely separate backups of my system made using borg to storage I have at borgbase.com, but this only happens a couple times a week, and is only my 'important' data and not large things like downloaded video/music/etc. I am thinking about switching borg out for restic though, since restic is also compatible with borgbase. ---------------------------------------------------------------------- AnderssonPeter · 2023-11-15 07:50 I use it but only on containers where I can configure it to not do major updates, sadly most images don't have the needed tags for this 😢 ---------------------------------------------------------------------- Byolock · 2023-11-15 08:00 Using Watchtower for approximately 2 years on about 20 Containers. I had 1 issue, where a container would not start after the update. The Error Message said I had an unsupported entry in the configuration file of the app. I looked up the changelog of that app, and found out that the option was removed and replaced by something else. Had to change one line in the configuration. Not really a problem for me. Though I decided to exclude my Home Automation Container and my kasm container ( my gateway to my network, a bit like guacamole ). Those may pose problems if they are offline unexpected. ---------------------------------------------------------------------- senectus · 2023-11-15 08:28 Yeah I used it, it broke paperless for me. I uninstalled it. ---------------------------------------------------------------------- SillyLilBear · 2023-11-15 09:13 The latest version isn't always the best version. In a home lab or home network, this is rarely a big problem, but in a production environment, I wouldn't recommend it. ---------------------------------------------------------------------- thekrautboy · 2023-11-15 09:23 As example, some software pushes out updates that can (and sometimes will) break your setup. Of course nobody pushes out something like that on purpose to mess with users. But mistakes happen all the time. And even if the dont, some version upgrades require the user to take manual steps, when these are ignored and with something like Watchtower just blindly upgraded, setups can and very likely will break. Imo its not worth the very short amount of time saved by automatic- updates versus the amount of time it costs to fix such a mess when it occurs. For example, NPM (Nginx Proxy Manager) had a update months ago that broke many users setups. They of course did warn about this in the changenotes, but i remember people here on sub saying "well damn i used watchtower and it updated npm overnight and i wake up and nothing works anymore, took me hours to figure out the reason and fix it". https://github.com/NginxProxyManager/nginx-proxy- manager/releases/tag/v2.10.0 ---------------------------------------------------------------------- GolemancerVekk · 2023-11-15 09:46 **Please** take such advice with a large grain of salt. OP's experience is very much not the norm. Especially for more complex apps like Jellyfin or Nextcloud, it's almost guaranteed you'll break them if you just update blindly. ---------------------------------------------------------------------- azukaar · 2023-11-15 10:00 Using an outdated version of a container (including DBs!) that have known vulnerabilities that will be very easy to exploits including by bots, is so much worse than the risk of a container breaking after an update. Just monitor your server properly and you'll be good ---------------------------------------------------------------------- tenekev · 2023-11-15 10:17 Watchtower checks and downloads updates for me. It renames the tag of the image in use. All I need to do is redeploy the container. It picks up the new image that is tagged correctly. This gives me best of both worlds: 1. Spares me the effort of checking for updates and downloading them manually. 2. Allows me to update manually, ideally checking the changelogs. ---------------------------------------------------------------------- Simon-RedditAccount · 2023-11-15 10:23 Yes, there are risks: * First, updates can break things. Already explained here. * Second, exposing Docker socket to Watchtower means you have to trust it ultimately. Any vulnerability in WT can lead to whole system compromise. Personally, I use DIUN. It just sends me notifications about available updates. I update things manually later. My system is pretty well isolated from outside world, so no need to hurry. *On a VPS, I would prefer a different approach though.* ---------------------------------------------------------------------- Calm-Size-1110 · 2023-11-15 10:25 You can set notifications so you know which container are updated recently. If that container stops working, then just revert to previous image. And configure when watchtower should run the update. I set mine to update at 8pm, so in case something breaks, I still have a few hours before bedtime to fix it. ---------------------------------------------------------------------- thekrautboy · 2023-11-15 10:50 I am happy in the camp of diun+dockcheck too, they both dont get enough love. ---------------------------------------------------------------------- Grosaprap · 2023-11-15 11:45 You understand that at the point where they open source it and publish it it would be essentially watchtower right? The point of having a custom-made script is so that you can customize it to your specific needs if it's a generalized item then just use watchtower. ---------------------------------------------------------------------- AnderssonPeter · 2023-11-15 12:44 Yes and there might be more people that need a custom script, and instead of having to start from scratch having a base to start from would be nice? ---------------------------------------------------------------------- unixuser011 · 2023-11-15 13:31 Don't backup the actual container, backup the volumes and the docker- compose file ---------------------------------------------------------------------- ProbablePenguin · 2023-11-15 13:43 Watchtower itself works great, it doesn't need a GUI for what it does. But updating containers in general always carries a risk of something breaking due to the new update. One thing you can do is make sure you're not using `:latest` tags in your compose files, and instead pin major versions like `postgres:13` ---------------------------------------------------------------------- roycorderov · 2023-11-15 13:44 I got my proxmox in production and I've installed before whatschtower and just broke me down 4 containers with bad updates so I stoped from using whatschtower... I would like any services that just notify me about any new docker image update whitout making any updating ---------------------------------------------------------------------- trisanachandler · 2023-11-15 14:10 That sounds like my strategy. It feels a little janky, but it works well, and I have had to recover things from it. ---------------------------------------------------------------------- CactusBoyScout · 2023-11-15 14:12 Yeah I let Watchtower handle everything except Qbittorrent because the trackers I use have to approve each update anyway. So it’s not going to be allowed immediately. ---------------------------------------------------------------------- CactusBoyScout · 2023-11-15 14:13 I just had a strange issue with Watchtower where it somehow failed to update itself. And it left a running but unhealthy duplicate of itself. Just restarting the old container fixed it. But I guess that’s a risk? ---------------------------------------------------------------------- MRobi83 · 2023-11-15 14:33 Curious how a custom script to perform the update would be different than watchtower doing it? Is an automated update not an automated update regardless of what triggers it? ---------------------------------------------------------------------- davidht0 · 2023-11-15 15:14 * 3 VMs in Proxmox hosting 70 containers get backed up everyday with ProxmoxBackupServer (VM in my primary NAS) to a NFS mounted folder on my primary NAS * Primary NAS (with 7 containers) gets backed up with Snapshot replication to my secondary NAS everyday. * Secondary NAS (with 7 containers) gets backed up with Snapshot replication to my primay NAS everyday. * And once a month I backup my primary NAS (not the whole thing,only the important folders) to a USB drive that I store at a friends house. ---------------------------------------------------------------------- Snowmobile2004 · 2023-11-15 15:29 Just get chatGPT to make you one ---------------------------------------------------------------------- Tangbuster · 2023-11-15 16:02 Used Watchtower on my Synology for a while and it worked well. No issues in that time. Now I’ve moved to a Nuc and am more experienced with Docker and understand a lot more of it but by no means am a professional by any means, I would say that I wouldn’t use Watchtower. I can definitely see it messing a config up and prefer not to deal with the headache of troubleshooting something without knowing it was an auto update. If I had the time, I may tag the apps I’m happy to auto-update but for now I prefer to have the higher availability. ---------------------------------------------------------------------- PuckSR · 2023-11-15 16:53 1. If you dont set it in the parameters, watchtower will not delete the old images which means you can overload your drive 2. Updates aren't always a good idea. Automating updates can cause issues. ---------------------------------------------------------------------- daYMAN007 · 2023-11-15 16:57 My script looks like this: [https://gist.github.com/dgalli1/010fb978ba e509dda43a1f31145a530f](https://gist.github.com/dgalli1/010fb978bae509 dda43a1f31145a530f) And is ment to update docker-compose.yml files. ---------------------------------------------------------------------- AnderssonPeter · 2023-11-15 16:58 Thanks I will have a look:) ---------------------------------------------------------------------- notdoreen · 2023-11-15 17:26 What do you use now? ---------------------------------------------------------------------- Tangbuster · 2023-11-15 17:29 For container updates? Nothing. I'm using DIUN which runs daily and tells me via a Telegram notification what new images are available for the containers I have up. It's useful but overall, I'll not really update my containers too often unless there is a clear bug fix or feature introduced. ---------------------------------------------------------------------- Cobthecobbler · 2023-11-15 18:05 I don't know if maybe I'm using watchtower wrong but i don't like how it behaves by default. It's always messing up my container names, not removing old containers and just spinning up new ones, etc and there's no interface so I can't view jobs that ran overnight, or see what's queued or in progress. It just does it's own thing and I really don't like that. I've installed and un-installed it a few times and it's been un-installed for a while now. I just redeploy my portainer stacks and pull down the latest image manually when I want to upgrade my containers now. At least I get some control ---------------------------------------------------------------------- notdoreen · 2023-11-15 18:31 Thank you ---------------------------------------------------------------------- zoommicrowave · 2023-11-15 19:43 Great script! Only thing I can recommend is adding a "docker image prune -af" command after all compose files have had new images pulled and are up…unless you want old images taking up hard drive space/you have a valid reason for keeping old images. ---------------------------------------------------------------------- ChickenMcRibs · 2023-11-15 20:12 Thanks will give it a try ---------------------------------------------------------------------- lilolalu · 2023-11-15 21:16 There is a common misconception that watchtower just updates all containers automatically. That's wrong. You can set labels if you want it to auto update, or just send a notification when an update is available - or do nothing at all. ---------------------------------------------------------------------- Digital_Voodoo · 2023-11-15 21:24 Apart from the container name, what did you change in the compose files? I tried this approach, changed only the container_name, and couldn't get the second instance to start. I'd appreciate your view/help on this. ---------------------------------------------------------------------- dysoxa · 2023-11-15 21:45 I use it and I recommend it. It's only dangerous if you leave your versions as "latest". For services that have a somewhat stable "latest" channel (by default, every published version is published there so be careful) and that can be down without big issue or have a proven seamless update process, I leave the version to "latest" and watchtower does its magic. Most services fall in that category. Others are more finicky or annoying when down, and for those I set the versions manually, and I edit my compose file when I want to update. Also be aware that, for some images, you can pin a major release, like `nextcloud:27`. This means that watchtower will update it to any minor version of 27, but will not update it to 28 on its own! ---------------------------------------------------------------------- MalcolmY · 2023-11-16 01:11 Do your run this script manually? ---------------------------------------------------------------------- Independent_Till5832 · 2023-11-16 10:01 If you want to have zero downtime, you can scale the container to two and if everything succeeded just kill the old container. (Need reverse proxy with balancing like caddy) ---------------------------------------------------------------------- daYMAN007 · 2023-11-16 19:08 I just run it manually whenever i got the time for it. But after reading this message I think I have to integrate this into home assistant so that it can ask me a yes now question when I'm at home to trigger the update via that.