I build a tool to monitor your dockerized services with notifications reddit / selfhosted ---------------------------------------------------------------------- dangrie158 · 2023-09-30 15:39 While I was on vacation this year the shop of my wife was suddenly down and we only noticed once the emails of customers rolled in. After fixing it on my phone I thought there has to be a better way to notice when your docker container die or are in a restart loop for whatever reason. Now I know there are lots of tools like [healthcheck.io](https://healthchecks.io) and [monocker](https://hub.docker.com/r/petersem/monocker), however none seemed to match my simple use case of getting notified *via email* when the running– or health-status of any docker container changes. So I did what every SE would do in that case and hacked another solution but this one is mine :). **TL;DR** \- What is this: Get notification via * EMail (BYOSMTP) * Discord * Telegram if any (or only selected) containers on your host die or change their health status. I also tried to handle unexpected restarts of the container the tool is running in gracefully without loosing any events that happened before and even while the service was down. To make sure you are not spammed by notifications if a container goes into a restart loop, there is also a configurable exponential backoff function. I tried to make using the tool as easy as possible (by getting inspired on how monocker is set up). I would love to get some feedback and hear what you think (also if I just missed the perfect tool for the job: let me know, however the journey was quite fun). **You can find the link here:** [**https://github.com/dangrie158/dolce **](https://github.com/dangrie158/dolce) ---------------------------------------------------------------------- thekrautboy · 2023-09-30 16:05 Neat, i am a fan of monocker and i also use autoheal, but both of them simply notify me through Pushover, i dont have a need to do this over email. In case the internet goes down, i use local notifications through Gotify/ntfy for important things. Going to give your tool a try, thanks for sharing! ---------------------------------------------------------------------- dangrie158 · 2023-09-30 16:33 Hi, thanks for the feedback. Indeed you currently need r/W access to the socket since you need to send HTTP requests over the socket to get an answer (although the software effectively only reads, i.e. doesn't change anything in docker). I like the idea of providing a TCP alternative though. Thanks for the suggestion, I will definitely add this once I get to it ---------------------------------------------------------------------- thekrautboy · 2023-09-30 16:40 > Indeed you currently need r/W access to the socket This is of course not ideal. Thanks for considering the TCP proxy. ---------------------------------------------------------------------- thekrautboy · 2023-09-30 17:52 One more thing you might want to add, a healtcheck for the Docker image. Something like this in the compose works: healthcheck: test: ['CMD-SHELL', 'pgrep -x "deno" || exit 1'] interval: 30s timeout: 10s retries: 3 But its better to have it builtin the image itself imo, plus i have no idea if simply checking if "deno" is running makes sense at all, or maybe to check if "tini" is running, or something else. Sort of ironic to have a tool that monitors healtchecks of other containers to not have a healthcheck on its own ;) ---------------------------------------------------------------------- dangrie158 · 2023-09-30 18:47 I just pushed version v2.2.0 which adds support. See [this](https://github.com/dangrie158/dolce#use-reduced-permissions) section of the readme for details ---------------------------------------------------------------------- ElevenNotes · 2023-09-30 18:53 Wait till you people learn what a load balancer is! ---------------------------------------------------------------------- thekrautboy · 2023-09-30 19:00 Oh fantastic! Going to test in a bit. ---------------------------------------------------------------------- dangrie158 · 2023-09-30 19:08 >One more thing you might want to add, a healtcheck for the Docker image. I'm not sure how I can easily check this. since the deno process is the only one running inside the container, I don't think it is really helpful to monitor if the process is running since the whole container would stop if the deno process dies. >And a very very minor thing, maybe change the label from dolce.enabled to dolce.enable to keep it uniform with tools like monocker and diun. Interesting. I had \`dolce.enable\` and then changed it to \`enabled\` since I think an adjective is more correct as a label than a verb. Now I think I know why I chose \`enable\` in the first place. I'll consider it for the next major version since this is a breaking change (although probably only I am using it atm :)) >Just tested it now, i really like the layout of the email, well done! Thanks >Maybe you could use the hostname from the actual container itself, instead from the host machine? That way people can set a custom hostname that is then displayed in the email instead of the fixed name of the host which may not always be one wanted. Or instead add a variable for the entire "Dolce@Hostname" thing. So one could set to "Docker@ServerB" or whatever. Yeah, there's always more to customise, but I think these seem reasonable. BTW: you can always override the templates that come with the container. I want to create a proper documentation site when I have the time that explains all of this but you can simply override the \`templates/emails/base.eta\` file. I also want to expose the environment variables to the templates so you have even more freedom. >And i must say i do not like having a "thirdparty" image in the email. Sure it looks neat but i dont think it should be there, why load external things that arent required. Do you have a good reason or is this just a general disliking? I personally don't mind it and I took extra care that the mail will still look good if your client just refuses to load external images. And of course you can always override the template :). >Also, maybe have a variable for setting the SMTP\_SENDERNAME in addition to just the SMTP\_FROM field. Not sure what you mean. You can already set SMTP\_FROM to "My Dolce@Home " (RFC 5322) if you want you client to display a "proper" name ​ Thanks again for the detailed feedback and trying it out ---------------------------------------------------------------------- dangrie158 · 2023-09-30 19:15 Wait till your service is unavailable because your requests get lost somewhere between you load balancer, 4 reverse proxies and ingress controller and you can't figure out what host is the problem because everything in your next cloud setup is hyperscalable :P ---------------------------------------------------------------------- thekrautboy · 2023-09-30 19:16 Works instantly, no issues! 2023-09-30T19:09:25.823767771Z INFO main starting dolce container monitor v2.2.0 2023-09-30T19:09:25.864793615Z INFO main created lockfile /var/run/dolce/lockfile for pid 7 2023-09-30T19:09:25.865227295Z DEBUG main connecting to Docker API socket at tcp://docker-socket-proxy:2375 2023-09-30T19:09:26.004739748Z INFO main connected to Docker 24.0.5 (API: 1.43) [...] 2023-09-30T19:11:01.945439575Z INFO main new container event received: 2023-09-30T19:11:44.806982175Z INFO main sending events notification to all registered notifiers with 2 events 2023-09-30T19:11:44.949773744Z INFO notifier sending mail to username@gmail.com via SMTP Notifier 2023-09-30T19:11:46.633873686Z INFO main earliest next notification at 9/30/2023, 7:28:24 PM And i have confirmed it works with read-only and limited access, more specifically exactly this setup: environment: - POST=0 - INFO=1 Perfect :) ---------------------------------------------------------------------- dangrie158 · 2023-09-30 19:18 Cool, thanks for testing and the quick feedback :) ---------------------------------------------------------------------- thekrautboy · 2023-09-30 19:34 > I'm not sure how I can easily check this. since the deno process is the only one running inside the container, I don't think it is really helpful to monitor if the process is running since the whole container would stop if the deno process dies. True, so the question would be to either have a healthcheck that is not 100% indicative of the actual software properly working, or to not have any healthcheck at all... im not a developer, i have no idea whats the best approach. Personally i try to add custom healtchecks to all containers that i deploy, if they dont have one already builtin. When its a webservice, its easy enough to check with wget or curl if the webserver is responding, and based on that just assuming its probably okay. Sometimes there is a proper API endpoint that reports actual health status, then i use that of course. But every now and then there are things like this, where there is no service to easily check for proper function and the only way would be to check if a process is running or not. > Interesting. I had `dolce.enable` and then changed it to `enabled` since I think an adjective is more correct as a label than a verb. Now I think I know why I chose `enable` in the first place. I'll consider it for the next major version since this is a breaking change (although probably only I am using it atm :)) Yeah it happened to me that i added it to a test container and simply because i was used to it being "x.enable" i used that, and of course it failed with dolce and i realized its "x.enabled" for this xD > BTW: you can always override the templates that come with the container. I want to create a proper documentation site when I have the time that explains all of this but you can simply override the `templates/emails/base.eta` file. Ohh that is great! I will try that out by mounting a modified copy of the file into the container. Only slight problem i see with that is casual Docker users might get tripped by the fact that Docker cannot create mounted files if they dont exist on the host, it fails and tries to mount it as a folder. So users need to create the file first, then start the container with the mount. So having a few more environment vars to customize the email layout would probably be ideal to avoid this issue with users in the future. Alternatively you could have your software check if a custom template file exits in a specific dedicated folder, if it does, it uses that. If it doesnt exist, it copies the internal default to that folder and uses it then. The result would be that users can mount that template folder to the Docker host (instead of a file), and after first startup they would find the template file in there and can modify it. And for someone who doesnt mount that folder, it still works as usual. > Do you have a good reason or is this just a general disliking? I personally don't mind it and I took extra care that the mail will still look good if your client just refuses to load external images. Honestly im not so sure even. Its just a odd feeling about it but... > And of course you can always override the template :). ... but with this option it doesnt matter about the logo of course :) > Not sure what you mean. You can already set SMTP_FROM to "My Dolce@Home " (RFC 5322) if you want you client to display a "proper" name Ohhh i didnt know that works, perfect! Maybe modify the readme part to show that as a option, you might get this question a few times in the future otherwise xD ---------------------------------------------------------------------- thekrautboy · 2023-09-30 23:32 [The Load Balancer.](https://media.giphy.com/media/myDXHnYYPxT4od4NEN/giphy.gif) ---------------------------------------------------------------------- zn448sk39 · 2023-09-30 23:50 You can curl your app in a healthcheck ---------------------------------------------------------------------- maximus459 · 2023-10-01 04:21 +1 to the gotify (etc) suggestion.. ---------------------------------------------------------------------- ewixy750 · 2023-10-01 04:55 Uptime Kuma isn't providing what you need? ---------------------------------------------------------------------- ElevenNotes · 2023-10-01 05:19 If you don't know that maybe you shouldn't run anything. ---------------------------------------------------------------------- ftrava · 2023-10-01 07:15 Same thing that popped in my brain while reading this ---------------------------------------------------------------------- dangrie158 · 2023-10-01 14:14 I actually didn't know about this project, thanks for sharing. Looks interesting and would probably do everything I needed but it looks quite a bit more involved/manual to setup where I tried to provide sensible defaults for a basic zero-configuration. Also its interesting that this seems to have the same problem that monocker has in that it only checks the docker container status periodically and thus may miss a unfortunately timed container restart cycle whereas Dolce won't miss that as it listens for events from the docker daemon. Still, looks very good but much more involved as a do-it-all monitoring solution where dolce tries to provide a simple solution to a (basically) simple problem :) ---------------------------------------------------------------------- ewixy750 · 2023-10-01 14:33 So in your project there's no need to manually add the containers and would notify you directly whenever the status changes oh all the containers it can find on the host? ---------------------------------------------------------------------- dangrie158 · 2023-10-01 14:46 Exactly. There is also a way to only monitor containers that are tagged with a special value, but by default it just monitors all containers