Post AzQuQ09wenVLN3OBlY by cichy1173@mastodon.social
(DIR) More posts by cichy1173@mastodon.social
(DIR) Post #AzPeGjDwSd7Robr4YC by louis@social.louis-vallat.dev
2025-10-20T21:19:55.924Z
0 likes, 0 repeats
Problem: FS [/]: Space is low (used > 80%)Host: sharkeyOh no :neocat_laptop_notice:
(DIR) Post #AzPhR63yhQJxPNpWpk by louis@social.louis-vallat.dev
2025-10-20T21:55:24.982Z
0 likes, 0 repeats
Shout out to my monitoring system, who saved another instance from a now distant "no space left" future, to Proxmox for being able to increase disk size without rebooting the VM, and for tools like growpart and resize2fs for making the filesystem a bit more comfortable, all while the machine was still cruising along as if nothing was happening :neocat_cool_fingerguns:
(DIR) Post #AzQfdMrA5BUgDUWDrs by cichy1173@mastodon.social
2025-10-20T22:39:04Z
1 likes, 0 repeats
@louis what monitoring solution do you use?
(DIR) Post #AzQfdO6nQXau6GqDsO by louis@social.louis-vallat.dev
2025-10-21T09:09:54.641Z
0 likes, 0 repeats
@cichy1173@mastodon.social My monitoring setup evolved quite a bit over the years, especially recently :neocat_think:At first, I used a TIG (Telegraf, InfluxDB and Grafana) stack for my internal monitoring, and Uptime Kuma for external (meaning public-facing services) alerting, but that wasn't sufficient.I really loved the graphs that Grafana showed and the amount of details and metrics that Telegraf collected, but I didn't have any alerting based on this data, it was only monitoring my infra and not telling me if stuff was going bad (i.e. host down or disk full).Setting up alerting in InfluxDB could have been possible but I didn't want to spend ages crafting custom rules for everything that could go bad on my systems.At some point, I already had a few machines go down due to their disk being full, and I only got the notification because they were hosting a public-facing service, so after another outage due to that, I searched for another solution and went with Zabbix.As of right now, I still have an on-prem Uptime Kuma for my public-facing monitoring (with a bonus of being able to show a clean public dashboard) and Zabbix for my internal monitoring and alerting. :neocat_cool_fingerguns:I like this setup a lot, the default monitoring configuration is gathering a lot of useful data, the graphs are well presented (although not as sexy as Grafana's), alerting configuration is well thought out with sane defaults that just work, the default templates are very good and you can customize them or even add missing points if needed, there's auto discovery of devices and monitoring elements (i.e. "for all new network interface discovered on a host, monitor these elements with these alerts configured by default").Highly recommend! :chefs_kiss:
(DIR) Post #AzQuPrq9aUv5ZM4Go4 by cichy1173@mastodon.social
2025-10-21T09:14:18Z
1 likes, 0 repeats
@louis I use #Zabbix too!! And i love this solution a lot. Actually I have a story too. I started monitoring with Uptime-Kuma just like you and I deployed it externally on cheap VPS to monitor availability outside my network. Then i learnt about Beszel which was like Zabbix in some way, but very simple. I deployed Beszel externally too on really cheap VPS (5 PLN for lifetime), because Beszel is really really lightweight.
(DIR) Post #AzQuPu1DUKxQKFG8KO by cichy1173@mastodon.social
2025-10-21T09:15:37Z
1 likes, 0 repeats
@louis But Uptime Kuma started to be really problematic. First, lack of possibility to maintain in using code, then some false alerts that started to shown up everyday. I read about them and they have been labeled by the author as `wontfix`. So I started to look for some other solution that will not only monitor websites, but also hosts in more customizable way.
(DIR) Post #AzQuQ09wenVLN3OBlY by cichy1173@mastodon.social
2025-10-21T09:17:21Z
1 likes, 0 repeats
@louis I had some experience with #Zabbix thanks to my job. So i thought that It would be cool to deploy it and use it also for monitoring my homelab. First, I deployed it on Proxmox (2 VMs - one with Server and Frontend, second with db) and I started using this for host monitoring. And it shown up as really cool solution, especially with templates for Home Assistant etc. I also shown this to my friends (cc @morganmlg ) and I invited them to project where we will share one external Zabbix
(DIR) Post #AzQuQ6qhmjH27PRQFU by cichy1173@mastodon.social
2025-10-21T09:19:36Z
1 likes, 0 repeats
@louis @morganmlg And just two weeks ago we deployed Zabbix on @hetzner using Opentofu. Arch is almost the same - Two compute instances, where first hosts Zabbix-Server and Zabbix-Frontend and second hosts MariaDB. We also use firewall and Cloudflare whitelisting IP list. Everyone of us has private user that have access only to his hosts (thanks to host groups) and every of us has deployed Zabbix Proxy in private network that gathers data and sends it to Zabbix Server using Active Proxy func.
(DIR) Post #AzQv9KeFKLOZwlB47c by louis@social.louis-vallat.dev
2025-10-21T12:03:48.317Z
0 likes, 0 repeats
@cichy1173@mastodon.social Nice setup! :neocat_cool_fingerguns: I tend to forget that Zabbix has a Proxy system for remote/inaccessible networks, and using it for sharing a monitoring system with friends is a great idea! :noted:Personally, I use a mix of active agents on most of the Linux hosts (wanted to have an active client/passive server/push topology) and SNMP for the rest (i.e. routers). :neocat_science: