How do you all monitor your server performance? reddit / selfhosted ---------------------------------------------------------------------- Michaelscarn69- · 2023-10-22 10:15 As in, when I watched YouTube tutorials, I often see YouTubers have a small widget on their desktop giving them an overview of their ram usage, security level, etc. What apps do you all use to track this? ---------------------------------------------------------------------- suddenlypenguins · 2023-10-22 10:18 If my girlfriend is annoyed because the lights won't come on then urgent maintenance is required. ---------------------------------------------------------------------- Affectionate-Fig-805 · 2023-10-22 10:20 zabbix ---------------------------------------------------------------------- bobbarker4444 · 2023-10-22 10:23 I just check the proxmox dashboard every now and then. Honestly if everything is working I'm not too worried about exact ram levels at any given moment ---------------------------------------------------------------------- Pesfreak92 · 2023-10-22 10:24 Uptime Kuma and Grafana. Uptime Kuna to monitor if a service is up and running and Grafana to monitor the host like CPU, RAM, SSD usage etc. ---------------------------------------------------------------------- Dizzybro · 2023-10-22 10:33 The fastest way? Probably netdata ---------------------------------------------------------------------- ElevenNotes · 2023-10-22 10:34 Netdata, monitoring a few thousand servers (virtual) that way. ---------------------------------------------------------------------- Reasonable-Ladder300 · 2023-10-22 10:45 Same here, also have some autoscaling mechanisms set up in docker swarm to scale certain services in case the load is high ---------------------------------------------------------------------- AstrologicalMob · 2023-10-22 10:56 I currently use thr classic "Hu seems slow, *checks basic things like disk usage and process CPU/RAM usage* I'll do a reboot to fix it for now". ---------------------------------------------------------------------- DJTopNotch · 2023-10-22 11:09 I use this method for monitoring my adguard home instance. Pretty reliable alert-monitor. ---------------------------------------------------------------------- squadfi · 2023-10-22 11:18 I personally use Influxdb , telegraf and grafana ---------------------------------------------------------------------- jln_brtn · 2023-10-22 11:18 Nobody mentioned htop 🤔 ---------------------------------------------------------------------- Pabsilon · 2023-10-22 11:19 Glances and some charts in home assistant. That way I can also notify myself one way or another. ---------------------------------------------------------------------- thekrautboy · 2023-10-22 11:23 Just to make sure: You are aware that a search option here exists, yes? And you keep refusing to use it for whatever reason? ---------------------------------------------------------------------- thekrautboy · 2023-10-22 11:24 htop is a selfhosted service? ---------------------------------------------------------------------- The_Axelander · 2023-10-22 11:25 I use checkmk with notifications to a telegram bot ---------------------------------------------------------------------- jln_brtn · 2023-10-22 11:26 For sure: [https://github.com/htop-dev/htop](https://github.com/htop- dev/htop) ---------------------------------------------------------------------- thekrautboy · 2023-10-22 11:28 I know what htop is, and no its not a selfhosted service. ---------------------------------------------------------------------- wasabi_chips · 2023-10-22 11:31 Netdata ---------------------------------------------------------------------- borouhin · 2023-10-22 11:31 Alerts are much more important than fancy dashboards. You won't be staring at your dashboard 24/7 and you probably won't be staring at it when bad things happen. Creating your alert set not easy. Ideally, every problem you encounter should be preceded by corresponding alert, and no alert should be false positive (require no action). So if you either have a problem without being alerted from your monitoring, or get an alert which requires no action - you should sit down and think carefully what should be changed in your alerts. As for tools - I recommend Prometheus+Grafana. No need for separate AletrManager, as many guides recommend, recent versions of Grafana have excellent built-in alerting. Don't use those ready-to-use dashboards, start from scratch, you need to understand PromQL to set everything up efficiently. Start with a simple dashboard (and alerts!) just for generic server health (node exporter), then add exporters for your specific services, network devices (snmp), remote hosts (blackbox), SSL certs etc. etc. Then write your own exporters for what you haven't found :) ---------------------------------------------------------------------- Xiakit · 2023-10-22 11:34 Easiest to setup uptime kuma and netdata ---------------------------------------------------------------------- SadanielsVD · 2023-10-22 12:00 This. If you have more servers you can also get them all connected to a single UI where you can see all the Infos at once. With netdata cloud ---------------------------------------------------------------------- dom9301k · 2023-10-22 12:01 Prometheus + Grafana, the same I use at my job. ---------------------------------------------------------------------- LNDN91 · 2023-10-22 12:04 Rainmeter if it's directly on their desktop/background. ---------------------------------------------------------------------- krysinello · 2023-10-22 12:10 Grafana. Have alerts set up and get data with node exporter and cadvisor with some other containers giving some metrics. I have alerts setup and they just ping me on a discord server I setup. High cpu and temps low disk space memory things like that. Mostly get high CPU or temp alerts and that's usually when plex does its automated things at 4am. ---------------------------------------------------------------------- Spaceman_Splff · 2023-10-22 12:10 Just set this up yesterday. I used a parent node and then have all my vms point to that. Took like an hour to figure it out ---------------------------------------------------------------------- atheken · 2023-10-22 12:11 One thing about using Prometheus alerting is that it’s one less link in the chain that can break, and you can also keep your alerting configs in source control. So it’s a little less “click-ops,” but easier to reproduce if you need to rebuild it at a later date. ---------------------------------------------------------------------- scotrod · 2023-10-22 12:15 Hey, did you use the cloud functionality or not? I'm tryna go all local with parent-child kind of capability but so far unable to. ---------------------------------------------------------------------- xardoniak · 2023-10-22 12:16 I use Uptime Kuma to monitor particular services and NetData for server performance. I then pipe the alerts through to Pushover ---------------------------------------------------------------------- ruffcodes · 2023-10-22 12:19 i am stuck on the, find a girlfreind step, help ---------------------------------------------------------------------- CactusBoyScout · 2023-10-22 12:19 Yeah if my girlfriend reports that the fans are loud, I know I’ve got an issue. ---------------------------------------------------------------------- Spaceman_Splff · 2023-10-22 12:20 The parent still is visible to the cloud portal. My understanding is the data all resides local, but when you login to their cloud portal, it connects to the parent to display the information. I’m still playing with it to confirm. My parent node shows all the child nodes on the local interface but the cloud still shows them all. ---------------------------------------------------------------------- Nagashitw · 2023-10-22 12:22 This is me. Can't hurt to just do a reboot ---------------------------------------------------------------------- nitsky416 · 2023-10-22 12:30 Try finding a boyfriend ---------------------------------------------------------------------- xX__M_E_K__Xx · 2023-10-22 12:31 https://github.com/nicolargo/glances ? ---------------------------------------------------------------------- trisanachandler · 2023-10-22 12:38 Honestly my load is so light I don't bother monitoring performance. Uptime kuma for uptime, I used to use prtg and uptime robot when I ran a heavier stack before I switched to an all docker workload. ---------------------------------------------------------------------- BloodyIron · 2023-10-22 12:46 libreNMS is the tool I use, and it connects to systems primarily via SNMP (use v3, do not use v1 or v2c). ---------------------------------------------------------------------- BloodyIron · 2023-10-22 12:46 Have you tried libreNMS? ---------------------------------------------------------------------- borouhin · 2023-10-22 12:54 When you have several Prometheus instances (HA or in different datacenters), setting up separate AlertManagers for each of them is a good idea. But as OP is only beginning his journey to monitoring, I guess he will be setting up a single server with both Prometheus and Grafana on it. In this scenario a separate AlertManager doesn't add reliability, but adds complexity. As for source control, you can write a simple script using Grafana API to export alert rules (and dashboards as well) and push them to git. Not ideal, sure, but it will work. Anyway, it's never too late to go further and add AlertManager, Loki, Mimir and whatever else. But to flatten the learning curve I'd recommend starting with Grafana alerts that are much more user- friendly. ---------------------------------------------------------------------- whiskyfles · 2023-10-22 12:54 I second Zabbix. Bonus points is that we use it at work, so I also win some knowledge about it. ---------------------------------------------------------------------- gold76 · 2023-10-22 12:56 Influx/telegraf/grafana stack. I have all 3 on one server and then I put just telegraf on the others to send data into influx. Works great for monitoring things like usage. You can also bring in sysstat. I have some custom apps as well where each time they run I record the execution time and peak memory in a database. This lets me go back over time and see where something improved or got worse. I can get a time stamp and go look at gitea commits to see what I was messing with. ---------------------------------------------------------------------- MalcolmY · 2023-10-22 12:57 Different toolchain, won't compile. Exit with error 1. hehe. ---------------------------------------------------------------------- Pabsilon · 2023-10-22 12:58 Yep! It has a native HA integration. ---------------------------------------------------------------------- Charming-Molasses-22 · 2023-10-22 12:59 I don't check it all the time like a maniac but I have a glances docker running on my main server. ---------------------------------------------------------------------- PaulBag4 · 2023-10-22 13:11 +1 for this method. Any monitoring platforms need maintaining. Girlfriend moaning appears to be self-sustaining ---------------------------------------------------------------------- speculatrix · 2023-10-22 13:12 I use Zabbix. Runs fine in a relatively small VM. Easy to write plugins. ---------------------------------------------------------------------- Majestic-Contract-42 · 2023-10-22 13:13 If one of my users ever complained about anything I would possibly look into it, otherwise it all works so I don't waste life energy on that. ---------------------------------------------------------------------- speculatrix · 2023-10-22 13:14 I third Zabbix. Runs fine in a relatively small VM. Easy to write plugins. It's mature and solid. I used to use munin, but the death of the munin exchange for plugins some years ago was a big set back. ---------------------------------------------------------------------- speculatrix · 2023-10-22 13:16 Bashtop is pretty. But not scalable. ---------------------------------------------------------------------- CryptoBaub · 2023-10-22 13:16 Yours is self sustaining? Mine always seems to be in resource conflict with time for system maintenance ---------------------------------------------------------------------- Theon · 2023-10-22 13:22 Netdata, I've meant to look into Grafana but it always seemed way too overcomplicated and heavy for my purposes. Maybe one day, though... ---------------------------------------------------------------------- d-cent · 2023-10-22 13:32 There has to be a linux distro for that by now ---------------------------------------------------------------------- how_now_brown_cow · 2023-10-22 13:34 TICK stack is the only answer ---------------------------------------------------------------------- Do_TheEvolution · 2023-10-22 13:41 [Prometheus + Grafana + Loki](https://github.com/DoTheEvo/selfhosted- apps-docker/tree/master/prometheus_grafana_loki) It is bit difficult at start, but really in the end you can monitor and get notification on anything thats happening on your system. ---------------------------------------------------------------------- JohnnyGiraffe · 2023-10-22 13:47 RHELeanor ---------------------------------------------------------------------- HCharlesB · 2023-10-22 14:02 Checkmk (Raw - free version.) Some setup aspects are a bit annoying (wants to monitor every last ZFS dataset and takes too long to 'ignore' them one by one.) It does alert me to things that could cause issues, like the boot partition almost full. I run it in a Docker container on my (primarily) file server. ---------------------------------------------------------------------- TheDeepTech · 2023-10-22 14:10 I recommend Checkmk. https://checkmk.com/ ---------------------------------------------------------------------- damn_the_bad_luck · 2023-10-22 14:12 When the fan gets loud enough to hear, I'll check it :P ---------------------------------------------------------------------- TheDeepTech · 2023-10-22 14:14 I use this as well! Works well and has built in intelligence for thresholds. ---------------------------------------------------------------------- ilikestreet · 2023-10-22 14:18 same here🤣 ---------------------------------------------------------------------- opensrcdev · 2023-10-22 14:32 InfluxDB metrics server and Telegraf agent to collect metrics ---------------------------------------------------------------------- opensrcdev · 2023-10-22 14:35 Glances is really nice. I've been using btop more recently though. ---------------------------------------------------------------------- MrWizard1979 · 2023-10-22 14:37 I introduced zabbix at work because I used it at home. It's nice to know services are down before the users complain ---------------------------------------------------------------------- dibu28 · 2023-10-22 14:47 Or the Docker Container 🤣 ---------------------------------------------------------------------- dibu28 · 2023-10-22 14:51 Windows Server? ) ---------------------------------------------------------------------- dibu28 · 2023-10-22 14:52 Btop ---------------------------------------------------------------------- djbon2112 · 2023-10-22 14:52 I second CMK. A TICK stack is unwieldy, Grafana takes a lot of setup, and all of this assumes you both know what to monitor and get stats on it. CMK by contrast is plug and play. Install the server on a VM or host, install thr agent on your other systems, and you're good to go. ---------------------------------------------------------------------- talent_deprived · 2023-10-22 15:07 I use sar for historical, my own scripts running under cron on the hosts for specific things I'm interested in keeping an eye on and my on scripts under cron on my monitoring machines for alerting me when something's wrong. I don't use a dashboard. ---------------------------------------------------------------------- SpongederpSquarefap · 2023-10-22 15:10 I have, works great but Zabbix has more work going into it overall and can scale a bit easier I think, so I'd go with that ---------------------------------------------------------------------- SpongederpSquarefap · 2023-10-22 15:10 Another +1 for Zabbix Not hard to set up and scales fairly easily with proxies ---------------------------------------------------------------------- boomertsfx · 2023-10-22 15:16 Can you do this locally? I remember they had some way to aggregate metrics. Netdata has gotten more leaky and unstable for me lately… when they first started it was awesome ---------------------------------------------------------------------- -rwsr-xr-x · 2023-10-22 15:24 [Prometheus](https://prometheus.io/), [Grafana](https://prometheus.io/docs/visualization/grafana/) and [Netdata](https://learn.netdata.cloud/) on my homelab (about 30 active nodes, another ~50 or so idle/powered down and only used as needed) ---------------------------------------------------------------------- Mother_Construction2 · 2023-10-22 15:25 I know that it needs a fix when my dad complaining that he can’t watch TV and the rolling door doesn’t open in the morning. ---------------------------------------------------------------------- -rwsr-xr-x · 2023-10-22 15:26 > Creating your alert set is not easy. Ideally, every problem you encounter should be preceded by corresponding alert, and no alert should be false positive (requiring no action). I doth protest! It's [trivially easy](https://grafana.com/blog/2021/06/08/grafana-8.0-unified-grafana- and-prometheus-alerts-live-streaming-new-visualizations-and-more/), if you use Prometheus and Grafana! ---------------------------------------------------------------------- basicallybasshead · 2023-10-22 15:28 Zabbix. Aslo for Windows, it could be Rainmeter [https://www.rainmeter.net/](https://www.rainmeter.net/) or HWiNFO [https://www.hwinfo.com/](https://www.hwinfo.com/). For Linux, Conky. ---------------------------------------------------------------------- Frostlike2816 · 2023-10-22 15:31 I have never felt more seen ---------------------------------------------------------------------- GoobyFRS · 2023-10-22 15:35 NewRelic ---------------------------------------------------------------------- M3ch4n1c4lH0td0g · 2023-10-22 15:36 Netdata ---------------------------------------------------------------------- BloodyIron · 2023-10-22 15:46 > scale a bit easier I think How so? ---------------------------------------------------------------------- surpyc · 2023-10-22 15:48 Icinga ---------------------------------------------------------------------- SpongederpSquarefap · 2023-10-22 15:50 It can be, but the agent is excellent, super lightweight and extensible GUI is really fast with Nginx - did you use it with Apache? If you leverage Zabbix's auto discover rules too, it can auto monitor so much stuff ---------------------------------------------------------------------- paolobytee · 2023-10-22 16:07 Netdata ---------------------------------------------------------------------- Usual_Wallaby2524 · 2023-10-22 16:16 Customer satisfaction is essential for a frictionless work environment... ---------------------------------------------------------------------- BloodyIron · 2023-10-22 16:20 I'm not a fan of a specialised agent in this function when SNMP gives me everything I want (and at times more). It has been a long time since I've used Zabbix, I don't know its current state, but I also haven't been motivated to switch at all. libreNMS has autodiscover capabilities too, btw. ---------------------------------------------------------------------- SpongederpSquarefap · 2023-10-22 16:22 SNMP is great for your network devices, not so great for VMs ---------------------------------------------------------------------- lordjustice17 · 2023-10-22 16:25 Same here. Glances with key sensors sent up in my HA dashboard. Plan on adding notifications for high temps eventually as well. ---------------------------------------------------------------------- BloodyIron · 2023-10-22 16:25 Actually it is. Every single one of my VMs give stats via SNMP. Every single device in my environment I want stats on (except k8s stuff, mind you) is via SNMP, and I not only get all the details I want, I get very granular details. ---------------------------------------------------------------------- SpongederpSquarefap · 2023-10-22 16:27 How extensible is that if you want custom checks? I've always seen Windows and SNMP as a pain - CPU usage was spiking higher during checks last time I used it a few years ago ---------------------------------------------------------------------- JoeB- · 2023-10-22 16:36 I use Telegraf + InfluxDB + Grafana for monitoring my home network and systems. Grafana has a learning curve for building panels and dashboards, but is incredibly flexible. I use it for more than server performance. I have a dual-monitor "kiosk" (old Mac mini) in my office displaying two Grafana dashboards. These are: [**Network/Power/Storage**](https://i.imgur.com/5PXPtvB.png) showing: * firewall block events & sources for last 12 hrs (from pfSense via Elasticsearch), * current UPS statuses and power usage for last 12 hrs (Telegraf apcupsd plugin -> InfluxDB), * WAN traffic for last 12 hrs ( from pfSense via Telegraf -> InfluxDB), * current DHCP clients (custom Python script -> MySQL), and * current drive and RAID pool health (custom Python scripts -> MySQL) [**Server sensors and performance**](https://i.imgur.com/5PXPtvB.png) showing: * current status of important cron jobs (using Healthchecks -> Prometheus), * current server CPU usage and temps, and memory usage (Telegraf -> InfluxDB) * server host CPU usage and temps, and memory usage for last 3 hrs (Telegraf -> InfluxDB) * Proxmox VM CPU and memory usage for last 3 hrs (Proxmox -> InfluxDB) * Docker container CPU and memory usage for last 3 hrs (Telegraf Docker plugin -> InfluxDB) Netdata works really well for system performance for Linux and can be installed from the default repositories of major distributions. ---------------------------------------------------------------------- Michaelscarn69- · 2023-10-22 16:43 I don’t get it :( ---------------------------------------------------------------------- Michaelscarn69- · 2023-10-22 16:44 I’ll look into it. Thank you. ---------------------------------------------------------------------- Michaelscarn69- · 2023-10-22 16:45 Thank you for this. I think I need a deeper understanding of Prometheus. I’ll look into it. You are awesome ---------------------------------------------------------------------- Michaelscarn69- · 2023-10-22 16:45 I’ll look into this too. Thank you. ---------------------------------------------------------------------- Revelmonger · 2023-10-22 16:49 Would you mind if I ask what your position title is? I've been looking for a job and id love one were I can use my self hosting skills. ---------------------------------------------------------------------- MothGirlMusic · 2023-10-22 16:50 We use zabbix here. Zabbix is amazing and we put it in all of our templates so any new servers and hosts pop up on zabbix dashboard preconfigured just like that. For logs and security we use an Elastik "ELK stack" which gives us a heads up if anything is wrong in the logs, and zabbix gives us a head up of the systems health all together. Between the two, our health monitor panel combines the two windows so we can see full server health and any problems right there as a todo list for the IT team ---------------------------------------------------------------------- 5c044 · 2023-10-22 16:55 I use Home Assistant already. They have a plugin for glances. I guess all I'm interested in is cpu temp and load. Any changes =somethings up ---------------------------------------------------------------------- whiskyfles · 2023-10-22 16:59 My title currently is 'Junior Linux Engineer'. I work at a company that does webhosting (managed) for clients all over Europe. I actually got this job because I could show so much about selfhosting etc. Selfhosting / Homelabbing is such a great place to start out with this, since most of the stuff thats used in the Enterprise hosting- world (things like: Kubernetes, Docker, Ansible, Virtualization (KVM, QEMU) is just Open-Source and free to use. ---------------------------------------------------------------------- kindrudekid · 2023-10-22 17:06 If get ahead of it by getting extra. Need 16 gb of ram and 8 cores ? Well let me add 64 gb to my cart and 12 core CPU. Hasn’t failed me ---------------------------------------------------------------------- sun_in_the_winter · 2023-10-22 17:12 Telegraf graphite grafana + telegram notifications ---------------------------------------------------------------------- MacGyver4711 · 2023-10-22 17:24 CheckMK for general monitoring, Grafana/Prometheus for Proxmox- cluster, Wazuh for IDS-purposes and UptimeKuma for general uptime on services. It's not like it's necessary, but it's nice to tinker in my homelab before implementing the same services on a "professional level" at work. My HomeAssistant is stable, so wifey is not being used as a monitor ;-) ---------------------------------------------------------------------- NurEineSockenpuppe · 2023-10-22 17:48 If nobody complains everything is fine. I run music bots game servers mostly so even if something fails it‘s nothing really that critical. When I‘m at home I usually ssh into my main host machine and have btop running on my second monitor. It shows me the processes, ram , cpu, network and disk space. Oh yeah and load averages. It also looks super pretty and supports skins :) ---------------------------------------------------------------------- Cylian91460 · 2023-10-22 17:51 I use btop, I use arch btw ---------------------------------------------------------------------- borouhin · 2023-10-22 17:54 Good luck, if you get into it, you'll be unable to stop. Perfecting your monitoring system is a kind of mania :) One more advice for another kind of monitoring. When you are installing / configuring something on your server - it's handy if you can monitor it's resource usage in real time. And that's why I use MobaXterm as my terminal program. It has many drawbacks, and competitors such as XShell, RoyalTS or Tabby look better in many ways... but it has one killer feature. It shows a status bar with current server load (CPU, RAM, disk usage, traffic) right below your SSH session, so that you don't have to switch to another window to see the effect of your actions. Saved me a lot of potential headache. ---------------------------------------------------------------------- dinosaurdynasty · 2023-10-22 17:55 I don't find it valuable so I don't. (Maybe run `top` as needed.) ---------------------------------------------------------------------- Cylian91460 · 2023-10-22 17:56 >Alerts are much more important than fancy dashboards. It depends, If you have to install lot of stuff or manage a lot of thing it's a good idea to have one but if you mainly do maintenance and you want to have something reliable yes you should have an alerts, for exemple I don't have a lot of thing install and doesn't rly care about reliability so I do everything in terminal, I use arch btw ---------------------------------------------------------------------- dangernoodle01 · 2023-10-22 17:57 Grafana doesn't monitor anything, grafana visualizes data. Maybe you meant Prometheus or zabbix? ---------------------------------------------------------------------- Large_Yams · 2023-10-22 18:04 Where do you run htop? ---------------------------------------------------------------------- Large_Yams · 2023-10-22 18:05 I don't track their performance, I just track if they're up or down. I use uptimekuma running on a free tier of fly.io so I can tell if my cluster had a catastrophic failure. There's no point in the alerting system running on the same system. ---------------------------------------------------------------------- thekrautboy · 2023-10-22 18:06 Do you know the difference between a simple standard application and a selfhosted service? ---------------------------------------------------------------------- maximus459 · 2023-10-22 18:19 Observium.. If it's just one server, Netdata is a better option.. ---------------------------------------------------------------------- LumePart · 2023-10-22 18:23 Zabbix for hardware, certificate monitoring Prometheus for service monitoring (e.g how many are actually using my Jellyfin server, so i know if I need to scale etc.) ---------------------------------------------------------------------- Michaelscarn69- · 2023-10-22 18:36 Thank you for this. I appreciate the support. ---------------------------------------------------------------------- xupetas · 2023-10-22 18:51 Nagios for service/QOS, Grafana for dashboarding for some items more specific. Planning on eventually switching to zabbix but nagios is so simple that i feel having a hard time justifying moving over 400 monitored services to it ---------------------------------------------------------------------- roh4 · 2023-10-22 18:52 First for PRTG. ---------------------------------------------------------------------- Nondv · 2023-10-22 18:59 a Docker Basement ---------------------------------------------------------------------- BloodyIron · 2023-10-22 19:12 I don't run Windows in the spaces where libreNMS exists, so I can't speak to that facet. It wouldn't surprise me if Windows had a poor implementation for yet another thing that's standardised by everyone else (SNMP in this case, NFS is another example). ---------------------------------------------------------------------- idontmeanmaybe · 2023-10-22 19:30 Yes. I have multiples servers sending to my nas box. The db and gui run there. I don’t use the cloud at all. ---------------------------------------------------------------------- pabskamai · 2023-10-22 19:32 🤣 We need awards back, greedy Reddit ---------------------------------------------------------------------- BouncyPancake · 2023-10-22 19:34 If its down, I assume performance is bad ---------------------------------------------------------------------- daniel280187 · 2023-10-22 19:45 >Network/Power/Storage Pretty cool dashboards. I liked the DHCP clients info, does it also report DHCP reservations? Where do you do DHCP, on the PFSense or somewhere else? ---------------------------------------------------------------------- lestrenched · 2023-10-22 20:05 I came across `monit` recently, seems nice ---------------------------------------------------------------------- joshiegy · 2023-10-22 20:14 Zabbix is... Trash. I'm sorry. Its old, its slow, its confusing. Ever tried something modern, mor real time, such as Prometheus, fluent bit, sensu, telegraf? ---------------------------------------------------------------------- joshiegy · 2023-10-22 20:15 No... Why? Its old, its trash. Might get hate for it, but its just not good. ---------------------------------------------------------------------- joshiegy · 2023-10-22 20:16 I'm running a tick stack with a couple of thousands of servers - way less CPU usage than checkmk/nagios or anything else from the previous millennium ... ---------------------------------------------------------------------- Windows_XP2 · 2023-10-22 20:34 Yeah but I don't think that any rational person is going to date me. ---------------------------------------------------------------------- thibmaek · 2023-10-22 20:34 Quick checks: Proxmox dashboard, htop or glances, Portainer Extensive monitoring: Prometheus (node-exporter), Rsyslog server, Loki, Grafana, Uptime Kuma, Alertmanager (via Gotify) ---------------------------------------------------------------------- FortunatelyLethal · 2023-10-22 20:35 I mean, you aren’t supported anymore. Try updating to a newer OS ---------------------------------------------------------------------- tenekev · 2023-10-22 20:42 I though people used lube for that... ---------------------------------------------------------------------- djbon2112 · 2023-10-22 20:43 How do you solve the problem of runaway memory usage? Even monitoring a few dozen hosts, memory usage would grow to many GB and continue to grow indefinitely until it OOM'd, and from my reading Influx has no way to prevent this. ---------------------------------------------------------------------- __aa__aa · 2023-10-22 20:59 I literally tried all. Nagios is the best one ---------------------------------------------------------------------- joshiegy · 2023-10-22 21:01 Have you had runaway memory problems with influx, or your apps? ---------------------------------------------------------------------- weilah_ · 2023-10-22 21:13 Uptime Kuma for my services Netdata + Prometheus + Grafana for server health (alerts and visualization) ---------------------------------------------------------------------- Spaceman_Splff · 2023-10-22 21:15 I don’t know if I’ll keep running this. Already the child nodes are complaining about increase write delays since installing the agents on them. ---------------------------------------------------------------------- djbon2112 · 2023-10-22 21:30 Specifically with Influx. ---------------------------------------------------------------------- JoeB- · 2023-10-22 21:59 >does it also report DHCP reservations? Thanks, and yes, **Type** "*static*" are DHCP reservations. ​ >Where do you do DHCP, on the PFSense or somewhere else? Yes, on pfSense. I use the Python function written by [pletch/scrape\ _pfsense\_dhcp\_leases.py](https://gist.github.com/pletch/037a4a01c956 88fff65752379534455f) (on Github) that scrapes the pfSense **status\_dhcp\_leases.php** page. Then added my own function for querying my TP-Link APs using SNMP to determine which AP a wireless DHCP client is connected to. I can throw the script up on Dropbox if you are interested. I am mediocre at writing Python, so it is pretty specific to my environment. ---------------------------------------------------------------------- io-x · 2023-10-22 23:27 I was looking at loki+grafana. is prometheus a replacement for loki in this setup and is it preferred? ---------------------------------------------------------------------- borouhin · 2023-10-22 23:41 No, they serve different purposes. Loki is for logs, Prometheus is for metrics. Grafana helps to visualize data from both. ---------------------------------------------------------------------- Makeshift27015 · 2023-10-22 23:52 I used librenms a few years back at my company. We found it great for a hundred machines or so, but after you started trying to scale it to several hundred (even with dedicated nodes and such) it would start falling over in painful ways. Would be great for an SMB or home network though ---------------------------------------------------------------------- 2000nesman · 2023-10-23 00:09 Prometheus and grafana ---------------------------------------------------------------------- dlbpeon · 2023-10-23 00:17 Xymon/Nagios. I'm old school. I've been using them since it was BigBrother/NetSaint. Simple monitors with email alerts to problems ---------------------------------------------------------------------- dlbpeon · 2023-10-23 00:21 Some projects are just feature complete and don't need updating. Us old school users just know all you need is an email alert to notify people of problems. No need for anything fancy, but users today suffer from FOMO if they aren't updating their server every other day! ---------------------------------------------------------------------- BloodyIron · 2023-10-23 01:16 Did you try distributed polling? ---------------------------------------------------------------------- Nasach · 2023-10-23 01:18 I use net data for both dashboards and alerts. Works great and easy to setup. ---------------------------------------------------------------------- Affectionate-Fig-805 · 2023-10-23 01:39 I'm sorry but I disagree. Zabbix is a complete monitoring and alerting tool. The vast amount of OS and devices supported alone makes zabbix a very valuable tool in any environment. The alerts of zabbix is almost reatime too, we've been using it for years to monitor our servers, network equipment, internet lines, website performance, etc and it never failed to provide the necessary alerts (sms/email) when needed. But then again, we are more concerned on the alerts given and not on the dashboard design. ---------------------------------------------------------------------- AttitudeImportant585 · 2023-10-23 01:58 When you've got a lot of variables, especially when dealing with a distributed system, that importance leans the other way. Visualization and analytics are practically required to debug and tune large systems ---------------------------------------------------------------------- lunakoa · 2023-10-23 03:01 Its not well liked but I use nagios core for alerts and jump to grafana which has data in prometheus, influxdb, and mysql backend for trends like cpu usage hard drive Temps etc. ---------------------------------------------------------------------- Jacksaur · 2023-10-23 03:23 What about InfluxDB? I hear that mentioned around Grafana a lot. ---------------------------------------------------------------------- weller_rocks · 2023-10-23 03:25 easiest by far to set up, plenty of metrics [https://www.netdata.cloud/](https://www.netdata.cloud/) ---------------------------------------------------------------------- weller_rocks · 2023-10-23 03:27 agreed .... BY FAR the fastest. Easiest learning curve as well ---------------------------------------------------------------------- weller_rocks · 2023-10-23 03:31 I thought the same thing but it's not bad actually, there are some pre build dashboards you can import for common metrics from Linux, windows, firewalls etc ..... netdata is much better though (IMHO) ---------------------------------------------------------------------- borouhin · 2023-10-23 03:34 InfluxDB is just a storage. If you have a service that saves metrics to InfluxDB (IIRC, Proxmox can do that), Grafana can read it from there. Grafana can aggregate data from many sources, Prometheus+Loki+InfluxDB+even queries to arbitrary JSON APIs etc. ---------------------------------------------------------------------- PirateParley · 2023-10-23 04:13 You got issue. Time to get rid off girlfriend. Homelab staying. ---------------------------------------------------------------------- CactusBoyScout · 2023-10-23 04:15 No I mean she’s like an early warning system for server issues. She’s the canary in my digital coal mine. ---------------------------------------------------------------------- PirateParley · 2023-10-23 04:18 oh definitely keeper.. ---------------------------------------------------------------------- Michaelscarn69- · 2023-10-23 05:16 Everything you have mentioned thus far is news to me and I guess I have to research on pretty much everything. I’m more intrigued on the alert feature. I hope NetworkChuck has a good tutorial on how to set those up. ---------------------------------------------------------------------- AlteranNox · 2023-10-23 06:22 Why did they get rid of awards? ---------------------------------------------------------------------- Dogeek · 2023-10-23 07:33 Oh lord, I have so much info to give ! For the setup, it's running on kubernetes 1.28.2, so YMMV. My monitoring stack is : - Grafana -- Dashboards - Alertmanager -- Alerting - Prometheus -- Time series Database - Loki -- Logs database - Promtail -- Log collector - Mimir -- Long term metrics&logs storage - Tempo -- Datadog APM, but with Grafana, allows you to track requests through a network of services, invaluable to link your reverse proxy, to your apps, to your SSO to your database... - SMTP Relay -- A homemade SMTP relay that eases setting up mail alerts, allows me to push mail through mailjet using my domain - Node-exporter -- exports metrics for the server - Exportarr -- exports metrics for sonarr/radarr etc - pihole-exporter -- exports pihole metrics for prometheus scraping - smart-exporter -- exports S.M.A.R.T metrics (for HDD health) - ntfy -- for notifications to my phone (other than mail) The rest is pretty much the same, if the service exports prometheus metrics by default, I use that, and write a `ServiceMonitor` and a `Service` manifest for that, it usually looks like that apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: traefik labels: app.kubernetes.io/component: traefik app.kubernetes.io/instance: traefik app.kubernetes.io/managed-by: kustomize app.kubernetes.io/name: traefik app.kubernetes.io/part-of: traefik spec: selector: matchLabels: app.kubernetes.io/name: traefik-metrics endpoints: - port: metrics interval: 30s path: /metrics scheme: http tlsConfig: insecureSkipVerify: true namespaceSelector: matchNames: - traefik --- apiVersion: v1 kind: Service metadata: name: traefik-metrics namespace: traefik labels: app.kubernetes.io/name: traefik-metrics spec: type: ClusterIP ports: - protocol: TCP name: metrics port: 8082 selector: app.kubernetes.io/name: traefik If the app doesn't include a prometheus endpoint, I just find an existing exporter for that app, most popular ones have that, and ready made grafana dashboards. For alerting, I create `PrometheusRule` object with the prometheus query and the message to alert me (depending on the severity, it's either a mail for med-low severity incidents, phone notification for high sev). I try to keep mails / notifications to a minimum, just alerts on load, CPU, RAM, and potential SMART errors as well give me alerts. ---------------------------------------------------------------------- Makeshift27015 · 2023-10-23 07:40 Yep, if I remember correctly we had a few hosts that we had mediocre connections to, and any sort of latency to hosts would significantly slow down librenms since it then delays polling for everything else. Entirely possible we set it up wrong though, and it may have improved since ---------------------------------------------------------------------- Michaelscarn69- · 2023-10-23 09:31 Yes ---------------------------------------------------------------------- Olleye · 2023-10-23 09:37 Use PRTG, up until 100 sensors it’s free. Best Monitoring tool ever ☝🏻🙂 ---------------------------------------------------------------------- pabskamai · 2023-10-23 11:02 They want us to re buy then with real money ey, I had a lot of karma from back in the streaming days. ---------------------------------------------------------------------- thekrautboy · 2023-10-23 11:34 Good attitude. ---------------------------------------------------------------------- Michaelscarn69- · 2023-10-23 12:09 Thanks ---------------------------------------------------------------------- chuchodavids · 2023-10-23 12:41 None. There is no need for a performance monitor for my home lab. I just have an alert if one of my main three services is down. That is all i need. ---------------------------------------------------------------------- BloodyIron · 2023-10-23 13:00 When-ish was this? And why were the links "acceptably intermittent" or something like that? I do appreciate your honest response here :) ---------------------------------------------------------------------- xX__M_E_K__Xx · 2023-10-23 13:08 It's too good to be true :D May I ask you share your config on the glances side to have a starting point? ---------------------------------------------------------------------- xX__M_E_K__Xx · 2023-10-23 13:08 May I ask you share your config on the glances side to have a starting point? ---------------------------------------------------------------------- GorillaAU · 2023-10-23 13:47 I can't help as I have upgraded to Wife 1.0. ---------------------------------------------------------------------- servergeek82 · 2023-10-23 13:51 Glances, uptime-kuma, and back end script that reboots service if down. If it doesn't work I get a notification via gotify. Simple and sweet ---------------------------------------------------------------------- Pabsilon · 2023-10-23 13:57 I'll get back to you when I'm back home ---------------------------------------------------------------------- lordjustice17 · 2023-10-23 14:01 I have it set up as a Docker container (documentation here: [https://github.com/nicolargo/glances/blob/master/README.rst#docker- the-fun-way](https://github.com/nicolargo/glances/blob/master/README.r st#docker-the-fun-way)), although it can be setup in a bunch of different ways including bare metal installs. Once you get it setup on whatever machine you're monitoring then it can be accessed via the web browser at [http://IPADDRESS:61208](http://IPADDRESS:61208). IIRC Home Assistant has a UI flow for setting it up which includes the option to select whichever sensors you're interested in. Personally I use RAM usage, CPU usage and temperature. For temperature I chose the one that generally seemed to cover the entire CPU temp rather than specific packages or cores. What sensors you'll have available will vary based on your hardware. ---------------------------------------------------------------------- UsernamesAreTooShort · 2023-10-23 14:23 This guy has a girlfriend. This guy has a home where lights are connected to a central server (smart bulbs certainly, because who wouldn't like to be able to say "Rhasspy, light all lights in bright blue and play megalovania"). If this central server ever goes down, the house lights will not function anymore, which is likely to be noticed by this guy's girlfriend first, since he's too busy playing Factorio to even care about being in the dark. (he was probably already in the dark when he started playing tho) The funny part is that the first thing that would tell him that his server is down is his girlfriend, meaning he has no "proper" monitoring of his system. Also I don't really monitor anything except a uptimerobot mail if my vps goes down. ---------------------------------------------------------------------- Usual_Wallaby2524 · 2023-10-23 17:03 It's hard to get in a fully virtualized environment you know... ---------------------------------------------------------------------- Usual_Wallaby2524 · 2023-10-23 17:04 How often do you recycle your canaries? Are they HA/DR protected perhaps? ---------------------------------------------------------------------- AlteranNox · 2023-10-23 17:42 I'm confused because awards were always something that the giver had to buy. ---------------------------------------------------------------------- Pesfreak92 · 2023-10-23 18:41 Yeah you are actually right. Prometheus gets the information and sends it to Grafana where I have Dashboards to show the data. ---------------------------------------------------------------------- Makeshift27015 · 2023-10-23 19:26 This would have been mid-2019, shortly before I left so I'm not sure how they got on with it after that. It's only now that I realise how long ago that was and how different LibreNMS probably is now, so my anecdote is quite dated. The actual connectivity to these hosts/devices wasn't _too_ awful in terms of speed, but the latency was 600ms+ and there was some older hardware that was not particularly speedy at outputting SNMP. I don't remember exactly how long it took to grab all the stats from some of the older Cisco switches, but it was definitely in the realm of double-digit seconds. ---------------------------------------------------------------------- pabskamai · 2023-10-23 20:30 I got a bunch of coins given to me, I was coin rich lol ---------------------------------------------------------------------- BloodyIron · 2023-10-23 20:53 Well the original motivation behind me asking about distributed polling is that libreNMS is known to be very successful in environments at the scale of tens of thousands of target devices, and larger. The latency you speak of is really not a shortcoming of libreNMS itself, as that kind of latency can wreak havoc on so many other things too. And appliances being slow to feed SNMP stats when requested, well that again sounds like a problem with that device, and not libreNMS. But I digress, this was more for discussion, and since you're long away from that environment, probably not all that helpful for you anyways ;) that is, unless you have a use-case for libreNMS lately. I've been using libreNMS continually I believe between now and well before 2019, and not quite a the scale you describe, but I wouldn't switch it for another tool (for these metrics). Reliable, fast, has given me huge value. Anyways, just wanted to hear your story, thanks for sharing! If there's any other details, I'm all ears! I myself rock it in a VM but I'll eventually migrate it into a k8s deployment (they have docker images btw), but I'm in no rush just yet ;) Have a nice day! ---------------------------------------------------------------------- Makeshift27015 · 2023-10-23 21:27 It is good to know that our config specifically was probably kinda terrible. If I find myself needing something like it again, I'll definitely consider it. ---------------------------------------------------------------------- AlteranNox · 2023-10-24 03:40 Yeah, but before they did that you had to buy awards to give to people. ---------------------------------------------------------------------- BloodyIron · 2023-10-24 04:08 Well I don't think I can reliably tell if the source of the trouble was specifically the libreNMS configuration (or _only_ that), as there could be other variables going on too. So I would not really want to say with any confidence what I think it "actually was" as the source of the problem. I can creatively come up with a bunch of possibilities that may not be true at all! But yeah, libreNMS _can_ do very well, and it certainly _can_ be misconfigured. As to your situation? I dunno! But I am glad to hear that you will ... "definitely consider it", so yay! :) I hope it was helpful info, either way. So thanks for hearing me out! \o/ ---------------------------------------------------------------------- Savancik · 2023-10-24 11:35 Girlfriend first Alert Manager second. Girlfriend is usually faster. ---------------------------------------------------------------------- akusei79 · 2023-10-26 17:50 This is an underrated and hilarious response ---------------------------------------------------------------------- nmp5 · 2023-12-20 22:47 Hey!! I absolutely love sysstat/sar, and that's what I've been using since I remember dealing with servers :) I'd love to convert the accumulated stats into Grafana. The thing is - I've never set up Grafana in the past. Do you have any nice tutorial how to set up sysstat with Grafana? Many thanks!