What SIMPLE log aggregator for homelab use? reddit / selfhosted ---------------------------------------------------------------------- domanpanda · 2023-11-21 08:10 First of all i DONT'T ask what DO YOU use, because i know some ppl have really advanced and fancy stuff at home. So if you want give advice please try to make it relevant to this post. Otherwise it wont be usefull much. Im looking for some simple log aggregator system which allows me to see logs in one place (like web panel). Logs i want to observe are: * syslogs from few machines * docker containers logs * logs in docker volumes (like nginx proxy configs) I dont need fancy filtering, querying, alerting, visualisations, webhooks etc. I just want to gather my logs in one place and be able to quickly check them out. For example ELK stack which i used in the past is complete opposite of this. Ive seen **Loki**, but it requires to learn graphQL and there are some weird issues with docker driver (like freezing all containers when loki goes down??). Ive seen **Graylog** tutorial by Lawrence Systems and it seems to be simpler on the surface, yet powerfull if needed. Im checking now how to monitor docker with it. Ive seen some swear by **splunk** \- that it's easy to setup - is it true? ---------------------------------------------------------------------- tcassaert · 2023-11-21 08:24 I read that you don't need all the fancy stuff, but [OpenObserve](https://openobserve.ai/) has all the fancy stuff and is still easy to setup. ---------------------------------------------------------------------- domanpanda · 2023-11-21 08:29 Thanks. Is it SaaS or can i selfhost it? ---------------------------------------------------------------------- thekrautboy · 2023-11-21 08:44 Graylog is fairly simple to get going, can handle almost anything, and if you use it with Opensearch instead of Elasticsearch its also not too hungry on resources. Splunk is sort of popular but requires a license, you can apply for a free one but they dont accept everyone. ---------------------------------------------------------------------- gmemstr2 · 2023-11-21 10:01 In my recent experience, Loki + Grafana + promtail is hard to go wrong with and I haven't experienced any glaring issues. I'm not sure what the setup is like for Docker containers specifically but it feels odd that everything would freeze if Loki goes down - it points to something not being configured right, but I guess it could also depend how Loki is being deployed (is it in its own Docker Compose file, or integrated with everything else?) ---------------------------------------------------------------------- domanpanda · 2023-11-21 10:10 I think this describes the problem. https://github.com/grafana/loki/issues/2361 ---------------------------------------------------------------------- dumbasPL · 2023-11-21 11:28 !remindme 48h ---------------------------------------------------------------------- RemindMeBot · 2023-11-21 11:29 I will be messaging you in 2 days on [**2023-11-23 11:28:35 UTC**](htt p://www.wolframalpha.com/input/?i=2023-11- 23%2011:28:35%20UTC%20To%20Local%20Time) to remind you of [**this link **](https://www.reddit.com/r/selfhosted/comments/180c2rg/what_simple_l og_aggregator_for_homelab_use/ka594nx/?context=3) [**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=Remin dMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2F selfhosted%2Fcomments%2F180c2rg%2Fwhat_simple_log_aggregator_for_homel ab_use%2Fka594nx%2F%5D%0A%0ARemindMe%21%202023-11- 23%2011%3A28%3A35%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)] (https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete %20Comment&message=Delete%21%20180c2rg) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remind mebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?t o=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside %20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Yo ur Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot& subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](ht tps://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot %20Feedback)| |-|-|-|-| ---------------------------------------------------------------------- xXAzazelXx1 · 2023-11-21 11:55 Splunk is not free , right? ---------------------------------------------------------------------- Undefined_ID · 2023-11-21 12:08 LibreNMS include syslog-ng in its docker version: [https://github.com/librenms/docker#syslog-ng- container](https://github.com/librenms/docker#syslog-ng-container) in addition to having one of the best open source software for monitoring, you will have an easy-to-use aggregator for your logs. ---------------------------------------------------------------------- osuhickeys · 2023-11-21 13:23 Dozzle ---------------------------------------------------------------------- domanpanda · 2023-11-21 13:41 I was refering to this one. But now i see that this license is very limiting? https://docs.splunk.com/Documentation/Splunk/9.1.2/Admin/MoreaboutSplu nkFree ---------------------------------------------------------------------- domanpanda · 2023-11-21 13:41 Oh i love it! I'll check it out ASAP. Thank you! ---------------------------------------------------------------------- chignole · 2023-11-21 14:38 I use dozzle to access my containers logs, it's simple and yet works very well for my needs, OP needs to monitor syslogs and i don't know if it's possible with Dozzle ? ---------------------------------------------------------------------- osuhickeys · 2023-11-21 14:46 https://dozzle.dev/guide/log-files-on-disk ---------------------------------------------------------------------- thekrautboy · 2023-11-21 15:24 The problem is that in that specific setup Docker is set to use the Loki logging driver for the whole Docker daemon on the host machine. This is of course the easy and nice way to set it up as it sends all container logs on that machine to Loki, without having to reconfigure each container. But apparently if the remote target for the logs is not available, the Docker daemon doesnt like that at all. You can setup Loki with Docker logs without using that special driver tho. For example i am using Graylog with Dockers GELF logging driver, if i take down Graylog, Docker keeps running without problems. So its not a general problem, but very specific to the Loki logging driver. ---------------------------------------------------------------------- Spaceman_Splff · 2023-11-21 15:32 I use graylog and it’s been great for logging and dashboards. I pair it with zabbix, but librenms is the simpler monitoring system. ---------------------------------------------------------------------- Xenkath · 2023-11-21 16:51 I really like [Seq](https://datalust.co/seq). It supports tons of log types, including GELF, which means you can configure your docker hosts to send all container logs to Seq with just a couple lines in Docker’s daemon.json file. Setup is pretty simple as a docker compose stack containing the Seq container and a container for ingesting each of the log types you plan to use. If you wanted to try it out, I’d recommend setting up Seq with seq- input-gelf and seq-input-syslog, and setting up nginx to send access and error logs to syslog as detailed [here](https://nginx.org/en/docs/syslog.html). ---------------------------------------------------------------------- domanpanda · 2023-11-21 17:46 Can it monitor containers or some files? ---------------------------------------------------------------------- Undefined_ID · 2023-11-21 18:42 Yes, you can monitor Docker and all its containers via `docker-stats`. Simply extend the `snmpd.conf` of your server with the script and configs that you'll find here: [https://docs.librenms.org/Extensions/Applications/#docker- stats](https://docs.librenms.org/Extensions/Applications/#docker- stats) You will see nice graphics of your containers, thanks to RRD (PIDs, RAM, CPU, Storage, etc) ! Regarding monitoring some files, I'm not sure to understand your need. Do you need to check if a file is modified? Its content? Good to know too, [LibreNMS is compatible with Nagios Checks/Plugins](https://docs.librenms.org/Extensions/Services/#nagios- plugins-source). Most of them are already integrated to the docker image of LibreNMS. For the others, you can still rebuild the LibreNMS image and integrate the check and its dependencies. ---------------------------------------------------------------------- mss-cyclist · 2023-11-21 19:21 Maybe syslog-ng is an option? ---------------------------------------------------------------------- domanpanda · 2023-11-21 19:48 By files i ment 'logging to files'. Currently my nginx logs each domain to separate files. ---------------------------------------------------------------------- Fungled · 2023-11-21 21:45 Lightweight and does the job for me too ---------------------------------------------------------------------- Undefined_ID · 2023-11-21 21:52 I see what you mean! But the advantage of all the solutions that have been listed in this thread is the use of the protocol Syslog. To monitor a logfile, you'll need an intermediary to translate it to syslog protocol. The simplest way for you is maybe to send directly your containers logs (nginx included) to your Syslog server. In a docker-compose, it looks like that: `logging:` `driver: syslog` `options:` `syslog-address: "udp://192.168.0.10:514"` `syslog-facility: "local0"` `tag: "{{.DaemonName}}: {{.Name}} {{.ID}}"` You can apply this settings globally to all your containers via the config of Docker: `/etc/docker/daemon.json` You'll find help here: [https://docs.docker.com/config/containers/logg ing/syslog/](https://docs.docker.com/config/containers/logging/syslog/ ) Or here: [https://www.commandprompt.com/blog/docker-logging-with- rsyslog/](https://www.commandprompt.com/blog/docker-logging-with- rsyslog/) Regarding the monitoring of the Docker daemon, or any other subsystems of your operating-system, you can simply install `rsyslog` package on the host and forward all or specifics logs to your Syslog server via a config like this in `/etc/rsyslog.conf`: `*.* action(type="omfwd" target="192.168.0.10" port="514" protocol="tcp"` `action.resumeRetryCount="100"` `queue.type="linkedList" queue.size="10000")` (support [log-queue](https://rsyslog.readthedocs.io/en/latest/rainersc ript/queue_parameters.html) in case the Syslog server is offline) Hope this help! ---------------------------------------------------------------------- the_ml_guy · 2023-11-21 21:59 Do let us know what you end up picking... ---------------------------------------------------------------------- rjrbytes · 2023-11-22 00:42 I use Splunk myself. If I recall correctly, I needed syslog-ng to ingest data anyway before Splunk could grab it. If you don’t need the capabilities of Splunk, you could also look at setting up tailon to browse your logs via the web. ---------------------------------------------------------------------- Spaceman_Splff · 2023-11-22 01:03 So i tried this out today and before i even get to the metrics part, apparently there is no syslog parser, meaning that you can not create fields based off of grok patterns or regex patterns so pull out specific data. Seems like it needs some more growth. ---------------------------------------------------------------------- tcassaert · 2023-11-22 07:59 This is usually the task for the log aggregator, not for the storage or dashboard tool. You collect logs, you parse them with something like [Vector](https://vector.dev), [FluentD](https://www.fluentd.org/) or [Logstash](https://www.elastic.co/logstash) to get the fields you want and then you store and view those fields in OpenObserve. ---------------------------------------------------------------------- domanpanda · 2023-11-22 08:02 I will test Loki, Greylog and LibreNMS (in this order) this weekend. The only thing which bothers me is that rsyslog and GELF protocols are unencrypted - they send plain data. So far all my internal traffic has been encrypted (almost, only to local DNS is not but from local DNS to external DNS is). I cant add TLS to things like switchpormtail/Greylog traffic so probably i will have to add another VLAN and divide my network even more. ---------------------------------------------------------------------- domanpanda · 2023-11-22 08:21 It's not about local syslog - this is easy. Its about remote syslogs from things like routers, switches, NASes etc. I assume that probably dozzle would need something like promtail or fluentd which would aggregate those logs before sending them to dozzle. ---------------------------------------------------------------------- domanpanda · 2023-11-22 09:23 TBH i was confused by it too at first. I mean, Greylog can work without parser, Loki is advertised always with Promtail and in case of OpenObserver i did not know that it requires additional parser until i saw documentation. ---------------------------------------------------------------------- Calm-Helicopter-3212 · 2023-11-22 11:25 Went down this rabbit hole and got it setup with both container logs and some important host logs. Thanks for that pointer! ---------------------------------------------------------------------- the_ml_guy · 2023-11-22 11:43 You could actually parse anything with OpenObserve. OpenObserve supports functions that you could use for any incoming log stream and parse them with ready-made parsers for nginx, apache, syslog, json etc and more. For syslog just add a function for the stream - https://vector.dev/docs/reference/vrl/functions/#parse\_syslog ---------------------------------------------------------------------- osuhickeys · 2023-11-22 14:08 Ah, those are new requirements. Dozzle is not the right tool for those types of logs.