Best platform for self-hosted logs? reddit / selfhosted ---------------------------------------------------------------------- CrappyTan69 · 2023-11-14 22:46 BLUF: Is there a "datadog-for-home"? I've a ton of stuff running at home ranging from a 5-node PI cluster with various containers running on them including things like self- written python scripts doing "super important" stuff, node-red running everything about my energy setup, pfsense, TrueNAS etc. Logging is painful and I've just lost about 4 hours trying to find a fault which stopped car charging. Lots of rabbit holes were entered as I currently don't have an end-to-end logging solution. ​ What does everyone else use? ---------------------------------------------------------------------- Mezutelni · 2023-11-14 22:58 I personally use graylog, it’s nice looking and easy to setup ---------------------------------------------------------------------- mati_tylec · 2023-11-14 23:05 I use Synology as a log server ---------------------------------------------------------------------- fractalfocuser · 2023-11-14 23:28 +1 Graylog ---------------------------------------------------------------------- dcabines · 2023-11-14 23:30 [Dozzle](https://dozzle.dev/) may be a bit basic for your needs, but it could be one tool in your toolbox. ---------------------------------------------------------------------- CrappyTan69 · 2023-11-14 23:32 Thanks! ---------------------------------------------------------------------- CrappyTan69 · 2023-11-14 23:34 Thanks ---------------------------------------------------------------------- Aurailious · 2023-11-14 23:35 Loki to Grafana. ---------------------------------------------------------------------- NikStalwart · 2023-11-14 23:51 Depending on your existing monitoring stack, some options might be: * [Grafana Loki](https://grafana.com/oss/loki/) * [Sentry](https://github.com/getsentry/) can be self-hosted for application logging * [Logstash](https://github.com/elastic/logstash) is self- explanatory, use with other parts of Elastic's software like Kibana for visualization. ---------------------------------------------------------------------- dlm2137 · 2023-11-14 23:58 Anyone here got a solution working in Unraid? I looked into this recently but got a bit stuck. ---------------------------------------------------------------------- jaank80 · 2023-11-15 00:02 ELK ---------------------------------------------------------------------- AnomalyNexus · 2023-11-15 00:20 Opted for loki since it allows using the same interface for multiple purposes ---------------------------------------------------------------------- Boomam · 2023-11-15 01:22 Agreed. Solutions like Loki and it's ilk fall down when it comes to real compatibility with standard logging mechanisms, aka, syslog. They rely on agents and/or intermediaries to get said logs, and often require transformation of the data to be parsed into anything remotely useful. Loki/similar is great for k8s logs and anything "cloud native" in style, but for logs from anything else, you cannot beat a solution like graylog as it meets the device where it's at. Can't install promtail on a printer for example. :-p ---------------------------------------------------------------------- thekrautboy · 2023-11-15 01:44 I like Grafana/Loki/promtail etc. But logseq and Datalust Seq are also nice. Graylog can do a lot but its too bloated and too hungry for my taste (damn elasticsearch). ---------------------------------------------------------------------- lvlint67 · 2023-11-15 03:01 A simple syslog server is ready. Look into ELK stack. I personally hate it but if you can learn it there's a chance you can build a career on it. ---------------------------------------------------------------------- guhcampos · 2023-11-15 04:00 I've been a Graylog user before, loved it, though haven't followed it in years. However, I don't understand how "cant install loki on a printer" applies? You can't install Graylog, either. You can do rsyslog with pretty much all these solutions. ---------------------------------------------------------------------- YYCwhatyoudidthere · 2023-11-15 04:02 I come from a Cybersecurity background which might explain my answer: Security Onion had proven adept at cross referencing logs and pcaps which is pretty awesome for troubleshooting And +1 for Dozzle to see real-time Docker logs. ---------------------------------------------------------------------- -SHINSTER007 · 2023-11-15 04:07 I wanted get started in infosec, where should I get started? What should I start learning ---------------------------------------------------------------------- TheSeloX · 2023-11-15 04:24 For most self hosted use cases Splunk's free 500MB (per day) license should be enough. It's way easier to set up and maintain than ELK and has tons of free extensions for parsing log formats and dashboards. ---------------------------------------------------------------------- RVA_dude88 · 2023-11-15 04:48 I've tried so many times to get it set up but it's so confusing in my opinion ---------------------------------------------------------------------- dhuscha · 2023-11-15 05:40 Also using Graylog, dead simple to set up with rsyslog and at work we even use the sidecar for window logs. ---------------------------------------------------------------------- floppydiet · 2023-11-15 06:13 Splunk ---------------------------------------------------------------------- psychowood · 2023-11-15 06:42 I considered it, [seems nice](https://docs.splunk.com/Documentation/Sp lunk/9.1.1/Admin/MoreaboutSplunkFree). Problem is that they [recommend](https://docs.splunk.com/Documentation /Splunk/9.1.1/Capacity/Referencehardware) a 12 physical cores and 12GB which is a waste for the usual selfhosted lab. ---------------------------------------------------------------------- TheSeloX · 2023-11-15 06:49 I think I ran it on my Raspberry Pi 4 4GB for up to 50 MB of daily logs without an issue. I didn't have more data so I don't know what the limit of the Pi would have been. ---------------------------------------------------------------------- TheSeloX · 2023-11-15 06:51 I ran it on a VM with 2 cores tops and a couple of GB of RAM for about 50MB of logs per day. ---------------------------------------------------------------------- ChickenMcRibs · 2023-11-15 07:14 Love dozzle. Super simple and straightforward ---------------------------------------------------------------------- psychowood · 2023-11-15 07:24 Interesting, thanks. Were the 6GB dedicated to splunk or shared with other services? ---------------------------------------------------------------------- TheSeloX · 2023-11-15 07:29 Dedicated. Nothing else was running on the VM. But it's been a while and I can't remember how much of the resources it really needed. Since it's free anyway I guess you can just try it. There should also be a docker image. ---------------------------------------------------------------------- VoidQuark · 2023-11-15 07:29 Utilize Loki, and explore my blogs on creating dashboards. You’ll discover how straightforward it is to build dashboards with Loki and Grafana. Avoid using outdated technology like Graylog. https://voidquark.com/blog/tags/loki ---------------------------------------------------------------------- skelleton_exo · 2023-11-15 07:30 I mean its not bad. But maintaining, ELK is a full time job. When I ran that at home, every update basically broke something. ---------------------------------------------------------------------- iavael · 2023-11-15 08:05 And using kibana for logs is full time pain. After all these years centralized logging software is still nowhere near as good as old CLI with grep and awk in term of UX and UI. ---------------------------------------------------------------------- iavael · 2023-11-15 08:08 Just write your logs as files on a centralized syslog server with good file structure and you'll be good. You may really underestimate how fast and convenient grep+less combo is in comparison to webui-based solutions. ---------------------------------------------------------------------- Fungled · 2023-11-15 08:16 In my experience running ES on a homelab box is just not worth it - indexes get corrupted every few months and it’s constantly trying to eat too many resources Using Seq now and it’s nice and lightweight ---------------------------------------------------------------------- Mezutelni · 2023-11-15 08:44 With graylog you can have rsyslog, gelf, or even RAW data, each both on TCP or UDP, so it's really flexible when it comes to exotic log- generating apps. ---------------------------------------------------------------------- Croatwink · 2023-11-15 09:36 It's a hybrid solution but I prefer putting my logs with an S3 provider, it's just cheap storage that I don't have to care about. And there are a lot of tools to do it with, like loki for example. ---------------------------------------------------------------------- hdddanbrown · 2023-11-15 11:30 Another vote for graylog, runs well with opensearch instead of elastic. ---------------------------------------------------------------------- [deleted] · 2023-11-15 11:37 [removed] ---------------------------------------------------------------------- xupetas · 2023-11-15 11:46 +1 for graylog ---------------------------------------------------------------------- Boomam · 2023-11-15 12:05 Graylog and similar ingest syslog directly Loki and similar need an intermediary, such as rsyslog to do so. ---------------------------------------------------------------------- NortySpock · 2023-11-15 12:09 Seconding Dozzle as a basic tool to get quick access to your docker logs I have been looking at [VictoriaLogs](https://docs.victoriametrics.com/VictoriaLogs/) as a log aggregator, but I don't have the same scale of logging needs as the OP. ---------------------------------------------------------------------- osuhickeys · 2023-11-15 12:50 You can also view [application](https://dozzle.dev/guide/log-files-on- disk) logs with Dozzle in addition to container logs. ---------------------------------------------------------------------- reddit-gk49cnajfe · 2023-11-15 12:50 s/grep/ripgrep 😉 ---------------------------------------------------------------------- substitute-bot · 2023-11-15 12:50 Just write your logs as files on a centralized syslog server with good file structure and you'll be good. You may really underestimate how fast and convenient **ripgrep 😉**+less combo is in comparison to webui-based solutions. ^^This ^^was ^^posted ^^by ^^a ^^bot. ^^[Source](https://github.com/anirbanmu/substitute-bot-go) ---------------------------------------------------------------------- reddit-gk49cnajfe · 2023-11-15 12:51 Did not know ^ was a thing...! TIL ---------------------------------------------------------------------- gotaede · 2023-11-15 13:02 Anyone else looking at openobserve. Looks OK for homelab, but not really stable ---------------------------------------------------------------------- SillyLilBear · 2023-11-15 13:28 Can you do as much with graylog as you can with Grafana and Loki? ---------------------------------------------------------------------- the_ml_guy · 2023-11-15 13:39 [https://github.com/openobserve/openobserve](https://github.com/openob serve/openobserve) . Built in rust - No JVM. Much lighter than the alternatives mentioned here and with extremely good UI. Beautiful dashboards. Could even run on raspberry pi. ---------------------------------------------------------------------- the_ml_guy · 2023-11-15 13:39 What do you mean by not stable? ---------------------------------------------------------------------- Boomam · 2023-11-15 13:55 Its an apples-to-orange comparison.   Graylog is a logging solution in its entirety. You want to do something with logs, ingest, visualize, alert, etc. - sure, it'll absolutely do it and do it well.   Grafana/its related stack is not a log-first platform, its a metrics and observability platform, designed to allow said logs to be visually integrated into the same dashboard as the other things it does, for "single pane of glass" usage.   So it technically does more, but its intended usage is a wider set of functions by design. It's good at most of them too, just not great at all of them, and imo, not as good at Graylog at log ingestion and diagnostics.   "Pick the right tool for the job". ---------------------------------------------------------------------- gotaede · 2023-11-15 14:04 They state in their documentation that the software is alpha ([https://openobserve.ai/docs/](https://openobserve.ai/docs/) OpenObserve is currently in alpha, but don't let that stop you from trying it out.) . To be honest I didn't bother to investigate why ingesting data stops working after a few days, might be my installation then. I'm very curious which organisation uses alpha software in production ---------------------------------------------------------------------- Chukumuku · 2023-11-15 14:15 Graylog ---------------------------------------------------------------------- the_ml_guy · 2023-11-15 14:17 Metrics is alpha grade for now. Logs is pretty solid which is what most organizations use it for. ---------------------------------------------------------------------- SillyLilBear · 2023-11-15 14:52 How do you get that metrics and observability when using graylog? Can you? ---------------------------------------------------------------------- amitn322 · 2023-11-15 15:30 Try BlackESK https://github.com/amitn322/blackesk ---------------------------------------------------------------------- AnderssonPeter · 2023-11-15 15:55 Loki + grafana does a great jobb ---------------------------------------------------------------------- AnderssonPeter · 2023-11-15 15:57 Elk is nice but it eats RAM like there is no tomorrow! ---------------------------------------------------------------------- Boomam · 2023-11-15 17:05 I'll link you to the Graylog site, as theres some good info, docs & pics on there - https://graylog.org/products/operations/ ---------------------------------------------------------------------- bobbarker4444 · 2023-11-15 17:21 I use the ELK stack (Elasticsearch, Kibana, Beats) ---------------------------------------------------------------------- guhcampos · 2023-11-15 18:22 Not sure they do? Logstash has its own rsyslog plug-in, don't know about Loki in particular, but rsyslog is generally available and implemented everywhere. ---------------------------------------------------------------------- guhcampos · 2023-11-15 18:22 I'm fairly sure you can have all that on all other solutions, maybe not GELF, but using GELF these days is a bit of a pain anyway since everyone has standardized on JSON ---------------------------------------------------------------------- slavik-f · 2023-11-15 18:23 I looked at the Synology log server, but it's too primitive. Can't really filter the data. Search is extremely basic. ---------------------------------------------------------------------- N4v41 · 2023-11-15 20:21 I tried some tools and the one that I am currently using is OpenObserve, it's light has a very good compression and is simple to manage, as an observability platform I think that open observe has some features that can be used instead of datadog like log injestion and open telemetry traces ---------------------------------------------------------------------- linuxmel · 2023-11-20 07:19 Dozzle is fantastic!