Centralized Syslog (remote and local hosts) reddit / selfhosted ---------------------------------------------------------------------- SillyLilBear · 2023-10-30 01:36 I want to centrally log messages for both internal and remote servers. I was looking at Loki at one point, as it seems to be a good option due to space savings. They also have a generous cloud option to get started and test it out. Anyone have a setup that you have been using for a while that works really well? I want to start getting CheckMK and Grafana up eventually along with it. ---------------------------------------------------------------------- ntman1 · 2023-10-30 04:22 [Wazuh](https://wazuh.com/) ---------------------------------------------------------------------- thekrautboy · 2023-10-30 04:54 Promtail, Loki, Grafana. ---------------------------------------------------------------------- SillyLilBear · 2023-10-30 05:35 I thought Wazuh was for security monitoring, I didn't realize it could be a general log drain. I've been meaning to check it out, but trying to get some other stuff up first. ---------------------------------------------------------------------- ElevenNotes · 2023-10-30 05:49 Telegraf (syslog plugin) > influxdb > grafana ---------------------------------------------------------------------- seiichiro0185 · 2023-10-30 06:47 I run syslog -> promtail -> loki -> grafana as my logging stack. The individual systems send the logs to promtail using the standard syslog UDP protocoll, promtail then adds some labels and puts them into loki. Grafana is used for viewing / searching the logs. This setup runs flawlessly for about a year now. logs from about 30 machines (local and remote via VPN) with a retention time of 31 days take up about 1.5GB of space ---------------------------------------------------------------------- SillyLilBear · 2023-10-30 06:48 That's what I was thinking of trying myself. How would you log from remotes without a vpn? SSH? I don't want to have to trust remote hosts to my LAN but I would like them to be able to log there. ---------------------------------------------------------------------- seiichiro0185 · 2023-10-30 06:59 You could do SSH with normal or reverse TCP tunneling depending an which way the connection goes (syslog protocol also can be done over TCP). Something like autossh might be an option here to reliably establish this, and you can do some tricks to get a "TCP-forwarding only" connection working. On the other hand I don't think a properly firewalled VPN is much of a problem for this case. The VPN from my remote VPS hosts terminates on my opnsense firewall as a separate zone, which only gets access to the few ports (like syslog) that need to be reachable from the remote. ---------------------------------------------------------------------- SillyLilBear · 2023-10-30 07:45 I don't want to trust the remote hosts to have a VPN to my lan. ---------------------------------------------------------------------- seiichiro0185 · 2023-10-30 07:53 In that case I would probably go with (auto-)ssh to invoke a connection from the logging host to the remotes with a reverse TCP tunnel for the log traffic. This way there is no remote-initiated connection to anything in the LAN, and you also don't need to expose any ports to the "outside" (you definitly don't want the syslog port exposed anywhere public, since it's not authenticated or secured in any way) ---------------------------------------------------------------------- arcadianarcadian · 2023-10-30 08:24 Graylog ---------------------------------------------------------------------- usa_commie · 2023-10-30 16:13 Should be higher up. There are few solutions that offer so much out of the box in terms of logging. ---------------------------------------------------------------------- shetif · 2023-10-30 17:10 Im sure there are fancy solutions with data visualization and whatnot, but syslog has an out of box solution shipped with most distros: rsyslog You can config a designated (or more) server(s) to receive your syslog entries. ---------------------------------------------------------------------- Defiant-Ad-5513 · 2023-10-31 07:33 Sadly i can't get it to work in my PI