Do you monitor network traffic? How? reddit / selfhosted ---------------------------------------------------------------------- moontear · 2023-11-22 11:59 With my zoo of docker containers and multiple servers hosted locally or on some cloud providers, I feel the need more and more to understand what kind of network traffic is happening. Seeing my outbound traffic on some cloud providers I'm sometimes wondering "huh- where did that traffic come from?". And honestly I have to say: I don't know. Monitoring traffic is a real hurdle since I'm doing a lot via tunnels / wireguard in between servers or to my clients. When I spin up a network analysis tool such as ntopng, I do see a lot of traffic happening that is "Wireguard". Cool. That doesn't help me one bit. I would have to do some deep package inspection I suppose and SSL interception to actually understand WHAT is doing stuff / where network traffic comes from. Honestly I wouldn't be sure what stuff would be happening if there were some malicious thing running on the server and I really don't like that. I want to see all traffic and be able to assign it to "known traffic" or in other words - "this traffic belongs to Jellyfin", "That traffic is my gitea instance", "the other traffic is syncthing" or something along those lines. Is there a solution you beautiful people in this subreddit recommend or use? Don't you care? ---------------------------------------------------------------------- dotinho · 2023-11-22 12:16 Following ---------------------------------------------------------------------- driversti · 2023-11-22 12:40 RemindMe! 3 days ---------------------------------------------------------------------- RemindMeBot · 2023-11-22 12:41 I will be messaging you in 3 days on [**2023-11-25 12:40:50 UTC**](htt p://www.wolframalpha.com/input/?i=2023-11- 25%2012:40:50%20UTC%20To%20Local%20Time) to remind you of [**this link **](https://www.reddit.com/r/selfhosted/comments/1817i70/do_you_monito r_network_traffic_how/kaak1r4/?context=3) [**CLICK THIS LINK**](https://www.reddit.com/message/compose/?to=Remin dMeBot&subject=Reminder&message=%5Bhttps%3A%2F%2Fwww.reddit.com%2Fr%2F selfhosted%2Fcomments%2F1817i70%2Fdo_you_monitor_network_traffic_how%2 Fkaak1r4%2F%5D%0A%0ARemindMe%21%202023-11-25%2012%3A40%3A50%20UTC) to send a PM to also be reminded and to reduce spam. ^(Parent commenter can ) [^(delete this message to hide from others.)] (https://www.reddit.com/message/compose/?to=RemindMeBot&subject=Delete %20Comment&message=Delete%21%201817i70) ***** |[^(Info)](https://www.reddit.com/r/RemindMeBot/comments/e1bko7/remind mebot_info_v21/)|[^(Custom)](https://www.reddit.com/message/compose/?t o=RemindMeBot&subject=Reminder&message=%5BLink%20or%20message%20inside %20square%20brackets%5D%0A%0ARemindMe%21%20Time%20period%20here)|[^(Yo ur Reminders)](https://www.reddit.com/message/compose/?to=RemindMeBot& subject=List%20Of%20Reminders&message=MyReminders%21)|[^(Feedback)](ht tps://www.reddit.com/message/compose/?to=Watchful1&subject=RemindMeBot %20Feedback)| |-|-|-|-| ---------------------------------------------------------------------- 2samedru · 2023-11-22 12:52 ReminMe! 2 days ---------------------------------------------------------------------- totally_not_a_loner · 2023-11-22 12:52 RemindMe! 5 days ---------------------------------------------------------------------- ronny_rebellion · 2023-11-22 12:54 RemindMe! 2 days ---------------------------------------------------------------------- zwamkat · 2023-11-22 12:58 It seems as if you would like to see all traffic identified up to layer 7. That is pretty much enterprise level traffic inspection. I’ve done a lot of it on the edge of our network using a Palo Alto firewall with pretty much all software licenses enabled. I could create full blown reports of single users and/or applications. I sure did point out some co-workers ánd applications who where misbehaving on our network. ---------------------------------------------------------------------- Normanras · 2023-11-22 13:03 I’ve gone down this rabbit hole and have yet to find a solution I like. The only routes I haven’t gone down yet are the grey log or sec onion, as the learning curve is steep. I do use crowdsec and that has been semi-helpful at showing me where a scanner is trying to poke around and on what service. I currently use ntopng’s community version and that’s been acceptable for now. Some parts are a bit confusing and the documentation didn’t help me understand, but the tables are really well laid out and I can easily see the server/cliebt relationship with in and outbound traffic. I’ll try and share screenshots of how it looks for me to see if that helps you. ---------------------------------------------------------------------- moontear · 2023-11-22 13:06 That is the gist I got so far as well. Enterprise level traffic inspection with hardware costing north of 10.000$. I am trying to wrap my head around that we are all running 20+ containers here in this subreddit, but we don't actually "know" what they are doing in terms of network traffic. I had one case where the crowdsec cloudflare bouncer container was generating a lot of traffic since it was caught in a loop updating crowdsec block lists - I would love to know this stuff! Everything self-hosted / selfmade I have found so far is pretty much stuff like \`nethogs\`, that shows you TCP traffic between local IP & remote IP and "it's a lot". Who is generating that traffic you may ask? Nobody knows. One of them containers... Maybe multiple? I also have container monitoring set up such as netdata. It shows me the TCP traffic per container, but only throughput - again I don't know what is happening. Is level 7 inspection too much to ask? Should I just nod and say "probably it is fine, maybe, I guess"? ---------------------------------------------------------------------- AnApexBread · 2023-11-22 13:10 I do. I monitor it in a lot of ways. 1. IDS at the router 2. Anomoli Detection at the router 3. Host based agents on everything I can 4. L7 Firewalls on everything I can 5. DNS based monitoring for everything Wireguard and Cloudflare Tunnels make network traffic monitoring difficult because it's all encrypted traffic. ---------------------------------------------------------------------- moontear · 2023-11-22 13:13 I also have ntopng running (basic setup is easy for everyone): `ntopng:` `image: ntop/ntopng:stable` `network_mode: "host"` `command: ["-i", "eth0", "--community"]` You can connect to it at port :3000 / :3001 for https. But look what I get when running it for a while, the largest datahog is Wireguard/Tailscale: It is cool, that it knows this - but I actually don't know what stuff goes over the wire there. It's all safe and good, that I use Tailscale/Wireguard - but I don't know who or what is using that tunnel for example. And the same goes for pretty much everything SSL / HTTPS which is everything today. ---------------------------------------------------------------------- SirLagz · 2023-11-22 13:15 Zabbx agent on my docker host, can monitor traffic per container. ---------------------------------------------------------------------- moontear · 2023-11-22 13:19 `netdata` can also monitor traffic throughput, i.e. "container X currently is using 10MBit/s". Can Zabbix do more? ---------------------------------------------------------------------- SirLagz · 2023-11-22 13:26 Bytes Received/sent per second Network errors received/sent Incoming/Outgoing packets dropped per second Incoming/Outgoing packets per second Also keeps historical data, so if I see a spike in traffic, I can check the graphs for all my containers and know which container it came from ---------------------------------------------------------------------- zwamkat · 2023-11-22 13:31 Are you able to run ntopng somewhere before the traffic enters a tunnel? That would create some first insight in your traffic without the general ‘name tag’ encrypted traffic. ---------------------------------------------------------------------- sirrush7 · 2023-11-22 13:32 Depending on what you run for a perimeter device, but elasticsearch is free and can give you incredible visibility into your network. That said, it can be a bit of a beast to learn. Simpler deployment is how I have it, running as Zenarmor Sensei inside my opnsense router/firewall which IS my edge. There's also Prometheus and grafana. Grey log. Lots and lots of options however, just need to feed these log engines your syslogs. That's the magic ticket! ---------------------------------------------------------------------- discoshanktank · 2023-11-22 13:36 What agents do you use on the hosts? ---------------------------------------------------------------------- discoshanktank · 2023-11-22 13:37 No guarantees but I’ll try my best ---------------------------------------------------------------------- driversti · 2023-11-22 13:41 😁 ---------------------------------------------------------------------- Storage-Solid · 2023-11-22 13:42 I would suggest looking at Wazuh and setting up a SIEM stack based on it. It would provide what you need and is highly customisable to needs. ---------------------------------------------------------------------- InvaderOfTech · 2023-11-22 13:53 I have been using libreNMS for over four years and love it. I started to play with checkmk for its agent but found the network side of checkmk is also lovely/easy to work with. I recommend looking at either of these. Both can run in docker or docker-compose. ---------------------------------------------------------------------- Jealy · 2023-11-22 14:00 RemindMe! 3 days to remind /u/discoshanktank to remind /u/driversti. ---------------------------------------------------------------------- Spaceman_Splff · 2023-11-22 14:11 Securityonion is a great ids system. I used their distributed system, so I have 1 mini pc as a sensor and another as a manager/search. Works wonderful. ---------------------------------------------------------------------- Critical_Egg_913 · 2023-11-22 14:17 I found this and will be testing it. Looks like it's free for home use. https://corelight.com/ ---------------------------------------------------------------------- _MetalHead89 · 2023-11-22 14:21 RemindMe! 8 hours ---------------------------------------------------------------------- AnApexBread · 2023-11-22 14:33 Depends on the host. For Linux hosts FileBeats (part of the Elasticsearch tools and included in Security Onion). For Windows hosts WinLogBeats + Sysmon +Swift on Security (Also part of Security Onion). ---------------------------------------------------------------------- BigRoofTheMayor · 2023-11-22 15:03 I have a pfsense and use ntopng ---------------------------------------------------------------------- wallacebrf · 2023-11-22 15:48 I use my fortigate router as it logs everything natively. Logs DNS request, outbound traffic, internal lan local traffic, and so much more ---------------------------------------------------------------------- nik282000 · 2023-11-22 16:11 I wrote a couple scripts that ingest my Apache and SSHD logs to tell me how many hits I had, how many unique hosts they came from and where they are in the world. It even spits out a nice map at the end of the day: https://imgur.com/aJ6aVZp ---------------------------------------------------------------------- KaiserVonLulz · 2023-11-22 16:18 RemindMe! 5 days ---------------------------------------------------------------------- Leonzola · 2023-11-22 16:22 Do you self host the ELK stack? Been tinkering with this myself but it's pretty heavy for one user. ---------------------------------------------------------------------- MisterSlippers · 2023-11-22 16:25 I've been a network engineer, security analyst, security engineer, and SOAR engineer over the course of the last 20 years; I don't want to think about any of that shit when I'm not being paid for it. I have backups of the things I can't replace, no port forwarding/ingress rules from WAN on the firewall, and the network is heavily segmented and uses least privilege. The random security stuff I leverage is set to drop/block and my family does a good job being vocal when something isn't working. If I needed to start over tomorrow, I'd just build a new server with Ansible playbooks on my GitHub. ---------------------------------------------------------------------- AnApexBread · 2023-11-22 16:29 I push them to a Security Onion Server I run in my house (it has an ELK stack in it). ---------------------------------------------------------------------- Leonzola · 2023-11-22 16:46 Smart. I'm going to try that out! ---------------------------------------------------------------------- pobrika · 2023-11-22 16:54 I use cadvisor to get stats and Prometheus and grafana. Works very well ---------------------------------------------------------------------- TheOldDesignator · 2023-11-22 17:23 RemindMe! 7 days ---------------------------------------------------------------------- tpwn3r · 2023-11-22 17:37 I own a WISP with about 100 clients and admin another wisp/fisp with around 500 clients. I love "the dude" by mikrotik. It pulls data via SNMP and gives me a great heads up overview of everything. it also graphs the data over time. I use LibreMNS also to pull network data via SNMP and it graphs historical data. ---------------------------------------------------------------------- r00m-lv · 2023-11-22 18:40 200 OK. I will remind u/Jealy to remind u/discoshanktank to remind u/driversti ---------------------------------------------------------------------- lcmoramos · 2023-11-22 18:53 RemindMe! 3 days ---------------------------------------------------------------------- elecboy · 2023-11-22 18:57 I use Pi-Hole & NextDNS for all my DNS and I check once a week, for extra security I run a Fortigate 61E with AV/IPS and of course VLAN just for IoT and NVR. ---------------------------------------------------------------------- VexingRaven · 2023-11-22 19:06 Mind sharing what tools you use for IDS and anomaly detection on the router? ---------------------------------------------------------------------- AnApexBread · 2023-11-22 19:33 I have Suricata running in my router for the IDS, and the I have Zeek running and sending all its logs over to AC Hunter CE for ingesting into RITA. ---------------------------------------------------------------------- davesnas · 2023-11-22 19:33 RemindMe! 3 days ---------------------------------------------------------------------- Fluffer_Wuffer · 2023-11-22 19:36 Sophos offer their firewall free for home use, I suspect this has SSL Decryption and inspection. Personally, I use a PA, they sell lab licensed VM for a couple of hundred - I work with them, so it kind of pays for it self.. though they don't generally sell to consumers, so you need to do it via you company, or a friendly VAR.. Honestly the best feature though often goes unmentioned - policy based routing, being able to selectivively route any app, port or IP👌 ---------------------------------------------------------------------- YYCwhatyoudidthere · 2023-11-22 19:40 +1 for Security Onion. Logs + pcap is a luxury. You mention DNS Security. Zeek playbooks in SO? Can you point to any good playbook sources? ---------------------------------------------------------------------- s7orm · 2023-11-22 19:53 I have my Unifi Switch mirror the trunk port and send that to Splunk Stream, but I haven't found it that useless to have that level of data. ---------------------------------------------------------------------- AnApexBread · 2023-11-22 20:47 For DNS I use Adguard Home with NextDNS as my upstream. Then on my router I block all DNS, DoT, and all known DoH except for my Adguard and use rewrite rules to force all DNS traffic to the Adguard Home. Pretty much every device will fall back to unencrypted DNS if it can't reach a DoH or DoT server. So that gives me full visibility into what DNS requests are happening. For Zeek the logs are coming off my router and being ingested into AC Hunter for RITA to churn through ---------------------------------------------------------------------- thezvrcak · 2023-11-22 21:47 RemindMe! 2 Days ---------------------------------------------------------------------- Simon-RedditAccount · 2023-11-22 21:48 >Do you monitor network traffic? Generally, no. But I seriously restrict container networking, most of my containers are unable to reach internet, unless absolutely necessary. Also, my firewall is not super-restrictive, but it is different from defaults :) Sometimes I do some monitoring though. ---------------------------------------------------------------------- chaplin2 · 2023-11-22 22:07 IDS and IPS s don’t work well with https traffic, 99% of that. ---------------------------------------------------------------------- AnApexBread · 2023-11-22 22:21 I know, that's why I use host based agents. The IDS do provide application categorization (or attempt to) and JA3 fingerprints. Additionally not all malware uses encryption. There's still plenty of command and control which happens over standard HTTP ---------------------------------------------------------------------- PaulEngineer-89 · 2023-11-22 22:50 Why not use Macvlan and join your Dockers to that rather than a bridge to Tailscale or Cloudflared? Then they are broken out so you can apply monitoring. ---------------------------------------------------------------------- NGL_ItsGood · 2023-11-23 00:14 What do you use for l7 firewalls? ---------------------------------------------------------------------- AnApexBread · 2023-11-23 00:46 Cloudflare Magic Firewall. It's built into warp. Install warp, configure policies in the cloudflare zero trust dashboard, login to warp with your Cloudflare account, install the TLS cert. Now warp is break and inspecting all traffic and sending the logs up to your Cloudflare dashboard. You can create a policy and it pushes that rule to all of the warp clients. ---------------------------------------------------------------------- NGL_ItsGood · 2023-11-23 01:13 Man I really need to get into cloudfare more, seems like they're the AWS for network perimeter solutions. Ty for the info! ---------------------------------------------------------------------- AnApexBread · 2023-11-23 01:15 I love Cloudflare. They offer a ton of stuff ---------------------------------------------------------------------- adamshand · 2023-11-23 07:58 That's a good idea, I might do that! ---------------------------------------------------------------------- CaffeinatedTech · 2023-11-23 09:02 Easy, set up a mail server, then 98% of your traffic will be inbound attack attempts on that. Like mine... ---------------------------------------------------------------------- psychowood · 2023-11-23 09:49 Do you mean I should monitor my email server running on a XP? ---------------------------------------------------------------------- SpongederpSquarefap · 2023-11-23 14:36 Yep, monitoring in multiple places with Zabbix I have pfSense as well (soon to be OPNsense) and that shows traffic per network it's connected to, so that's great for live traffic Zabbix monitors the networks and collects traffic data Zabbix also monitors all containers and their network traffic ---------------------------------------------------------------------- psychowood · 2023-11-23 14:40 I was recently thinking about setting up a transparent squid proxy at router level, I'm curious if it could be useful in this context. ---------------------------------------------------------------------- YYCwhatyoudidthere · 2023-11-23 15:39 Why AdGuard and NextDNS? I ran this way for a while, now I run local DNS on a Linux server pointing to NextDNS as upstream. My next step is DNS alerting like "new domains" but haven't implemented yet. ---------------------------------------------------------------------- AnApexBread · 2023-11-23 15:55 DNS redirection for clients with hard coded DNS. You can't redirect DoT (to my knowledge). So my router redirects all the DNS to Adguard and then Adguard DoTs them up to NextDNS. >Why AdGuard and NextDNS? I ran this way for a while, now I run local DNS on a Linux server pointing to NextDNS as upstream. That's what Adguard Home is. It's like Pihole, a local DNS server, but with more features. ---------------------------------------------------------------------- Jealy · 2023-11-25 14:01 Psst, don't forget to remind him! ---------------------------------------------------------------------- discoshanktank · 2023-11-25 14:42 Thanks! ---------------------------------------------------------------------- discoshanktank · 2023-11-25 14:42 Ok it’s time bro ---------------------------------------------------------------------- driversti · 2023-11-25 14:45 Awesome, thank you mate 😉🤝 ---------------------------------------------------------------------- Jealy · 2023-11-25 22:44 We're all in this together!