A mailserver incident post-mortem reddit / selfhosted ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 00:39 So I recently managed to make my Self-Hosted mailserver an **Open Relay**. My mailserver (dockerized mailcow) currently runs on a little NUC under my stairs. It has worked well with only minor problems over the 3 or so years I've had it running; I got spamhaused once, etc. The problem all started with me trying to patch a percieved security hole. See, docker doesn't respect firewalls like UFW or firewalld (all based on iptables of course), instead opting to allow ports through iptables as you add `-p` flags to your containers in spite of any other rules you may have. Now I thought this was rather terrible. I don't want to have to look both at my firewall and at all my docker port bindings to check if something is open. So as many of us would do, I started trawling the internet for solutions and started to learn about why this behaviour existed. According to some articles/stackoverflows/etc. the way to stop docker messing with iptables and creating its own rules is to disable the feature in the daemon.json. Seems simple enough. The only caveat that I found mentioned was that container networking would break (in terms of internet reachability) but that's ok because I just had to add a firewalld rule to allow masquerading and that problem was solved. Now the problem I failed to see was that of NAT changing. Prior to disabling the iptables flag, the mailserver would see connections' IPs as their real public ones. However afterwards, every single IP was that of the internal docker network default route. I didn't think much of it at the time, merely that it would be more annoying to see who was connecting but that was fine because I had what I wanted. Firewalld was now the sole controller of my ports 🎉 Little did I know (or maybe I did and just forgot) that postfix has a trusted list of IPs and it will relay anything from them without question. These IPs include internal IPs such as that of the default route... So essentially every SMTP request was being NATed to have a sender address of 172.22.1.1 and postfix started sending EVERYTHING 😵‍💫 It wasn't long before a plethora of bots had saturated my poor NUC with HUNDERDS OF THOUSANDS of emails. I got home this evening to lag spikes in Tarkov which prompted me to check the server where I found this mess. After taking everything down, re-enabling the iptables and flushing all the postfix queues, I was able to spin back up and not have the whole thing start spiralling again. Some tips for those hosting mailservers: - Use a mail server checker like https://mxtoolbox.com/SuperTool.aspx - Setup monitoring and alerts for server CPU usage/spikes in requests/etc - Don't fix what ain't broke I'm gonna go cry myself to sleep now and pray that the big mail hosts like Google and Microsoft take pity on me and my screw up. (We all know I'll never be able to send another email to Microsoft again, who am I kidding) ---------------------------------------------------------------------- wing03 · 2023-11-29 01:54 Wow! What residential internet service lets port 25 through? ---------------------------------------------------------------------- zcubed · 2023-11-29 02:16 I did something similar way back in 2002 and my ISP shut me down hard. I'm surprised any ISP in this day and age allows a residential connection send any emails out. This is one of the many reasons I won't ever host an email server again. Good luck! ---------------------------------------------------------------------- Nimrod5000 · 2023-11-29 02:17 Right? Might have a proxy ---------------------------------------------------------------------- Nimrod5000 · 2023-11-29 02:17 You should write up an article on how to setup docker with firewalld ---------------------------------------------------------------------- ChicagoAdmin · 2023-11-29 02:21 Honestly, this. I don’t know why anyone even tries it in this day and age, unless it’s an experiment. ---------------------------------------------------------------------- jimmyhoke · 2023-11-29 02:40 My parent’s ISP does actually. I used to run a mail server at home before I moved out. ---------------------------------------------------------------------- jimmyhoke · 2023-11-29 02:40 I run a mail server on a VPS, and it works pretty well. ---------------------------------------------------------------------- ChicagoAdmin · 2023-11-29 02:41 Talking about from a residential subscriber line. ---------------------------------------------------------------------- jimmyhoke · 2023-11-29 02:42 Oh yeah I used to do that. I didn’t even make it to spam. ---------------------------------------------------------------------- adamshand · 2023-11-29 03:55 How do you get good judgement? Experience. How do you get experience? Bad judgement. :-) ---------------------------------------------------------------------- naylo44 · 2023-11-29 04:21 I did practically the same, while deploying a mail solution docker in an LXC container... ---------------------------------------------------------------------- lestrenched · 2023-11-29 04:23 1. Use a VPS. 2. A write-up please. This is beyond my current understanding of Docker networking and more resources would really help ---------------------------------------------------------------------- perk11 · 2023-11-29 04:37 Yeah you might as well write off the IP. At least with Google it will take months to recover, and if you don't have a good volume of quality emails go through it the chances are even lower. ---------------------------------------------------------------------- buttstuff2023 · 2023-11-29 05:29 I've never had one that doesn't allow that. ---------------------------------------------------------------------- buttstuff2023 · 2023-11-29 05:32 They can't really stop you from sending emails out without potentially breaking legitimate mail. They can stop email from being submitted to you by blocking port 25/465/587 inbound though. ---------------------------------------------------------------------- buttstuff2023 · 2023-11-29 05:32 You should write that ---------------------------------------------------------------------- Nimrod5000 · 2023-11-29 06:04 I dont have the time to go down that rabbit hole right now ha ---------------------------------------------------------------------- senectus · 2023-11-29 06:15 definitely on my list of why i dont run a mail server :-P ---------------------------------------------------------------------- flecom · 2023-11-29 06:49 yep, I'll bet op won't make that mistake again! I did it back when I was running mdeamon on NT4, but that was a long, long time ago ---------------------------------------------------------------------- haroldp · 2023-11-29 07:52 Most consumer ISPs these days block outgoing 25, and it's been that way since the late 90s. Third party mail providers generally ask you to use 465 or 587 for that reason. ---------------------------------------------------------------------- haroldp · 2023-11-29 07:59 Don't feel *too* bad. Back in the day, it seemed like every other edit to sendmail.mc would result in an open relay. I don't miss Sendmail. :) I would also encourage you to start monitoring your mailq. For a little home server, if you have more than 10 messages in there, you should probably have an alarm going off. You might also do this for other things like web servers that might send out a contact form email or things like that. ---------------------------------------------------------------------- Cylian91460 · 2023-11-29 07:59 Yeah same, the only port blocked in the port 1, they use it for ssh (they don't want to add my key :( ) on router (for maintance and update who also goes through ssh). ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 08:21 I'm with Plusnet in the UK right now; Never had problems with blocked ports. I believe Sky also allows it. ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 08:25 I'm praying mine hasn't realised 😅 And thanks, I'll need it ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 08:34 Yeah I was afraid of that. Good(?) news is I'm switching ISP soon so my IP will disappear soon. So long, static IPv4, hello CGNAT and IPv6. I'll need a VPS somewhere to proxy any incoming IPv4 traffic at that point anyway. ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 08:36 I've been trying to avoid one but the time has probably come. I've been thinking about writing more/making a YouTube channel so perhaps you'll see something on it :) ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 08:37 Yep, never again 🥹 ---------------------------------------------------------------------- RecursiveIterator · 2023-11-29 09:16 From my experience with 3 ISPs in the UK, they only block ports on their end when you request a static IP. ---------------------------------------------------------------------- lilolalu · 2023-11-29 09:35 For future purposes: https://github.com/chaifeng/ufw-docker ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 09:38 That or podman! ---------------------------------------------------------------------- KittensInc · 2023-11-29 09:46 Honestly, it's about time Postfix gets replaced with a *modern* email server. It's a dinosaur designed around 1990s UNIX, with dozens of leftover footguns eagerly waiting to go off. The fact that it even *allows* local mail submission or trusted subnets is already problematic, if you ask me. It is 2023, email should *only* be allowed after proper authentication & authorization! ---------------------------------------------------------------------- lilolalu · 2023-11-29 09:49 Yeah, but for an existing dockerd setup "ufw-docker" is a quick way to have a working ufw setup without changing the entire container ecosystem. its just a shell script that sorts the ufw iptables rules in a way they work with dockers virtual networks & bridges, nothing very fancy, but it works. ---------------------------------------------------------------------- 404invalid-user · 2023-11-29 10:09 what even if the is allows that port if you don’t have a valid RDNS the big ones would just reject your email ---------------------------------------------------------------------- kweevuss · 2023-11-29 10:21 I’ll add another one to the list of monitoring. Add an alert of sent email threshold. I log to gray log, and if more than 5 are sent in 5 minutes it will send me an alert. ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 10:32 Ironically I've had all my emails go through without so i must just be extremely lucky ---------------------------------------------------------------------- 404invalid-user · 2023-11-29 10:45 ha yeah you are i could never get it to work ended up just installing it on a vps if you use mc tools to check all the block lists and you are not there then you’re extremely lucky ---------------------------------------------------------------------- UEF-ACU · 2023-11-29 11:40 Spectrum in the US allows the customer to replace their provided router with their own equipment, so all they provided to me is the modem, after that I have my own router, so I can pretty much do whatever I want. Never had an issue hosting a mail server, sending/receiving works perfect. ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 11:42 I've been put on a few as of this morning and I've requested delisting on those that offer it. However previously I was on none of them which is a miracle. ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 11:44 Which ones? I have a static with Plusnet right now and I had a static with Sky iirc which also had no blocks ---------------------------------------------------------------------- bufandatl · 2023-11-29 12:01 Hm. Lag spikes in Tarkov and you check your server? I mean Tarkov. But yeah I can feel your misconception here. But I am also the other way around I uninstalled firewalld and do all on iptables level. I am just more used to iptables. And so the sole controlling instance is iptables. In the end it’s all netfilter in kernel space. ---------------------------------------------------------------------- weselko · 2023-11-29 12:03 This makes no sense. Postfix supports auth, since forever, with varius mechanisms. What is a modern mailserver in your opinion? Honestly curious. ---------------------------------------------------------------------- weselko · 2023-11-29 12:04 A VPS doesn't protect you from configuration mistakes. ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 12:05 Haha it was a combination of things that prompted the check. Tarkov was just the most entertaining. My brother had complained earlier in the day about slow speeds too. The second I started btop and saw the CPU at 100 I knew something was up. The DMARC reports should also have been a clue something was wrong. ---------------------------------------------------------------------- weselko · 2023-11-29 12:08 Don't sweat it. Running a mailserver is lots of fun and mail admins are really pleasant people. I managed to get all my "mistakes" delisted. They just need to know your not a spammer, they usually don't care much for such accidental bursts of email. ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 12:11 Thanks, that eases my mind a bit. Honestly some of the spammers are crazy. I think my nuc chugged through 800k+ emails before I caught it. ---------------------------------------------------------------------- McQueen2063 · 2023-11-29 12:14 receiving mail on residential IP on port 25 is no problem.no proper rdns needed for receiving. sending mail is different. there you do want to have all dns settings under your control. so how to send mail from residential ip? I relay to a supersmall VPS with a staric IP with good reputation and that one does the final delivery. that vps has proper dns settings and from there, smooth sailing. however, this is slightly off topic, because it has nothing to do with the rather funny screw up OP did :) ---------------------------------------------------------------------- stephenc01 · 2023-11-29 12:19 This is the way ---------------------------------------------------------------------- bufandatl · 2023-11-29 12:21 Yeah I‘d guess that. Was just funny that you mentioned Tarkov as often as their servers shit them selves. ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 12:21 Yeah lol, I did initially think it was their side ---------------------------------------------------------------------- weselko · 2023-11-29 12:25 Just don't give up on it. Folks here usually don't selfhostmail, but you already dipped your toes, build on that experience. If you do end up needing a VPS to relay your email, consider a antispam service provider, they handle your incoming mail and you can relay your outgoing to. I know its not selfhosted, but for fighting spam only the comercial solutions keep the filters updated. ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 12:28 I've been looking at protonmail pricing all morning :\_) Most people around me are just saying I shouldn't bother. ---------------------------------------------------------------------- KittensInc · 2023-11-29 12:44 Of course Postfix supports auth, I'm not disputing that. The problem is that it **also** supports completely anonymous submission from localhost and from local networks, and there are half a dozen ways to accidentally turn your server into an open relay. This made sense in the 1990s when every machine was hosting its own mail server for the two dozen local users, but we don't live in that world anymore and support for it should've been removed already. I haven't looked too closely into it, but something like [Stalwart](https://stalw.art/) seems closer to my expectations: just a no-nonsense batteries-included secure-by-default mail server. There are also dozens of "mail in a box" setups out there who *try* to do the same thing, but they all end up being [Rube Goldberg machines](https://mailinabox.email/static/architecture.svg) built on top of legacy software. ---------------------------------------------------------------------- weselko · 2023-11-29 13:10 I get your point and in the context of selfhosted it makes lots of sense. But I wouldn't write of postfix, its included with all the linux distros, in a lot of them as a default mailserver. And would strongly argue that it still has its place, even though its configuration isn't beginner friendly. Kinda like a crocodile, it hasn't changed much in all those years, but maybe it didn't have to. ---------------------------------------------------------------------- TheSmashy · 2023-11-29 13:11 >So I recently managed to make my Self-Hosted mailserver an Open Relay. This is bad. Just because you can self-host smtp doesn't mean you should. I've run corporate email on prem for years, for tens of thousands of users, and I have no interest in self-hosting email for myself. I've also migrated over a hundred thousand mailboxes to EXO, and there was a good reason for that. I personally think smtp is riskier than https. ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 13:18 I appreciate your experience however it's good learning and my family gets use out of it without paying a proton family subscription so I call that a win. Every system has minor failures, they're a part of life. Just look at Google drive right now. At least if I screw up, it's on me and I'm not paying someone like you to mess it up for me. ---------------------------------------------------------------------- weselko · 2023-11-29 13:23 Another thing, check if you can use your providers mailserver as a smarthost. Mine supports it. Anyway, do what feels comfortable for you. If its giving you stress, get rid of it and focus on what brings you joy. ---------------------------------------------------------------------- RecursiveIterator · 2023-11-29 13:48 Virgin Media, Vodafone, and I forgot what I had before. ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 13:53 Noted, I'll steer clear haha ---------------------------------------------------------------------- TheSmashy · 2023-11-29 14:15 That's a fair point, and it seems like you have learned a lot. And I understand difficulty trusting others, and paying for a service when all you have is an SLA to protect you (maybe). ---------------------------------------------------------------------- AmaTxGuy · 2023-11-29 14:39 Sounds like something my grandpa would say... Very smart man and so true on the advice. As I told my son.. Difference between an apprentice, journeyman and master Apprentice doesn't know how to fix his mistakes Journeyman knows how to fix his mistakes Master knows how to fix his mistakes so that it looked like it never happened That is a dadism ---------------------------------------------------------------------- ImSussman · 2023-11-29 14:51 Plusnet don’t mind you running a mail server, I recall reading articles about it on their forum. May be worth a quick search for your own reassurance :) ---------------------------------------------------------------------- ImSussman · 2023-11-29 14:56 They also allow you to control the firewall and port 25 at https://www.plus.net/member-centre/broadband/firewall ---------------------------------------------------------------------- NGL_ItsGood · 2023-11-29 15:16 Excellent write up! I've saving this one for sure. How did you go about investigating this and troubleshooting? That deserves a write up by itself! ---------------------------------------------------------------------- Alfagun74 · 2023-11-29 15:16 Another great tip for selfhosting mailservers: - don't. ---------------------------------------------------------------------- NGL_ItsGood · 2023-11-29 15:18 I am assuming that threshold would be logged in the postfix logs? ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-29 15:44 Thanks :) The process looked something like: - log into server, open btop - see pinned CPU and mysql process - search docker ps for mysql - find mailserver instance - log into mailcow, check logs - see repeated greylist warnings from google - panic - a lot - reboot the compose stack - check logs, no change - panic again - search internet - find stack overflow post on something or other - realise iptables change was bad - revert - restart stack - still flooded - check mail queue, won't load - search how to flush mail queue via command line - flush all the things - it worked - block some ips using gateway firewall - write Reddit post-mortem - cry ---------------------------------------------------------------------- NGL_ItsGood · 2023-11-29 16:25 wow, good on you for keeping a calm head and not just turning everything off and hiding under the covers lol ---------------------------------------------------------------------- buttstuff2023 · 2023-11-29 16:41 I just tested with four separate West coast ISPs and none of them are blocking outgoing port 25. Maybe it's a regional thing ---------------------------------------------------------------------- haroldp · 2023-11-29 16:50 That's awesome. Maybe things are rolling back? I just tested AT&T and they are definitely still blocking. They all blocked 25 in the late 90s, or early aughts when *every* home win95 machine was a node in a botnet sending out spam. If you go to any third party mail provider's mail client setup instructions (gmail, yahoo, etc), you will see them pushing 465 or 587. ---------------------------------------------------------------------- buttstuff2023 · 2023-11-29 17:05 AT&T broadband or mobile? I tested mobile and it's not blocking for me. ---------------------------------------------------------------------- haroldp · 2023-11-29 17:11 AT&T broadband (fiber). It probably makes sense that the mobile networks are less restricted since they have never had a spam bot problem, as far as I know. ---------------------------------------------------------------------- TheTuxdude · 2023-11-29 17:36 I don't believe this is about hosting your own equipment like routers (with recent regulatory changes, all ISPs in the US are now required to allow this - i.e. be able to run your own equipment without paying a monthly fee to the ISP). But this is more about ISPs blocking sending on specific ports like port 25 for SMTP. Instead you need to use some mail relay to send your email, who in turn will send the mail over SMTP on port 25. ---------------------------------------------------------------------- UEF-ACU · 2023-11-29 18:17 Spectrum is the second largest ISP in the US and I have zero issues sending or receiving on my mail server, no relay in place ---------------------------------------------------------------------- ohv_ · 2023-11-29 18:45 Nahh. ---------------------------------------------------------------------- TheTuxdude · 2023-11-29 19:07 Are you able to send emails using any custom MX domains or only using the ISP provided email address? Some of these restrictions/relaxations-to-restrictions are also not universal across the country. So if it's working for you, that's great. ---------------------------------------------------------------------- UEF-ACU · 2023-11-29 20:11 Custom MX domain ---------------------------------------------------------------------- UEF-ACU · 2023-11-29 20:12 Wild that they block it elsewhere, never knew that was a thing, never had a problem with self-hosting mail servers with custom domain ---------------------------------------------------------------------- KN4MKB · 2023-11-29 22:06 This is why ISPs typically block port 25. Also, I love containers as much as the next guy but for the reasons mentioned I reduce complexity in all areas of critical systems were it doesn't belong such as a email server. You are not the first to do this with docker hosted email servers and you won't be the last. ---------------------------------------------------------------------- Hotcooler · 2023-11-29 22:15 And Google should forget about it in about 1.5 months or so, if you go and plug a form that says that may be it'll do something in two weeks or so. I did something very similar with dockerised mailcow couple years ago. Though caught it sooner. ---------------------------------------------------------------------- TheTuxdude · 2023-11-29 23:23 Yeah no charter/spectrum where I am located at. But I have had a gsuite account linked with a custom domain that I use for other purposes anyway, so forwarding mail using that mail server actually works out pretty well for me. I have my relay mail servers packaged as docker containers that I can install on any of boxes in my home network to be able to send mails using this relay, so very minimal configuration and installation required. :) ---------------------------------------------------------------------- hezden · 2023-11-29 23:44 So u the reason we cant have nice things, bro ---------------------------------------------------------------------- thechubbypanda0 · 2023-11-30 00:19 I'd argue it's the spammers :_) ---------------------------------------------------------------------- Intelligent-Till-184 · 2023-11-30 01:26 I called ATT after smoking a joint and saying "what's the worst that can happen", told the rep I was doing IT Research, and that I would like SMTP/Port 25 open for some testing. The customer service rep confirmed with me that SMTP Port 25 was to be opened, I said "yup, just doing some self hosted email testing, nothing commercial". To my utter surprise, he then did the clicky clicky on his end and I now have a mailcow stack (which I just turned off to verify I'm not falling victim to the same thing OP did lololol) ---------------------------------------------------------------------- wing03 · 2023-11-30 01:41 If you have a static IP from your ISP, I've wondered if you could simply name your server the in-addr-arpa.n.n.n hostname that resolves when you nslookup the IP address and if that would pass.. ---------------------------------------------------------------------- chaplin2 · 2023-11-30 06:14 Docker bypassing ufw is very bad ---------------------------------------------------------------------- davidyoungcos · 2023-11-30 19:12 Managing email is touch. [https://federated.computer](https://federated.computer) is how I get self-hosted software without the self-hosted hassle... ---------------------------------------------------------------------- kweevuss · 2023-11-30 20:58 Somewhat. So I am just counting the frequency of the logs in postfix. I’m using mail cow, but I assume postfix is postfix. So I just look for the log when an email is sent. And I have graylog alert if it sees more than 5. But of course whatever ammount makes sense