https://gabrielsieben.tech/2024/05/17/thinking-out-loud-2nd-gen-email/ Skip to content Gabriel Sieben * Home * Blog * Contact * AR Sandbox * Homelab * + Back Thinking out loud about 2nd-gen Email Posted byGabriel Sieben May 17, 2024May 17, 2024 4 Comments on Thinking out loud about 2nd-gen Email Note: This is just me, thinking out loud; you absolutely do not need to think that I have carefully thought this through, or that this is a good idea. With expectations set as low as possible, let's continue. There are many old pieces of tech still in use, but there's one that grinds my gears every time I try to use it: Email. For users, email works pretty well. Sometimes it sends too many emails to Junk, but Email is old, reliable, easy to understand, and relatively easy to search. It's a good system, and I'm not eager to replace it with Slack anytime soon. However... the backend for email, is a mess. In escalating order (and "we" is used in a very imprecise, broad hand-waving sense for technologists): * Many things in Email have no spec; even basic things. For example: When you reply, are you replying at the top of the message, or the bottom? It might even be a political question, depending on who you ask. This has been worked around by email clients basically guessing the order and rarely even showing the email's original text, putting layers between the user and the actual message. * What HTML are you allowed to put in an email? Well... it depends. When there's no spec, and Microsoft Outlook abuses the Microsoft Word HTML renderer, it gets ugly. There's no guarantee the receiver even has an HTML renderer, and then it's even more ugly. * How do you encrypt an email, from the provider itself? Well, we invented this thing called OpenPGP, but almost nobody used it. Then it turned out to have major flaws. * We couldn't always make sure emails were authentic. So we invented SPF. * Then it turned out SPF didn't fix everything. So we invented DKIM. * Then it turned out DKIM didn't fix everything. So we invented DMARC. * And then it turns out DKIM itself has major flaws that also bypass DMARC by extension. * Also, because we threw on yet another layer called BIMI, which itself relies on DMARC, when DMARC relies on DKIM, and DKIM has flaws, then it's exceedingly bad for users. * Even if your sender has DMARC, 68.2% of records have their policy set to p=none. This tells DMARC to basically... do nothing! (Sure, it technically speaking notifies the domain owner, but it's clear the domain owners don't care.) * Did I mention all of the above, plus aggressive anti-spam policies, makes self-hosting email insanely difficult? * Last, but not least, there's the inane juggling of IP reputation. Some IP addresses are "cleaner" than other addresses, especially on shared systems like SendGrid or AWS SES. This makes signing up for a mass-mailing account, for whatever reason, messy; and causes countless surprise instances of legitimate emails going to Junk. Combine that with IP Address depletion, and the number of mostly clean addresses is shrinking over time. My gut reaction to the above is that we've got a lousy spec, with decades of cruft and unofficial spec, and we aren't that great at securing it, or making sure messages are authentic. So... could we do better? Thus the hypothetical: 2nd-gen email. Your initial reaction might be: That would be pointless, because not everyone would opt into it, and it would break compatibility all over the place. My thought is... that's not necessarily a given. Imagine this: * We create a new DNS record, called MX2. Most email services, then, would have an MX2 and MX record. Older services only have MX. * If an ancient, 20 year old email client, tries to send a message - it finds the MX record and sends the message just like normal. A modern client sees the MX2 and sends the message there if it exists; otherwise, it falls back to MX. * From there, the email services which implement MX2 would publish a public date, on which all messages sent to them by the old MX record, will be automatically sent to Junk. If just Microsoft and Google alone agreed on such a date, that would be 40% of global email traffic. If the above looks slightly familiar, it's because this strategy already worked, in a sense, with the transition from HTTP to HTTPS. We threw away a multi-decade-old protocol, for a new and more secure one. We set browsers to automatically upgrade the connection wherever possible, and now warn users about insecure connections when accessing HTTP (especially on login pages). Nevertheless - users can still visit HTTP pages, ancient browsers still work on HTTP, but most websites have gotten the memo and upgraded to HTTPS anyway. The incentive to upgrading to MX2 would be simple: Your messages, while they still would arrive, would go to Junk automatically past the publicly posted date. No business wants that, even if users are already trained to expect, and act, like that can happen. Thus, the incentive to upgrade without truly breaking any day-to-day compatibility. Personally, I think that such a transition could go even faster than the HTTP to HTTPS transition. Self-hosted email is not very popular in part because of the complexity of the current email system, so between Microsoft, Google, Amazon, Zoho, GoDaddy, Gandhi, Wix, Squarespace, MailChimp, SparkPost, and SendGrid - you have most of the email market covered for the US; anyone not in the above list would quickly fold. The relative centralization of email, ironically, makes a mass upgrade to email much more achievable. What would a 2nd-gen email prioritize then? Everyone has different priorities, but I'd personally suggest the following which would hopefully win a broad enough consensus if this idea goes anywhere (though experts, of which I am not one, would have plenty of their own ideas): 1. A standardized HTML specification for email; complete with a test suite for conformance. Or, maybe we just declare a version of the HTML5 spec to be officially binding and that's the end of it. 2. Headers for email chain preferences, or other email-specific preferences (i.e. Is this email chain a top-reply chain, or a bottom-reply chain? The client shouldn't need to guess, or worse, ignore it.) 3. If an email has a rich, HTML view; it should be required to come with a text-only, non-HTML copy of the body as well; for accessibility, compatibility, and privacy reasons. 4. All MX2 records must have a public key embedded in the record. To send an email from the domain: - A hash of the email content, and all headers, is created. - This hash is then encrypted with the private key, corresponding to the record's public key. - This header is then added to the email, as the only permitted untrusted header. - When an email is received, the header containing the hash is decrypted with the DNS public key, and the rest of the email is checked against the hash for integrity and authenticity. 5. Point #4 is a lot like DKIM and DMARC right now, except: - There would always be an automatic reject policy (p=reject) . Currently, only 19.6% of email services which even have DKIM are this stringent. - If headers do need to be added to an email, the spec can carefully define carve-outs for where untrusted data can go (i.e. if the spam filter wanted to add a header). - There also could be standardized carve-outs for, say, appending untrusted data from the receiving server to a message body (i.e. your business could add data to the body's top or bottom indicating that the message from an external recipient and you have legal obligations, but your email client can also clearly show that this was not part of the original message and is not signed). - As such, the signing would not need to work around email compatibility to such an extent as DKIM, reducing the likelihood of critical flaws. 6. By simplifying the stack to the above, eliminating SPF, DKIM, and DMARC (and their respective configuration options), and standardizing on one record (MX2) for the future, running your own self-hosted email stack would become much easier. Additionally, the additional authenticity verifications would hopefully allow spam filters to be significantly less aggressive by authenticating against domains instead of IPs. 7. Point #6 is the biggest change - we're no longer authenticating, or caring, about the IP Address that's sending the email. Every email can and always would be verified against the domain using MX2 records and the public keys in them. Send a fake spam email? It doesn't have a signature, so it gets tossed without any heuristics. Send a real spam email? Block that domain when there's complaints. Go after the registrar (or treat domains belonging to that registrar as suspicious) if needed. This would mostly eliminate the need for IP reputation by replacing it with domain reputation - which, at least to me, is a far superior standard with more understandable and controllable outcomes(1). 8. Clients which implement MX2 can, optionally, have an updated encryption scheme to replace OpenPGP. Something like Apple's Contact Key Verification. Hopefully there would be forward secrecy this time. If you have got great counterarguments, let me hear them. (1) This would, perhaps, be the one and only "new feature" we could advertise to users. Not getting emails? You can just type in the name of the website, and always receive the emails. Edit 1, for clarification: For bulk senders, there would be multiple MX2 records on the domain, each containing a public key for every authorized sender. One of those records would have a marker indicating it as suitable for incoming mail. Posted byGabriel SiebenMay 17, 2024May 17, 2024Posted inUncategorized Published by Gabriel Sieben Gabriel Sieben is a software developer from St. Paul, MN, who enjoys experimenting with computers and loves to share his various technology-related projects. He owns and runs this blog, and is a traditional Catholic. In his free time (when not messing with computers), he enjoys hiking, fishing, and board games. View more posts Post navigation Previous Post Previous post: "Parental controls? What parental controls?" Join the Conversation 1. [dd3bb3c] 2. [df12a26] 3. [41ac83e] 4. [b393492] 4 Comments 1. [dd3bb3c]Brian Walker says: May 17, 2024 at 1:56 pm I think point 6 is at odds with your goal of having the big email providers support this standard. The existing complexity is their most. Reply 2. [df12a26]Kevin says: May 17, 2024 at 2:11 pm From my perspective as an academic user with emails from multiple orgs (as is common for many academics), email is becoming borderline unusable. Orgs have become so paranoid about phishing that on top of the SPF/DKIM/DMARC pile, they're putting in all kinds of poorly-thought-out-and-implemented homegrown restrictions. There's basically no way anymore to be confident that you're receiving all the email you should be, or that others are receiving emails that you send. Somehow, we need to put up a wall to keep out the dregs of the open web, while ideally maintaining some degree of discoverability and serendipity. Something like what you propose (signing and domain reputation) seems like it would solve that. Reply 3. [41ac83e]dan says: May 17, 2024 at 2:34 pm Any mail server has the duty to deliver, in confidence, and not classify as spam, any legitimate email that is sent. 'Legitimate' means that it is sent in good faith and acceptable to the recipient. DKIM, SPF, HTML, MX2 and other schemas do not address this requirement. The primary major problem with the existing email system is the failure to deliver valid, legitimate email. The MX2 proposal seems to aggravate this problem, by suggesting that even more legitimate email would be not properly delivered. Reply 4. [b393492]ricky712 says: May 17, 2024 at 3:10 pm Many moons ago, I experimented with sender-pays email using hash cash, and I think if you are going to build a second version of email, sender-pays should become a core part of the the new environment. After thinking about the problem for several years, I believe a sender-pays system has three essential components. The first one is who stores the message. Current email fails because spammers push the cost of storage on the receiver. Forcing the sender to hold the messages until the receiving server fetches them makes it more expensive for spammers/ marketers to send large volumes of emails. the receiver only gets a notice of the message and the initial headers to help distinguish who the message is from and what it's about. The second component is a mechanism for imposing a mechanism for limiting the rate at which a server can send email. Today all the techniques used today put that cost on the receiver. Using a hashcash token puts the cost on the sender. The problem with the first version of hashcash tokens for email is that it was a fixed cost. The third component fixes that problem by allowing the receiver to specify the cost of sending a message based on the sender's reputation. This allows the receiver to reduce the cost to zero if the individual or the server has a good reputation. There was no way to change the token size based on the sender's reputation or the amount of available compute to solve the hashcash problem. Reputation management is not complex. The more messages received that are not objected to, the cost of sending slowly drops. Declaring a message spam, however, jacks up the price quickly. The Hashcash tokens provide another benefit in that if a site is blocked inappropriately, it can still be used to send messages to clear up the problem because if you're willing to pay the computing cost, you can still deliver a message in contrast to the blacklists which prevent you from ever communicating again. As for fixing spf/dkim, I think we might be better off using a variant of wire guard to encrypt the mail message stream and letting the mail server put a series of public keys in their DNS record. Reply Leave a comment Cancel reply Your email address will not be published. Required fields are marked * [ ] [ ] [ ] [ ] Comment * [ ] Name * [ ] Email * [ ] Website [ ] [ ] Save my name, email, and website in this browser for the next time I comment. [Post Comment] [ ] [ ] [ ] [ ] [ ] [ ] [ ] D[ ] Gabriel Sieben, Proudly powered by WordPress. Privacy Policy