[HN Gopher] Email Validation Logic is Wrong (2021)
___________________________________________________________________
Email Validation Logic is Wrong (2021)
Author : Tomte
Score : 41 points
Date : 2023-01-10 17:08 UTC (5 hours ago)
(HTM) web link (www.netmeister.org)
(TXT) w3m dump (www.netmeister.org)
| flexiflex wrote:
| I often find that my single character local address fails, and I
| need to add a plus something.
|
| e.g. x@familyname.com is rejected, but
| x+whydidyourejectthis@familyname.com works
| jedberg wrote:
| I used to work at Sendmail, and the people who literally wrote
| Sendmail, which had to understand every possible valid email,
| would always repeat this mantra:
|
| There is exactly one way to validate an email address, and that
| is to send it an email.
|
| No matter what regex you use, you will be wrong. Sure, use the
| 800 line regex to do a pre-validation and ask the user "are you
| sure this is correct?" if it fails, but let them use it anyway if
| they click yes, because it might just be valid.
|
| Then send it an email and if it bounces or they don't click the
| verification link, move on and delete it.
| ComputerGuru wrote:
| Yeah, but sending emails that bounce via SES or _any_ hosted
| service _hurts your business_ because it lowers your trust
| rating. You are expected to filter crap from reaching their
| servers. "But @jedberg told me the only way I could _know_
| that foo!fu*k_your-momma "<>"123:hello@fuckyou.com is an
| invalid email was to send to it" won't fly.
| jedberg wrote:
| Obviously you have to take other precautions and not just
| blindly try to validate any email you get. But my point was
| you shouldn't reject an email _only_ because it fails regex
| validation.
|
| Also, those services don't ding you if you get a single
| bounce from a new email. They only start penalizing you if
| you repeatedly send to the same email and it bounces. You
| could always put checks in place to store previously failed
| email addresses.
|
| And if your business is generating a bunch of take signups,
| it sounds like you need to put some rate limits on your sign
| up page.
|
| And of course that's not deliverable, you can't use a :
| outside of a quoted part. :) But that is actually a perfect
| example of why you can't use a regex. Maybe they modified
| their mail server to accept it.
| hooverd wrote:
| Yea, SES has a suppression list you manage that by default
| doesn't send. You can rig up an event bus to automatically
| add bounces to it.
| ComputerGuru wrote:
| The initial bounce still counts against it. (Also the
| suppression list isn't set up automatically.)
| hooverd wrote:
| As long as your volume is high you can get away with the
| occasional bounce, especially if you add them to the SES
| exclusion list.
| HWR_14 wrote:
| What's the case not to just attempt to send the email with a
| link? That's how I know if an email is correct or not.
| icedchai wrote:
| Some email services (like Amazon SES) track your "bounce rate"
| and ding you on it. When you're sending too many bounces, you
| can have operational issues. They'll shut you off.
| HWR_14 wrote:
| Sure, given a magic oracle avoiding the bounces is good. But
| what percentage of bad emails can be detected before testing?
| Especially at the level that a regular expression can detect?
| jp191919 wrote:
| I can't stand websites that say my email is invalid because it
| has a "+" in it. If the site doesn't allow it I usually take my
| business elsewhere.
| davchana wrote:
| The more bad thing is, where they take the one with + at sign
| up, but their login page throws error on email with + sign.
| Indian Pension Fund, a government website does it. You can
| signup with example+enps@anything.com but can't login. If you
| try to login with example@any, it will say email not found.
| They only way around is either go in person & submit a paper
| form; or signup from scratch with a new email.
| avgcorrection wrote:
| Why are email addresses so complex? Who needs all that junk?
| tgv wrote:
| History.
| icedchai wrote:
| It's all legacy, tech debt from the early Internet days (and
| before.) My first email address was through a UUCP feed, back
| in 1991 or so. We used bang paths. My email was something like
| uunet!host2!host1!username.
| ComputerGuru wrote:
| My problem with all these is simple: unlike dealing with people's
| names which, for the most part, they did not choose and have
| legitimate cultural reasons for the presence of special
| characters therein, it's not my problem if someone wants to
| register on my site with an (RFC-compliant!) email like
| @1st.relay,@2nd.relay:user@final.domain or a quoted email like
| (following payload is unquoted, quotes you see are part of the
| email address) "<>"@netmeister.org
|
| I _am_ going to put a (sane and reasonable) maximum length for
| email addresses on my postgres db column so it 's not stored in
| TOAST; I don't care if you're technically allowed 256 bytes
| before the @ sign: I'm not wasting space in my database so you
| can feel special.
|
| I _do_ validate that the domain not only resolves but that it
| also has a valid MX record because this benefits ~100% of my
| users and protects them from their typos. It 's not my problem if
| you want to register for _my_ service before you set up your
| nameservers - its yours.
|
| etc, etc, etc.
|
| It's a fun thought experiment but there's no way in hell I'm
| going to advise any company to actually allow any of these and
| open themselves up to a can of worms.
|
| A more sane list of rules would be things like "make sure you
| support - in the prefix or the domain" and others of its ilk.
| Also, I came across an email address that was created
| automatically from Active Directory to Azure with the apostrophe
| preserved (think "Sean O'Henry" turned into
| "sean.o'henry@example.com" and it blew my mind when that "just
| worked" in testing. _These_ would be helpful rules because _you
| 'll actually encounter them in the real-world_ and not supporting
| them will genuinely inconvenience _real people_ and not someone
| 's PhD research.
| tyingq wrote:
| Not an "instead of" approach, but the best thing I'd
| implemented when running an ecom site was a typo detector that
| prompted people to fix their email if it looked wrong, like
| "joe@gnail.com", "Did you mean joe@gmail.com?".
|
| At the time I used "mailcheck":
| https://github.com/mailcheck/mailcheck
|
| There appears to be a more modern implementation here:
| https://github.com/ZooTools/email-spell-checker
|
| It reduced the amount of badly entered emails more than any
| other approach I tried.
| citrin_ru wrote:
| > I do validate that the domain not only resolves but that it
| also has a valid MX record because this benefits ~100% of my
| users and protects them from their typos. It's not my problem
| if you want to register for my service before you set up your
| nameservers - its yours.
|
| In absence of an MX record MTA will use an A record and a small
| but non-zero fraction of domains really has an A record
| pointing to a mail server.
| jcranmer wrote:
| The HTML <input type="email"> regex is a good starting point
| for "what's a valid email address": [1] /^[a-
| zA-Z0-9.!#$%&'*+\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}
| [a-zA-Z0-9])?(?:\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?
| )*$/
|
| If you want to be a little more precise, the '.' characters in
| the local part cannot be consecutive nor occur at the start or
| end of the string. Also, the syntax does not account for EAI
| (add any non-C1 control non-ASCII Unicode character to the list
| of allowable characters), nor does it account for IDN
| (complicated, but it's IDN, there's libraries for it).
|
| Unfortunately, email is in a weird space where there's a lot of
| clearly incorrect validators out there that fail on common
| stuff (e.g., dashes in domain names, weird TLDs, etc.), but
| trying to find a good one is hard because many people want to
| "help" by flexing how good they are at covering really, really
| weird cases that you honestly shouldn't want to support, like
| relay routing, or RFC 822 comments in email addresses (which
| aren't actually part of the email address!).
|
| [1] https://html.spec.whatwg.org/multipage/input.html#email-
| stat...
| hvdijk wrote:
| Except that the HTML spec explicitly states just before that
| regex that what it checks is not whether it's a valid e-mail
| address, and that it intentionally rejects valid e-mail
| address.
| JamesCoyne wrote:
| mirror
| https://web.archive.org/web/20221217225311/https://www.netme...
| johncessna wrote:
| 15. If you want to validate an email address send them a
| validation email.
| chadlavi wrote:
| This is the way.
|
| Or if you REALLY want a regex for an input, /[^\s]+@[^\s]+/
| seems as sure as you can get without excluding a valid address
| somewhere. At least one non-whitespace character followed by an
| @ followed by at least one non-whitespace character. Any more
| specific than that and it's dicey.
|
| But also just send an email instead if you need to validate the
| email address.
| deepspace wrote:
| Ha, that is how I have always done it. Verify that there is
| at least one @ in the "address", and then send a verification
| email to it. The software that is most qualified to validate
| a user's email address, is the user's email server software.
| Why would I want to duplicate everything it does?
| arvindh-manian wrote:
| See also https://beesbuzz.biz/code/439-Falsehoods-programmers-
| believe...
| advisedwang wrote:
| Down for me; https://archive.ph/ucEpi
| ok_dad wrote:
| I have a TLD that is 4 letters and non-standard to the usual
| .com, .net, .org, or even .io addresses you see everyday. I
| regret it so much, because I still found places where 4-letter
| TLDs are flagged as invalid. Additionlly, sub-domains are often
| "invalid", and I use addresses like `site@subdomain.domain.weird`
| to keep track of who sells my information. I had to buy a .com
| domain to have a backup for those sites that invalidate my normal
| addresses. Nowdays I have started to use Fastmail's "masked
| email" feature for a lot of sites (so I just get
| `random.words1234@fastmail.com`), but that requires more config
| than just my usual scheme.
| Tomte wrote:
| Single letter local part used to be a big problem, and still
| sometimes is, but it's getting better. Hey, Microsoft has
| stopped declaring my primary mail address invalid a few years
| ago.
| iinnPP wrote:
| I have had a similar problem with the 4 letter extension
| causing issues. What is really annoying is the sites that allow
| the signup with the 4 letters but then proceed to fail at every
| other opportunity up to and including opt out(GAP). I spent an
| hour with them over the phone before anyone even understood
| what I was saying.
|
| I am also now using Fastmail addresses for anything I don't
| have some trust for and isn't a problem if lost. I no longer
| use my domain for things outside of this because it seems like
| an obvious step to simply link an unknown domain together, even
| if a laughable amount of companies don't.
| CodesInChaos wrote:
| While the RFC allows a lot of weird email address edgecases, I
| see little reason to allow most of these when a user signs up to
| my service. How many legitimate users will use quoted strings,
| dotless domains or even IP addresses?
| usea wrote:
| There is no reason to block them, and doing so hurts people.
| jcranmer wrote:
| If you try to handle email precisely correctly according to
| the specification, you will find that users will complain
| that you're handling email incorrectly. The biggest aspect of
| this is case-sensitivity: local parts are officially case
| sensitive, but the overwhelming practice is that email
| addresses are case insensitive, and people expect to find
| email addresses via case-insensitive lookup.
|
| As I say in my sibling comment, there are generally two
| purposes you might have with an email address. If your
| primary purpose is actually handling email, then that is when
| you need to be perfectly precise for email. But if your
| purpose is in using the email address as some sort of
| "universal internet ID"--this is true for the vast majority
| of uses of email addresses--then restricting the set of
| potentially valid email addresses is not only valid but a
| good idea.
|
| Quoted string local-parts and IP address literals in lieu of
| domains are things which are generally broken by middleware
| software that deals with email addresses anyways, to such a
| degree that there is no way anyone who has such an email
| address is using it as anything other than a "do you actually
| support this" email address--it can't be a valid unique-ish
| identifier for them. Additionally, allowing them to creep
| into parts of your system may break assumptions of other
| databases, which increases the chance of weird, deep failures
| that may cause security vulnerabilities. That alone is a
| pretty good reason to block them.
| JumpCrisscross wrote:
| > _no reason to block them_
|
| Sure there is. It may signal more edge-case behavior to come.
| If you're building a developer tool, you should probably
| support it. But if you're building a consumer product, that
| may not be a customer you want.
| usea wrote:
| That's true. If you're already in the business of hurting
| people, then it's a good way of identifying those you're
| trying to hurt.
|
| I pass no judgment here. I only mean to admit my incorrect
| assumption.
| JumpCrisscross wrote:
| > _already in the business of hurting people, then it 's
| a good way of identifying those you're trying to hurt_
|
| What?
|
| Some customers cost more to serve than they will ever
| make you. Still serving them may make sense. But often it
| doesn't. If you're bootstrapping a gummy bear start-up, a
| customer who pings you weekly with edge-case support
| tickets is unlikely worth the engineering effort to
| appease. If you're building an AWS competitor, on the
| other hand, you may want to hire them.
| CodesInChaos wrote:
| Which people are hurt by rejecting 1, 2, 3, 12, 13 and 14
| from this list?
| jcranmer wrote:
| The basic rule of thumb I use this: are you implementing email
| at the MTA level (needing to build/parse RFC 5321 commands or
| RFC 5322 blobs directly), or are you using email closer to a
| "universal internet ID" purpose (i.e., application
| perspective)?
|
| If you are in the former category, then yes, follow the spec to
| the letter. If you're in the latter, then screw the precise
| guidelines of the spec and reject emails that are very unlikely
| to be valid: no quoted localparts, no IP address literals. In
| addition, go ahead and say that email is case-insensitive (more
| precisely, case-preserving).
| SAI_Peregrinus wrote:
| Does it matter? Even if the email is a bog-standard
| USERNAME@gmail.com, you've got no way to know if the email is
| real until you send a challenge email to the address. If you
| have to do that anyway, why bother validating anything
| unnecessary?
| CodesInChaos wrote:
| 1,2,3,14 add a lot of complexity which increases the
| potential for bugs and security issues.
|
| 12 is almost certainly a typo
|
| 13 is probably a spammer
|
| Also <input type=email> will reject several of these as well,
| so if you have such an edge-case email in your database,
| you'll run into issues with html forms.
| davchana wrote:
| Correct. Anytime a site ask me for an email before giving me
| a file, a code or download, I use something@gmail.com (the
| literal word something) & most of the time the site gives me
| what I am looking for. If they send that link in email only,
| then I use a temporary email.
| roflyear wrote:
| Well, why block them unless you have a good reason to?
| mgkimsal wrote:
| it implies support, and may place more burden on testing. if
| you test with XYZ only, but also allow ABC... then ABC stops
| working... you'll have pissed off customers.
| c22 wrote:
| Whereas if you block ABC outright you'll just have pissed
| off people who aren't customers!
| mgkimsal wrote:
| Sure, but that's a different story. You _might_ piss them
| off and lose customers. You may also be knowingly
| increasing your support costs.
|
| For the average company trying to navigate these sorts of
| decisions, supporting users with emails like
| "a$123@1-2-3.12397.museum" probably isn't high on their
| list of priorities, regardless of whether it's "valid" or
| not.
| mkehrt wrote:
| How many legitimate users will use pluses or dots which you
| forgot to allow in your email validator?
|
| How many legitimate users will use _capital letters_ , which I
| had to add to an _in-production_ email validator on an app with
| millions of MAUs?
|
| Just send a confirmation email to whatever string they provide
| you.
| Thaxll wrote:
| Yeah RFC says something reality is very different, allowing
| multiple @ in 2023 is just stupid and not how emails work, as a
| matter of fact gmail does not allows it.
| velcrovan wrote:
| When validating email addresses in my Atom feed generation
| library, I use a "common-sense subset" of RFC 5322 [1].
|
| If that email address validation is "wrong" anywhere in the sense
| of conflicting with real-world usage, it's probably mostly in not
| allowing UTF-8 characters in the local or domain parts. I'm
| guessing that for that I'll need to dig into RFC 6531 and 5890.
|
| [1]: https://docs.racket-lang.org/splitflap/mod-
| constructs.html#%...
| paxys wrote:
| The article comes off as a "gotcha! Your service is broken!!" but
| ask any developer or product owner about this and they will
| simply say that they don't care about the 0.00001% of users
| signing up with such emails.
___________________________________________________________________
(page generated 2023-01-10 23:01 UTC)