[HN Gopher] How IMAP works under the hood
___________________________________________________________________
How IMAP works under the hood
Author : michidk
Score : 197 points
Date : 2025-03-29 09:05 UTC (2 days ago)
(HTM) web link (blog.lohr.dev)
(TXT) w3m dump (blog.lohr.dev)
| therein wrote:
| Interesting no attempt has been made to make it at least be less
| heavy on networked bytes. Especially since it is old and was
| meant to be used on a connection with no compression or
| encryption.
|
| HasChildren could have been Parent, HasNoChildren could have been
| Leaf or Child. And so many more things.
| philipwhiuk wrote:
| The protocol has ossified and been entrenched. In general more
| efficient usage of IMAP relies on extensions to the protocol.
|
| A modern replacement (JMAP) hasn't been adopted by major
| providers.
|
| If you really cared about data transfer size you'd use
| something like Protobuf.
| jgalt212 wrote:
| IMAP COMPRESS?
| FuriouslyAdrift wrote:
| JMAP is fully supported (since 2019) for Fastmail
|
| https://www.fastmail.com/dev/
| n3storm wrote:
| Jmap was developed by Fastmail, meanwhile all primary FLOSS
| mail servers do not support it and just new projects like
| Stalwart are starting to bring it up.
| calvinmorrison wrote:
| Well and you could of course server JMAP over something like
| Protobuf, JSON is used for ease of use but it's not like
| thats required feature.
|
| The bigger benefit of JMAP is it's well designed for dealing
| with deltas and changes and syncing data requiring less
| roundtripping than IMAP.
| Mailtemi wrote:
| Can confirm that (implemented JMAP). Deltas and asking for
| the entire mailbox instead of folder by folder is really
| good. And as a side effect: 1. Less client-side logic for
| sync. 2. There are many JSON parsers compared to IMAP,
| making it way easier to use. For example, in C++, you only
| need a JSON headers-only library, whereas IMAP is meh--only
| Linux, or clunky usage. Btw, I made one from scratch (IMAP
| Parser). 3. Goodies like offloading HTTP to the mobile
| network stack, which supports TLS 1.3 and offline
| (background) sync, compared to manually extracting
| certificates from an IMAP connection to validate against
| the device keychain. It's just 10x easier.
| Aloisius wrote:
| IMAP 4rev2 merged in a most of the common extensions into the
| base protocol. Sadly they left out the THREAD, QRESYNC and
| OBJECTID, though they reference them.
|
| Of course, major providers like Gmail don't support that as
| well.
| avar wrote:
| Networked bytes generally don't matter, networked packets do.
| Would your proposed change move the needle on that?
| shiandow wrote:
| It could have, but for stuff you only do once per session that
| seems excessive. Better to have names that need no explanation,
| especially for stuff that I think is completely optional.
| zaik wrote:
| Despite people always complaining about their perceived
| inefficiencies, standard protocols like IMAP or XMPP always
| seem to work on a crappy connection, when most of the modern
| web doesn't.
| Loudergood wrote:
| Back in the day when I worked at a WISP with 2.4Ghz mountain
| top antennas the one thing that would work when we were
| having massive icing issues was email. It was very impressive
| how it would get through even with massive packet loss and
| transfer rates measuring in bytes per second.
| nirui wrote:
| Probably wrong context, but the more code I wrote, more I like
| the these `Has`+Noun style naming than just Noun. Reading
| `HasChildren` will give you a clearer expectation of what the
| function would do and return, while `Parent` gives far weaker
| indication.
|
| Maybe they thought the same when they were designing the
| protocol.
|
| Also, in the context of email, given the size of each mail
| (including headers and body), these bytes "waste" maybe
| insignificant.
| throw0101d wrote:
| > _Interesting no attempt has been made to make it at least be
| less heavy on networked bytes._
|
| Kind of a surprising observation given the first spec (IMAP2)
| was released in 1998, when dial-up was still a thing:
|
| * https://datatracker.ietf.org/doc/html/rfc1064
|
| IMAP4 was in 1994:
|
| * https://datatracker.ietf.org/doc/html/rfc1730
|
| ITU V.32 gave us 9.6 kbit/s in 1998, and V.34 was 28.8 kbit/s
| in 1994:
|
| * https://en.wikipedia.org/wiki/List_of_ITU-T_V-
| series_recomme...
| throw0101d wrote:
| s/1998/1988/
| ocdtrekkie wrote:
| When every new Gmail client ships with an entire web browser
| embedded to load their hundred megabytes of JavaScript, I think
| we've long jumped the shark on caring about brevity in the
| length of information in the protocol itself.
|
| It might have mattered back then but now it would be less than
| a rounding error.
| lotharcable2 wrote:
| IMAP had its day in the sun, but the advent of big webmail
| providers (especially gmail) has killed off the advancement of
| email clients. Now all major development is focused on trying
| to recreate Gmail to varying degrees of success. It all ends up
| internal to one or another corporation so they are just all
| endlessly reinventing the wheel with IMAP just being relegated
| to a afterthought front end to some sort of search-based
| backend.
|
| Actually having a email client software running on your machine
| is extremely nitch and is mostly in the realm of self-hosters
| and legacy holdouts that won't let their clients go.
|
| A most advanced modern approach is to just use POP3 to download
| your emails to a local Maildir and have them indexed there non-
| destructively. And then sync between your various machines that
| you want your email accessible using some sort of file sync or
| P2P solution.
|
| I use notmuch for this. It automatically indexes and tags
| emails and thus enables much more advanced email management
| solutions then what can be offered over something like IMAP.
|
| The main advantage of this is that 'folders' are managed
| virtually. There is no shuffling or copying or editing of
| emails done normally. I only have to worry about backing up my
| emails and notmuch config as all the rest can be regenerated
| relatively quickly.
|
| This is more or less replicating what Gmail and other webmail
| providers do server side.
|
| Where as the traditional approach shuffling and moving and
| deleting of emails on some imap server is fairly dangerous and
| expensive operation. Mistakes can lead to data loss and are
| often very difficult to reverse.
| csb6 wrote:
| There is an extension that adds compression. [0] I think most
| widely-used servers implement it (at least Gmail and Dovecot
| do)
|
| [0] https://www.rfc-editor.org/rfc/rfc4978.html
| rmccue wrote:
| I started writing a guide to IMAP back when I was working on an
| email client:
| https://github.com/rmccue/griffin/tree/master/docs/imap
|
| Pulling large amounts of data for things like threading can be
| difficult on certain servers; my preferred approach ended up
| being to pull every ID and thread ID to maintain an in-memory
| tree. (This was, iirc, partially because Gmail's implementation
| was slightly crippled with relation to threading.)
|
| (I never finished the guide because I stopped on the project,
| alas - if IMAP were easier to work with, I might have finished
| it! And sadly, no JMAP support on Gmail, and the gateway was
| broken.)
| jeffbee wrote:
| > Gmail's implementation was slightly crippled
|
| Gmail is not "crippled". A tiny but vocal community of old
| nerds have a petrified mental model of email that they
| associate with unix IMAP software from the 1990's, but those
| concepts do not appear in the IMAP standards anywhere.
| dbcurtis wrote:
| That is an immature view on how real products and real
| standards work. The standard document may say one thing, but
| what people do in the real world is the real standard.
|
| For context: I spent 11 years at Intel managing pre-silicon
| and post-silicon processor validation. No processor that does
| only and exactly what the Programmers Reference Manual says,
| and takes the phrase "undefined behavior" seriously, will be
| successful. Google would do well to adjust their philosophy.
| jeffbee wrote:
| If an x86 implementation was imperfectly compatible with
| Intel CPUs, nobody would buy it. Gmail, on the other hand,
| is a massive market success. It is those who shout that
| IMAP must be exactly and only whatever mutt+uw was doing in
| 1997 who are on the wrong side of history.
| XorNot wrote:
| It's a free email account, it is not at all clear that
| "weird IMAP" is core to that success?
|
| Certainly I moved away to Fastmail, which has better IMAP
| support (but mostly because Google having full control of
| my email address was becoming too big of a risk and
| Google Apps is expensive for your own domain).
| rprospero wrote:
| It's been an odd running theme for me today that I've
| misinterpreted posts. Up until your final sentence, I
| thought that the thesis of your post was:
|
| The standard document may say one thing, but what people do
| in the real world is the real standard. If your software
| has issues with the world's most popular IMAP server, you
| need to adjust your software to be compliant with the
| standard.
|
| I'm personally more sympathetic to your actual conclusion,
| but it's odd how often a single argument can be used to
| support two conflicting beliefs.
| rmccue wrote:
| Yeah, agreed! I titled my guide "Practical IMAP" for
| reason (and almost called it "IMAP As She Is Spoke"). The
| standards are useful to a point, but actually to
| jeffbee's point the internet has evolved a lot since
| then, and how to actually work with modern email is a
| pretty underdocumented - including by Google themselves
| on the Gmail-specific parts.
| donnachangstein wrote:
| If history has taught us anything, it's that Google is
| happy to willfully ignore, rewrite, and use their market
| dominance to snuff out any existing standards if they see a
| way to seize control and make money off something.
| ptx wrote:
| Which concepts are you referring to?
|
| RFC 9051 says that it was "developed for" RFC 822 and that
| one should now refer to RFC 5322 instead. And RFC 5322 does
| discuss threading, which is the what the comment you're
| replying to was talking about, so that concept does appear
| (by reference) in the IMAP standards.
|
| Are there other parts of the community that implement IMAP
| using the same "mental model of email" that Gmail is using?
| Or are you saying that anyone using IMAP with anything but
| Gmail is an old nerd and a tiny minority?
| rmccue wrote:
| My memory is hazy on it, _but_ from memory, there were some
| issues around IDs and persistence which don't occur on other
| servers, as well as the auth being a bit funky. (Gmail also
| uses its own extension to the protocol for IDs instead of
| using eg OBJECTID.)
|
| To be clear, I have no opinion on IMAP or what things
| _should_ do; I certainly was not using IMAP software in the
| 1990s :) However, trying to implement a client that works
| across providers does mean trying to operate per the
| standards, which was a struggle and why I started documenting
| it :)
| mjl- wrote:
| > I started writing a guide to IMAP back when I was working on
| an email client
|
| I would be very interested in hearing from developers about how
| they write email clients that need to work with all the servers
| out there, with the varying levels of IMAP4 (original, rev1,
| rev2, combinations of at least a dozen extensions) and various
| levels of buggy behaviour.
|
| I'm assuming a client developer would implement various
| "profiles". With an "advanced" profile for servers that
| implement things like CONDSTORE/QRESYNC, and a "very basic"
| profile that only do the absolute minimum. And probably a
| profile or two in between. When you encounter unexpected
| behaviour (eg bad syntax in protocol), you would downgrade the
| profile for that server for a while (it may get fixed)? If it
| works like this, I'm curious to the profiles developer choose.
| If it's not like this, I wonder how client developers work
| around compatibility issues (can't just keep reconnecting and
| trying the same if it results in an error!).
| csb6 wrote:
| As someone currently working on an IMAP client, my strategy
| has been to start by implementing the base IMAPrev2
| specification and checking that it still works with a rev1
| server. RFC 9051 has guidance on how to handle servers that
| use older IMAP protocol versions and is largely backwards
| compatible.
|
| The goal is to have the client work without any extensions.
| E.g., if the server doesn't have the UNSELECT capability it
| falls back to selecting a nonexistent mailbox (which has the
| same effect). Obviously there are some very useful extensions
| that the client needs to support, but those can be viewed as
| optimizations to reduce network traffic or add extra features
| if the server supports them.
| camgunz wrote:
| I've been working on some email stuff and I think probably four
| things are vexing about IMAP:
|
| - The grammar is hard. I built a parser using lpeg and I'm
| incredibly glad I did--doing it ad hoc won't lead to good
| results.
|
| - It's an asynchronous protocol. You can send lots of requests to
| a server and you have to tag them so you can match them up with
| responses later. You don't generally want to do that in a client;
| i.e. you don't want to do these transactional things over an
| async connection and track state across all of it. You want to
| like, block on deleting things, renaming things, sending things,
| etc.
|
| - IMAP is multi-user--it's built around the idea of multiple
| clients accessing the same mailbox at the same time and streaming
| updates. Another thing you really don't want to deal with when
| building an email client.
|
| - There's functionality that you basically shouldn't use; the big
| one is search. Even the specs more or less say "good luck using
| this".
|
| You can group all this under the a heading of "we thought people
| would use this over telnet", but attachments and non-plain-text
| email I think made that non-viable.
|
| I think this all means probably every non-web email client treats
| IMAP like POP and keeps its own store. I haven't done a survey or
| anything, but I'd be surprised if that weren't true.
| mr_mitm wrote:
| > I think this all means probably every non-web email client
| treats IMAP like POP and keeps its own store. I haven't done a
| survey or anything, but I'd be surprised if that weren't true.
|
| Pretty sure mutt doesn't. It only caches the headers.
| ldite wrote:
| same for alpine (https://alpineapp.email/)
| camgunz wrote:
| Oh, yeah I guess that's what I mean, and then your connection
| can be used basically (again) like POP
| mr_mitm wrote:
| Ah, now I see what you mean. True.
| throw0101c wrote:
| > _Pretty sure mutt doesn 't. It only caches the headers._
|
| Seem like it can cache message bodies:
|
| * http://www.mutt.org/doc/manual/#message-cachedir
|
| * https://neomutt.org/guide/optionalfeatures.html#body-
| caching
| pferde wrote:
| I've found the server-side search functionality works very
| well, if you have good server implementation. Dovecot's, for
| example.
|
| And as for treating IMAP like POP, yes, there are clients that
| only pay lip service to "having IMAP support", only so that
| they can have one more green checkbox in feature list that
| their present in their marketing. But there are also more
| serious clients.
| jcranmer wrote:
| > - There's functionality that you basically shouldn't use; the
| big one is search. Even the specs more or less say "good luck
| using this".
|
| Message sequence numbers. Every folder in IMAP has its emails
| numbered from 1-N, with no holes, so if you delete a message,
| everything after it has its message sequence number decremented
| to close the hole. Except IMAP is multiclient, and the message
| can be deleted by other connections than the one you're
| currently on. But the server is only allowed to tell you about
| message deletions at certain points, so now the server has to
| keep essentially per-client message sequence number state and
| carefully make sure that everyone is kept in sync... and it's a
| recipe for disasters in practice. Any sane client will instead
| use UIDs for everything (and any sane server will implement all
| the UID extensions to let UIDs be used for everything).
|
| The other fun corner case I recall is that IMAP part numbering
| is a little unclear what happens around body parts of content-
| type message/rfc822. So I crafted a message that had a
| multipart/mixed with one leg being a message/rfc822 whose body
| was a message/rfc822, and tested the output on all 4 IMAP
| server implementations I had accounts on at the time to see how
| they handled the part numbering. I got back 4 different
| results. None of them were considered correct by the IMAP
| mailing list discussion on the experiment.
|
| > I think this all means probably every non-web email client
| treats IMAP like POP and keeps its own store.
|
| The distinction I would use is thin client versus thick client.
| Most clients like Outlook or Thunderbird are thick clients,
| which need to maintain their own local database for other
| reasons (like supporting offline mode or having database
| features not necessarily supported by an IMAP server, like
| arbitrary tagging). If you've got a local database, it's much
| saner to use IMAP essentially as a database synchronization
| protocol rather than trying to build a second implementation of
| all of your features on top of IMAP's native features,
| especially given that IMAP server implementation of these
| features is generally questionable at best.
|
| IMAP was originally designed, it seems to me, to make thin
| clients easy to write (I can see how something like pine would
| be a very thin veneer over the protocol itself). But it's been
| clear over the past few decades that most clients are of the
| thick variety; things like QRESYNC and CONDSTORE were added to
| make the database synchronization steps a lot easier.
| mjl- wrote:
| > Any sane client will instead use UIDs for everything
|
| Yes! Since last year there's the experimental UIDONLY
| extension that allows clients & servers to operate entirely
| without message sequence numbers. Saves quite a bit of
| accounting and possibly memory (for large mailboxes). It's a
| bit surprising the RFC so recent.
|
| > [... thick vs thin clients ...]
|
| For some programmatic email access, IMAP seems quite easy to
| use. Just write a programmer-readable line of text as command
| (eg fetch/search/store/etc) and parse the response lines
| (that's a bit trickier in many cases, probably needing a
| library, but you may be able to get away without it some of
| the time) and you're done.
|
| About thick clients: Storage capacity is growing much harder
| than my email, also on mobile. I would be fine with thick
| clients that sync all messages. Allows for simpler protocols
| and simpler implementations. An email access protocol is then
| not much more than a two-way sync of email message files and
| associated "dynamic data" like flags/keywords.
|
| But IMAP and JMAP are doing many things at once:
| synchronization for thick clients, and online access for thin
| clients. JMAP is specifically targeting the web (thin
| clients), so what was old is new again (although base IMAP
| isn't too helpful to (web-based) thin clients either).
| mjl- wrote:
| > You can send lots of requests to a server and you have to tag
| them so you can match them up with responses later
|
| Yes, you can match the final OK/NO/BAD responses with the
| original command based on the tag. The annoying thing is that
| each command typically sends most of its data in "untagged
| responses". And IMAP servers can send unsolicited untagged
| responses at any time too. It's quite tricky to handle all
| those responses correctly in email clients. Indeed, normally
| you just want to send a command, and get all the data for that
| command back until the finishing response. IMAP clients
| typically open multiple connections to isolate commands.
| camgunz wrote:
| Oh shit, of course. The main thing is you can't rely on a
| double \r\n to terminate a response; you have to track the
| tag, and yeah it's mega annoying. But multiple connections
| fixes that, wow.
| userbinator wrote:
| _You can group all this under the a heading of "we thought
| people would use this over telnet"_
|
| No, the grammar is too convoluted for that. It's more like "we
| thought text-based protocols are better".
|
| POP3 and SMTP are relatively easy to use manually. IMAP is not.
| bpev wrote:
| For the extra interested... reminded me of this one:
| https://explained-from-first-principles.com/email/
| nashashmi wrote:
| Can IMAP be used as a file server system? Outlook has this
| functionality, where files can be stored directly outside of
| emails.
| SSLy wrote:
| most servers would reject entries without any headers as
| malformed
| thesuitonym wrote:
| This is one of those questions where the answer is
| _technically_ `yes ', but for all practical purposes should be
| considered `no'.
| superkuh wrote:
| Of course these days the mega-corp walled garden email providers
| don't really follow standards like IMAP. IMAP will not work with,
| say, Google's gmail or Microsoft office365, or AT&T ISP email,
| etc, etc. They have each implemented their own proprietery out-
| of-band authentication system that only works over HTTPS using
| the OAuth2.0 toolkit to build it. Any email client that does not
| explicitly design for each particular OAuth2.0 implementation
| (each megacorp's is slightly different) will not be able to
| connect over IMAP (unless they login via HTTPS using a web
| browser and set up "app passwords" for google, or similar for
| others).
| calvinmorrison wrote:
| When I worked at fastmail there was of special fix code. You
| see it with firefox and chrome too, oh this popular site is
| breaking lets put a hardcoded if statement in. I specifically
| remember magic fixes for iCal.
| jeffbee wrote:
| Struggling to think of a way in which "IMAP will not work with
| gmail". Please explain.
| Aloisius wrote:
| It can, but it does require doing a lot of Google-specific
| things (set up a google cloud account, create a consent
| screen, get a security review, justify your usage of the IMAP
| API instead of the web APIs to them, find the right scopes,
| etc) or instruct users to go through multiple screens in
| their google settings to create an app password.
|
| Google _really_ doesn 't want you to use IMAP. They're trying
| to push everyone to their neutered web apis instead.
| jeffbee wrote:
| You seem to be taking the perspective of an application
| developer or something like that? Certainly for users all
| they need to do is roll in with their favorite IMAP client
| and use it. All of what you said applies not at all to
| users.
| Aloisius wrote:
| _> roll in with their favorite IMAP client and use it_
|
| That's just it. Lots of client developers, especially
| open source ones, balked.
|
| So to use something like mutt with gmail requires a user
| go into their google settings, set up 2fa then create an
| app-specific password. And if a user is on a Google
| Workspace account with "insecure" passwords turned off,
| they either have to do all the gcloud/consent/etc. stuff
| themselves or steal a client secret from another client.
|
| Oauth client secrets aren't really compatible with open
| source and oauth flows don't work well in terminals.
| Google's onerous process didn't help and on top of that,
| using oauth means getting hit by Google's quotas.
|
| Who knows how long Google will support app-specific
| passwords? Or perhaps they'll start forcing 2fa via their
| own gmail app every login.
| slightwinder wrote:
| > IMAP will not work with, say, Google's gmail or Microsoft
| office365
|
| Except they do, to some degree. It works well enough that my
| Thunderbird allows me fetching or moving of mails. Not sure
| about advanced features like search or server-side filtering,
| never tried them, but this seems to be a bit more wacky with
| other clients & servicers too.
|
| > They have each implemented their own proprietery out-of-band
| authentication system that only works over HTTPS using the
| OAuth2.0 toolkit to build it.
|
| True. Gmail at least had a long while application-passwords. I
| think they changed this only recently? Or are they still a
| thing?
| XorNot wrote:
| OAuth works fine for major providers in Thunderbird...unless
| you're using hard tokens with pin number requirements right
| now (the entry form doesn't display).
| tiahura wrote:
| I've been looking to migrate from Exchange-Outlook, but there
| really aren't any options. There just isn't an open source
| solution to have an integrated email / tasks / events / contacts,
| with consistent labels across item types and reminders.
|
| With Outlook, i can use a custom view to see every item in a
| category flagged for follow-up. I can also set a reminder on a
| contact, or drag an email or contact onto my calendar to create
| an event.
| beagle3 wrote:
| Thunderbird has it all. I don't like the way thunderbird does
| it, but I like outlook even less... so thunderbird it is.
| tiahura wrote:
| Thunderbird doesn't do most of this. It uses tags for emails
| and categories for events. Also, no way to view all items
| with same category. Can't drag contact to calendar or task
| list. Can't set specific reminder times. ...
| azhenley wrote:
| I've been trying to get approval from Google for the sensitive
| scopes to use IMAP, and they classified us as needing "CASA Tier
| 3 Security Assessment". It looks like it is going to be a long,
| tedious, opaque, and expensive process.
| isaachinman wrote:
| What are you building?
| azhenley wrote:
| An email client for Gmail.
| accrual wrote:
| That was a pretty interesting read. I didn't realize one could
| interact directly with an IMAP server like we can with FTP and
| telnet + HTTP.
| chasil wrote:
| POP is a little simpler, but IMAP is designed for concurrent
| access by multiple clients to the same account.
|
| What this article doesn't address is OAUTH, which is required
| more often now.
| kazinator wrote:
| That's a hugely important use case, especially nowadays.
| People have mail clients running at the same time from
| multiple devices; e.g. desktop and mobile. Also multiple
| browser tabs with an instance of their webmail.
| steeeeeve wrote:
| This is the kind of thing I would have expected to read in 2600
| back in the day. And why I _always_ looked for 2600 at the
| bookstore.
___________________________________________________________________
(page generated 2025-03-31 23:01 UTC)