[HN Gopher] UUCP must stay; Fetchmail sucks (2001)
___________________________________________________________________
UUCP must stay; Fetchmail sucks (2001)
Author : segfaultbuserr
Score : 94 points
Date : 2021-01-07 14:18 UTC (8 hours ago)
(HTM) web link (docs.freebsd.org)
(TXT) w3m dump (docs.freebsd.org)
| mapgrep wrote:
| I have personally found that as you get serious about keeping
| your own archive of email you need something like fetchmail, but
| not fetchmail.
|
| I used getmail for years and it is solid. duckerude posted this
| elsewhere in the thread, where the author of getmail rants a bit
| about fetchmail. http://pyropus.ca/software/getmail/faq.html#faq-
| about-why
|
| Once I left Gmail I was interested in something faster and more
| efficient, because I was using the transfer tool for more than
| just backup, it became the main way I receive email on my
| machine. Then the mail gets indexed (for text search) by another
| program and displayed by a third.
|
| Anyway I found mbsync, which is just this amazing, efficient,
| very solid transfer program that's been around for many years
| just quietly doing its job. I almost wrote it off because its
| homepage is on sourceforge but if you poke around a bit you'll
| see it is actively developed, there's an active mailing list,
| etc. I love it. It's kind of interesting how this project I had
| never heard of just does its job well and works and the program
| by the semi-famous software pundit whose programming essay went
| viral really does not (from what I have heard).
| throwaway201103 wrote:
| I've used fetchmail for years to get email from IMAP inboxes
| (mainly Exchange server) and never had a single problem with
| it. Was surprised to see that there are major complaints.
| hpfr wrote:
| I just got started with mbsync and am having several issues
| with Gmail and Outlook.com. These are old accounts and I do
| have well-behaved IMAP emails from smaller hosts, but it would
| be nice to manage everything with mbsync. Trashing emails in
| mu4e and then syncing results in them being deleted from my
| local [Gmail]/Trash while they somehow remain in the remote. I
| thought Outlook.com would be fine considering they use more
| standard folders rather than the tagging system which seems to
| result in duplicates, but I have the same issues there. This is
| with Expunge None, which I expect would leave emails alone
| until the server empties them from the Trash.
|
| The bizarre thing is that no one online seems to have these
| issues? The mbsync configs for Gmail just seem to work for
| people, or something. I do have auto-expunge turned off in
| Gmail, so I'm kind of at my wit's end with it. It's a shame
| because it is an amazing program. It synced 10,000 emails
| without a hitch. I only started getting issues when I tried to
| sync from my maildir to the IMAP store.
| fiddlerwoaroof wrote:
| I've only run mbsync unidirectionally as a backup utility (in
| addition to tarsnap on my Mailserver). I wonder if the
| bidirectional features just aren't used that much?
|
| Also, Gmail's tags don't map nicely onto folders.
| anthk wrote:
| I use a fdm + s-nail combo. I know I can use s-nail over imap,
| but this way I am sure I always have my mail available.
| 0xbadcafebee wrote:
| Can confirm that fetchmail is garbage. Found bugs, reported them,
| no fix. Wrote my own tool instead. Such is life in open source.
| KirinDave wrote:
| I wonder if many folks who only joined the world of software
| development after DVCSs (and indeed, just VCSs) became
| popularized can understand why Fetchmail was not trivial to
| fork and fix.
| aatharuv wrote:
| The only time I ever interacted with ESR on fetchmail, I
| reported a bug for an issue I ran into (with Kerberos 4 IMAP),
| submitted a patch, and it was accepted without discussion in
| the next release.
|
| Admittedly, this was 19 years ago, and it was a relatively
| small fix.
| dvfjsdhgfv wrote:
| This doesn't seem so much about UUCP vs fetchmail as about
| cathedral vs bazaar style of development. In hindsight, it's
| clear that both of these work, for better or worse, with each
| having its own disadvantages and benefits.
| mprovost wrote:
| Some context for those who are unaware: fetchmail was written
| by the same person (Eric S Raymond/ESR) as the "Cathedral and
| the Bazaar" essay.
| freedomben wrote:
| If you have any interest in open source development (which
| these days we _all_ should given that 95% of the lower parts
| of our stack are open source projects) it 's well worth
| reading the essay/book.
|
| If you don't want to do that, at least read the wikipedia
| page:
| https://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar
| segfaultbuserr wrote:
| I found this interesting post while reading some historical
| documents, so I submitted it to HN. It seems that most
| people have missed the point. I think the point is: while
| _The Cathedral and the Bazaar_ was a good campaign pamphlet
| and a general overview of FOSS development, but ESR 's
| attempt of using his fetchmail as an example was arguably
| the greatest flaw of the book ("to such an extent that it
| could damage the Open Source movement"). From a technical
| perspective, fetchmail had numerous problems and security
| vulnerabilities. ESR was also criticized for mismanaging
| the project by putting it in "maintenance mode" early and
| refusing to accept patches that could address those issues.
| After a group of new maintainers had taken over the
| project, one of the first things they did was to
| systematically eliminate countless buffer and integer
| overflows from the program.
| tptacek wrote:
| Not really. I read it when it came out and remembered
| liking it (we were all on kind of a buzz about open source
| and Linux at the time) and then again much more recently
| for a thread on HN, and I found it to be mostly a rehash of
| other, better material --- Brooks and Kernighan, in
| particular. That, plus the modern context that what he was
| writing about was Fetchmail, and Fetchmail as it turns out
| is hot garbage, to me makes the whole thing very skippable.
|
| It's also responsible for hatching one of the great canards
| of open source, "Linus's Law" (I'm sure Linus is thrilled
| with having his name attached to that), "many eyes make all
| bugs shallow", which has in the last 10 years become one of
| the great jokes of computer science.
| dvfjsdhgfv wrote:
| I remember reading his essays (there were several more)
| and thinking, "He is seems so sure of everything, let's
| see how it plays out".
|
| He wasn't completely wrong about everything, but the main
| premise - that careful planning and development yields
| worse results than guerilla ("bazaar") way of working -
| didn't pass the test of time. His main argument was the
| Linux kernel. But when you look at this project now, it
| seems quite far from the bazaar style. There are many
| bazaar-style projects that failed and many cathedral-
| style projects that do very well. So his main argument is
| moot.
|
| We can treat this essay as a historical document, not
| something anyone should refer to as to how the open
| source development should be done (even though he has
| some good points about leadership style for example).
| segfaultbuserr wrote:
| > _We can treat this essay as a historical document_
|
| It's why I'm rereading this book today.
|
| One thing I've been thinking about since morning was
| ESR's "Linus Law", "given enough eyeballs, all bugs are
| shallow". Today the idea is heavily criticized. The very
| existence of fetchmail by the author was a counterpoint -
| the code quality was below average, it had numerous
| security vulnerabilities, and arguably basing the whole
| book on fetchmail is the book's greatest flaw (not to say
| the criticisms of ESR's personal qualification from many
| hackers).
|
| But if you read the original context...
|
| > Linus was directly aiming to maximize the number of
| person-hours thrown at debugging and development, even at
| the possible cost of instability in the code and user-
| base burnout if any serious bug proved intractable. Linus
| was behaving as though he believed something like this:
| Given a large enough beta-tester and co-developer base,
| almost every problem will be characterized quickly and
| the fix obvious to someone.
|
| The original context was strictly about beta-testing -
| about how the Linux kernel didn't collapse on its own
| weight despite the lack of a BSD-style "core" team and
| heavily scrutinized release. Instead, it achieved system
| stability with the help from a large number of beta-
| testers and co-developers who are willing to run on the
| bleeding edge. It's still how the kernel and the majority
| of the FOSS world. "Build today's git snapshot and see
| whether it explodes" can be quite effective.
|
| Then ESR simplified it to...
|
| > Or, less formally, "Given enough eyeballs, all bugs are
| shallow."
|
| And now it means "people can find all bugs, because there
| are always a lot of eyeballs to read the source code".
| Now it can refer to all sorts of things beyond its scope.
| It is a serious mistake. Notably, security
| vulnerabilities are different - it usually does not have
| any user-visible effects and can only be triggered by
| carefully crafted malicious inputs, not normal uses. This
| is the reason it doesn't work for security. Thus, maybe a
| more accurate reflection of this rule could be: Given
| enough eyeballs, all user-visible bugs (e.g. crashes) are
| shallow.
|
| At the end of the day, my conclusion was: Perhaps "Linus
| Law" was not as wrong as it seemed. ESR simply
| misinterpreted his observations.
| tptacek wrote:
| The "law" isn't that "all crashers are shallow". That
| would be a silly law, because you don't need many eyes at
| all to notice crashers.
| db48x wrote:
| Actually, you do, especially in the kernel. No one person
| could ever replicate all the crazy combinations of
| hardware, software, and usage that people need.
| indymike wrote:
| Think this story is a developer who submitted a patch to they
| guy who wrote the manifesto for open source. The patch was
| rejected for non-technical reasons, which led to the developer
| pointing out the irony of the situation. It's actually funny,
| and if you ever used fetchmail, you'd know just how accurate
| Terry Lambert's assessment of fetchmail ("abomination before
| God") was...
| _jal wrote:
| I'm glad someone else actually got it. Didn't think this was
| all that complicated, but context is everything, I guess.
| kazinator wrote:
| Fetchmail was only supposed to be something like "offline IMAP"
| for someone with only occasional network connectivity, and whose
| mail program is too braindead to download and cache messages for
| offline reading when connected.
|
| Of course, fetchmail wouldn't have the envelope info, because
| that occurred in an SMTP chat that is not part of the message
| landing in your inbox.
|
| Something that accesses delivered mails in IMAP or POP3 boxes
| cannot be a full-blown MTA. E.e. we can't program a mail
| exchanger machine to accept the mails on behalf of hundreds of
| users, by delivering them into a giant mailbox, and then
| continuing the delivery path by sucking them out with a
| fetchmail-like tool, and forwarding to more forwarding hosts.
|
| The whole complaint of this irate BSD person is that you can't do
| this kind of nonsensical thing.
|
| There is a sort of valid criticism in that fetchmail (or at least
| did originally, I don't know about now) worked by re-sending the
| messages to the user over a mail transport mechanism, rather than
| delivering it into a mail directory structure. E-mail clients
| that download and cache messages certainly do not do anything of
| that sort. However, that shields fetchmail from having to know
| how the user's mail software likes to have the messages stored.
| Arguably, sending the mail is the right interface.
|
| Though the envelope info isn't available, it's assumed that it
| was already processed by the remote mail server. The remote
| server did all he spam checks on the SMTP info, and whatnot,
| decided that the message is deliverable to the user, so there it
| is.
|
| Fetchmail assumes that if the mail is in the remote box, then
| it's yours, and can be effectively forwarded. The envelope sender
| is Fetchmail itself.
|
| If that is broken, so are mailing lists. When you receive a
| message from someone via a mailing list, you also don't have the
| envelope which the mailing list robot's server received. The
| envelope is that of the mailing list's own delivery; the mailing
| list robot is the MAIL FROM: sender.
|
| For that matter, you are not even the original intended
| recipient! The mail was originally to the list, and now it is to
| you.
|
| Not having an envelope isn't a show-stopping impediment to
| further forwarding and delivery. You just have to understand and
| accept that it's more of a forwarding style of delivery, like a
| mailing list robot.
|
| Overall, it's almost like complaining that a dog which fetches
| the mail from your door and brings it to the kitchen is not a
| trained postal worker, does not understand and handle addresses
| right, and could not be reliably used to service an apartment
| building with six tenants, all by itself, such that the mailman
| could just put everything in one bin and leave it to the dog.
| duckerude wrote:
| A similar rant: http://pyropus.ca/software/getmail/faq.html#faq-
| about-why
| thwarted wrote:
| Fetchmail grew to be a mail transport swiss army knife and
| encompassed half baked functionality of a domain-wide MTA rather
| focusing on single user async email consolidation and collection
| for a logical single account, which is largely what this rant is
| about. Half of the half-bakedness is due to lack of MTA protocol
| support for MUA interactions on the receiving side.
|
| pop3 was never meant as a spool for MTA to MTA email transfers,
| but was instead meant for an MUA to access. It's not fetchmail's
| fault that the envelope TO isn't available via pop3. That, at the
| time, email service providers bolted on entire domain delivery to
| a single mailbox, and then users of those services put multiple
| recipients behind that single mailbox. Fetchmail made this work
| as best it could. Also missing at the time was any kind of
| serious connecting-client-as-MTA or MTA-to-MTA authentication
| (rather, relying on DNS and TCP!), which might have made async,
| receiver controlled MTA initiation more workable.
|
| This was also due to common SMTP MTAs not having good (or at
| least easy/obvious to configure) store-for-later-pickup
| functionality where an MTA could query for email destined for
| itself rather than be always-on connected to the internet waiting
| for another MTA to initiate delivery. Fetchmail filled a hole
| that existed in email delivery at the time, but it was really a
| hole that should have never existed. The quality of the fetchmail
| code is another, albeit serious, issue in itself.
|
| These days we have a better understanding of the differences
| between MTAs, MSAs, and MUAs.
| znpy wrote:
| Would anyone that's well-versed in UUCP, Fetchmail and related
| mail protocols and technologies care to explain whether this
| discussion from 2001 (20 years ago!) is still relevant today?
| jstimpfle wrote:
| > When mail is delivered to a POP3 maildrop, envelope information
| is destroyed. To combat this, you would need to tunnel the
| enveleope information in headers.
|
| Isn't that exactly what Envelope-to is supposed to be? Protocol-
| level data? It's not really part of the e-mail, but added when
| sending it. If you forward the mail to another machine, there
| will be a new envelope.
|
| I can see that it's all a gray-ish area with email (because email
| is a mess (which might be part of why it's been successful)) but
| I can't see a reason to complain about losing Envelope-to.
| tyingq wrote:
| More context:
| http://pld.cs.luc.edu/courses/412/spr20/mnotes/bazaar.html
| rasengan wrote:
| I used to use UUCP to get my email when I was on a dialup, non
| permanent connection. I was able to host my own mail server.
| There were a lot of technologies back then helping the internet
| stay somewhat decentralized. Now I, like most, just use Gmail.
|
| What has become of our beloved free(digital)land?
| KirinDave wrote:
| Well, part of this is on us. Let's think on the reasons Gmail
| is so popular:
|
| 1. It's very easy to get to.
|
| 2. It has incredibly fast search that has 0 setup.
|
| We have never really even tried to address problem 1 as an open
| source community. Networks, name lookup, and VPNs remain
| incredibly complex topics that beginners cannot hope to wrestle
| with. The best we have is .mdns which either works magically or
| perversely refuses to work.
|
| Similarly for free text search, the software world simply
| hasn't delivered a lego-like solution for email search. You CAN
| rig up any number of open source projects but it is neither
| easy nor instant. And even other professional products like
| Apple Mail struggle with a mere gigabyte of email.
|
| Despite the fact that it's 2021 and every successful email
| provider aggressively solves these problems, the open source
| world still debates about the utility of ubiquitous search or
| pretends that local networking isn't a pressing problem.
| creeble wrote:
| Agree 100% on both of these, and #2 is seldom listed as a big
| issue. Had IMAP addressed search in a better way, I think it
| would have made a huge difference.
|
| Of course, it's not simple to set up a mail server that stays
| clean of spam RBLs, or that is Gmail-acceptable out the gate.
| But that's just the bar that got set as people went to Gmail
| because of the lack of other good alternatives.
|
| Question is, is Gmail hegemony fixable now?
| thayne wrote:
| I think the best approach to breaking the Gmail hegemony is
| to replace email with something better. Some features I
| think such a system should have are:
|
| 1. End to end encryption 2. search as a first-class feature
| 3. decentralized/federated 4. standard minimal rich text
| format (maybe something like markdown), instead of an
| inconsistent subset of html 5. Fix some of the legacy
| limitations of email, like having to be ascii safe, line
| limit of 70 characters etc. 6. Possibly make it easier to
| have an identity that isn't tied to your service provider
|
| The big questions are how would you get the general
| population to switch from email, and what to do about spam.
| secabeen wrote:
| Yeah, none of the IMAP server authors really thought of
| search as a first-class feature, nor did they seem to want
| to do the development to build the backend support.
| Resource constraints were also an issue back then, as well
| as a client-first approach. Most large mail servers only
| had enough resources to run the IMAP and POP services
| themselves; adding search index maintenance would have
| overloaded their CPUs. (Not that they couldn't have added
| more CPU, but this was all pre-cloud.)
| Avamander wrote:
| > Question is, is Gmail hegemony fixable now?
|
| Maybe, if we get maddy working near-perfectly and kept up-
| to-date.
| varjag wrote:
| I use offlineimap/mu4e over Exchange account at work, in
| parallel to a personal Gmail account. Would not say now that
| Gmail has a perceptible edge there.
|
| Seriously, mu4e search on modern hw is incredibly fast (and I
| have a 10 year work email archive).
| throwaway201103 wrote:
| For the search problem, notmuch is pretty easy and has better
| search capabilities than gmail.
|
| https://notmuchmail.org/
| teloli wrote:
| I do agree with you, but let's add:
|
| 3. dealing with spam is a massive PITA
| Avamander wrote:
| Most of spam problems could be solved if mail software came
| with sane defaults. Sane rate-limits, sane IP reputation,
| sane security rules, some backscatter protection.
| Something. Most setups are so widely open because good
| configuration is arcane knowledge.
| thayne wrote:
| 3. A whole lot of free storage, so most people don't have to
| worry about needing to delete old emails.
| jerf wrote:
| "We have never really even tried to address problem 1 as an
| open source community. Networks, name lookup, and VPNs remain
| incredibly complex topics that beginners cannot hope to
| wrestle with."
|
| I kinda disagree. It's probably easier than ever to set up
| your own mail server, in some abstract sense. You can get a
| virtual machine, use docker, heck, someone can hand you a
| complete image that you just have to bring up and set up with
| some config.
|
| The problem is, it literally doesn't matter how much the
| 'open source' community comes together, it simply _can not_
| provide a turn key solution as good as
| Desired email account: [________]@gmail.com
| Password: [________] Verify Password:
| [________] [X] I agree to have all my data used in
| arbitrary ways
|
| It's not possible. There is no way to set up a server that
| easily, even in principle.
|
| Or at least, not in a sane way. I can set up a site where you
| feed me your credit card number and pick a domain name, and I
| set up your AWS account for you, register your DNS name for
| you, configure DNS, and stand up everything you need and set
| it all up... but then we've got a split ownership interest. I
| can hand it all back to you, but you don't understand the
| setup. I can give you root on the system, but when you change
| anything, my automation stops working.
| ryukafalz wrote:
| >It's not possible. There is no way to set up a server that
| easily, even in principle.
|
| I partially agree, but I think we could get a lot closer
| than we are now. It feels like the main reason this isn't
| possible is because you need to go through a registrar to
| get a DNS name, and that's tricky to do as part of a FOSS
| project. Maybe you could integrate with the APIs of a few
| registrars, but... it's not ideal.
|
| As far as the "run thing on server" side of it goes,
| though, projects like Sandstorm[0] have gotten really far
| re: making it a simple process. I stood up this instance of
| Etherpad with a few clicks on a web UI, for example: https:
| //sandstorm.terracrypt.net/shared/aR2HXaoLSkLuXLhhAQon...
|
| Sandstorm in particular doesn't quite work for mail servers
| just because the software is heavily oriented towards
| webapps, but there's no reason a similar system couldn't
| work in principle.
|
| [0] https://sandstorm.io/
| Avamander wrote:
| > It feels like the main reason this isn't possible is
| because you need to go through a registrar to get a DNS
| name
|
| I can't agree, getting a domain is much easier than it is
| to configure the abysmal shit that is
| Postfix+Dovecot+WebGUI. God forbid you want proper search
| as well.
| sumtechguy wrote:
| Another issue that needs to be solved is 'bad actor'.
| Basically someone gaming the system and overloading it with
| spam. There is no real nice neat way to fix that. Instead it
| is a mishmash of blacklists/whitelists/blocklists and sorta
| intelligent alg filtering. Getting that all filled and
| working is not trivial either. Oh its 'doable' but kind of a
| pain for even someone with decent experience at it.
| ywei3410 wrote:
| To add to that; with re-assigned IP's, widespread port 25
| blocking /most/ ISP connections won't allow you to run an
| email server on a residential connection.
|
| Spam/botnets have a lot to answer for.
| theamk wrote:
| Spammers and abuse.
|
| I used to run personal email server some years ago. The spam
| was harder and harder to manage - first local SpamAssasin was
| fine, the I had to tweak rules, then I tried distributed
| filtering (DCC)... and then I gave up and started paying for
| Fastmail. This ended up working much better than I could ever
| do.
|
| Usenet was similar -- I remember looking at the newgroups which
| were at more than half spam (and some were 95%+ spam) and then
| just leaving for forums. (plus there were a major UX annoyance:
| unlike with email, there is no simple way to sync read status
| across multiple devices)
|
| Our free(digital) land has been killed by a handful of bad
| actors. I no longer believe in "free for all" systems. The best
| we can do is many independent systems, each with its own rules
| and policies.
| Mediterraneo10 wrote:
| I run my own mail server with only Spamassassin for
| filtering, and I get little spam to my inbox in spite of my
| e-mail address being posted in plaintext across the internet.
| It has sufficed to 1) simply activate distributed filtering
| within Spamassassin, and 2) set a _slightly_ more stringent
| standard than the default for regarding a message as ham,
| since any legitimately sent message will easily meet that
| higher standard. I also fed my entire e-mail archive to sa-
| learn, and what spam I occasionally get I also feed to sa-
| learn.
|
| In ten years of running this mail server, I have only had one
| persistent spammer I couldn't manage to get filtered with
| these settings, but then I just created a custom rule.
| layer8 wrote:
| Greylisting and DNSRBL always worked sufficiently well for
| me, in addition to Spamassasin.
| throwaway201103 wrote:
| I think worst than the spam problem is the deliverability
| problem. Worst case I can hit the delete key on spam. (as an
| aside, most of it is immediately obvious from the subject
| line alone, which makes it hard to believe that it's such a
| hard problem). But getting mail delivered from a private
| server to major email providers like gmail, yahoo, outlook,
| is difficult.
| icedchai wrote:
| Me too! I ran AmigaUUCP on my Amiga, dialed up at 9600 baud to
| my upstream. This was around 1992 or so. I had Internet email
| and a Usenet feed of maybe 15 groups or so. Fun times.
| cmrdporcupine wrote:
| Yep, I remember spooling usenet and email to my Atari ST over a
| 1200 baud modem over UUCP in 1990, 1991. Bang path and all.
|
| And you're right, the centralization is crazy, and opposite of
| what techno utopians predicted/promised at the time.
| toyg wrote:
| Sadly, the upsides are just too good. I love
| decentralization, but I have to recognise that I've not lost
| a message since I moved to gmail in 2005; before then, losing
| _entire mailbox files_ was not uncommon, as well as missing
| messages from unreliable mailservers, spam management, and so
| on. I 'm sure I could have done better etc etc, but the
| bottom line is that with gmail-like services you don't have
| to. I enjoy doing sysadmin but not 24/7.
| cmrdporcupine wrote:
| Oh hell yes, I basically don't delete email ever and as a
| result I have an archive of every email (pointless or not)
| since 2005. (Though I just checked and my first email on
| gmail is porn spam). Meanwhile everything I had from every
| account prior to that, dating back to 1991 or so, is lost
| forever. Apart from some very embarassing forever-archived
| usenet posts.
| vidarh wrote:
| My first access to the web around '93 was via CERNs e-mail to
| web gateway coupled with UUCP exchange from my Amiga to my
| local BBS which exchanged with its upstream 4-6 times a day.
| cmrdporcupine wrote:
| Yeah I used to dial into the local university's gopher
| "campus information server" -- which was just a dial-up to
| telnet to a gopher client -- hit the ^] telnet escape
| character, close the connection, and re-open to another
| host where I had a shell and go from there. If I recall
| right it was probably gnu.ai.mit.edu that gave out free
| shell accounts, or something else in that domain connected
| somehow to Stallman.
|
| There was no ISP in my area back then, not even dial-up.
| Again, around 1991.
___________________________________________________________________
(page generated 2021-01-07 23:00 UTC)