I've had a few run-ins with mail retrieval systems like fetchmail, and have
finally decided that perhaps it may be worth it to investigate changing the
system. It may be possible to modify fetchmail (or even getmail) to suit my
needs/requirements, it may be not. In the meantime this is all just concept
and nothing concrete is yet in place.
Most people that read this will know that bandwidth in South Africa is rather
expensive. Those that live here will know what I refer to when I mention "the
CAP". It's something that the telecomunications providers here must enforce
due to the fact that the monopolistic provider apparantly doesn't have the
required infrastructure to properly handle it if eveybody just used their
bandwidth as required. Bottom line is: Bandwidth is precious, and in some
Now, it sometimes happens that there are bugs in software (the particular
case that sparked this round of thinking was clamav), in a mail system this can
result in temporary failures (4xx rejects). So what happened was that fetchmail
would download a 4MB email, pass it to exim, exim would scan it using clamav,
clamav would fail with "Zip module failure", exim would issue a 4xx temporary
fetchmail would discard the message and repeat the whole process 5 minutes
later. Keep this going for 5 days and you end up using quite a lot of
bandwidth, not to mention keeping a pipe clogged up by re-downloading the same
data over and over and over again.
I've been having a bit of a discussion with some people on the glug-chat
mailing list (GLUG == Gauteng Linux User Group). And a few ideas have been
throws around, including the fact there are many other possible bandwidth
savings in the whole fetchmail process. I'll attempt to discuss some of these
in detail, others are so obvious that I really couldn't particularly care expanding
Firstly there are a few things that I would absolutely require should
a system be written from scratch, and would really like to have seen in
Various aspects of the system can (and should) be plugable, for example some
abstract mail retriever that can be implemented by various implementations to
provide different mechanisms for retrieving email. Two such implementations
would typically be POP3 and IMAP.
- plain, SSL and STARTTLS:
These are possibly POP3/IMAP specific, but these at least should be able to use
plaintext connections (like that provided by just about all ISPs - google being
the only exception afaik), straight SSL off the bat, also refered to as pop3s
and imaps (they are to pop and imap as https is to http). A newer mechanism
that is also seen a lot now is the starttls mechianism where the connection
starts in plaintext and then gets "upgraded" to TLS/SSL at a later stage.
- POP3 + IMAP:
I personally don't have a need atm for ODMR, ETRN or UUCP and would thus
currently only require implementation of POP3 and IMAP.
- Delivery methods:
I would like to see different delivery methods, including direct delivery
into Maildir, mbox, IMAP, SMTP and command delivery, for example invoking
/usr/sbin/sendmail, procmail or some other command. The only one I really
require atm is SMTP, but these methods should be plugable in any case.
- Headers first, body later:
As far as I can tell fetchmail downloads the entire message (including headers)
before initiating communications with SMTP (although it keeps the SMTP
connection open once it opened it). This means that even if a 5MB message gets
rejected before we even get to the data phase of the smtp connection we still
downloaded the full 5MB whilst we could have gotten away with only the headers
of the email. Both POP3 and IMAP allows for only retrieving the headers of an
- single-drop and multi-drop support:
Some ISPs provides us with a catch-all mailbox for a domain (including all the
collateral disadvantages that comes with this). This is also known as a
multi-drop. Same as fetchmail this should be supported in any mail fetching
- multiple configuration backends:
fetchmail requires it's complete configuration to be passed in a fetchmailrc
file, or on the commandline. This is rather annoying and personally I'd like
to be able to have different modules that specifies where to download email
from, in my case I'd be particularly interested in getting this configuration
from a MySQL database. Another almost obvious option would be ldap.
- multithreaded downloads:
Downloading mail in a single stream is potentially inefficient, and is
definitely unfriendly to other users. This is particularly true in the
environments in which I'm working mostly, where a single server retrieves mail
from an ISP for an entire domain and re-injects into a local mail system.
Consider the case where one user receives a 10MB email from an internation POP3
server after your cap has been reached. This single 10MB email can easily take
an hour to download in such conditions (it's fscked I know, not much I can do
about it). It would be better if the system could fire up multiple mail
retrieval sessions in parallel so that at least a few large emails would be
required to block up the entire process.
- configurable domain/host names:
fetchmail does not allow for specifying what the From: <> header should
contain in cases of bounces. It just assumes that is can use the fqdn of the
host. This however is simply not true. Take my setup for example, my local
server is called xacatecas.lan, if it tries to send a bounce using return path
<> (in according with the spec) and the From: header containing "Mailder
Daemon" <email@example.com> then that bounce is going to get
bounced in all probability simply because xacatecas.lan isn't resolvable
publicly on the internet. Instead it should be using firstname.lastname@example.org
as the From: address for bounces (I've had to work around this braindeadness
using exim rewrite rules).
The above doesn't address the biggest issue I've experienced, that of
temporary mail delivery failures after transmitting the body of the
message, in this case fetchmail just discards the message and re-downloads
again at a later stage, as explained earlier. It would be much better if the
system had it's own little queue where it could temporarily store emails in
order to retry delivery at later stage without having to re-download the
message again at a later stage. This "queue" could simply be treated as a type
of multi-drop and injection into this queue could be treated as a type of
delivery, keeping to the abstractions of retrieval and delivery. Any message that
is stuck in this store for an extended period of time should be bounced.
Alternative logging schemes. Personally I'd like to be able to report
maildrop specific errors back via some kind of user-accessible web interface.
Eg, if retrieval from mail.kroon.co.za fails when trying to download mail for
email@example.com and the error is "Authentication failed" then I'd like to
store that in my database so that I can give that feedback to the user.
Currently I need to jump a hundred and one hoops in order to achieve this,
including some pretty insane parsing of the output generated to stderr.