Sourcing for a new email service

Couple of years ago I wrote about the initial troubles with Fusemail service not long after having switched to it. The grade for the service level during last couple of years is “fair” – the service is usable, and has many good features, but its stability is not stellar. For this reason it is in my plans to move the email service out of Fusemail.. but where to?

There are several hosted email solutions available – such as Rackspace Webmail or BlueTie Business Email. From time to time I’ve reviewed Rackspace’s features, and among other things they still did not support user-by-user adjustment of spam filter aggressiveness, or the ability to alias users and domains to actual accounts. Especially for the latter reason using Rackspace would be prohibitively expensive for me, so the search had to continue.

I think I have now found a workable solution: an in-house Postfix+Dovecot server with Google Postini as the edge (MX) service. Postfix is an incredibly robust and infinitely configurable mail server that I have only had positive experiences with. But I would rather not have to deal with spam filtering, and I also would rather not have to stress about “high availability” of the in-house server. Yes, the server will generally be up and receiving email, but should there be a hardware failure, I would want the inbound mail to be both cached and accessible while the server is repaired. Postini Message Continuity system provides this at $25/year/user (including spam/virus filtering), which roughly equals the annual cost of Fusemail service. If the receiving email server becomes unavailable, the arrived messages are cached, and are accessible via Gmail. Once the in-house service is then restored, the cached emails are quickly delivered.

However, one issue remained: in my configuration several of the aliased domains have ‘forward’ addresses that don’t have an associated user account at Fusemail, but that Fusemail has allowed to be forwarded to external destinations without an extra fee (or a separate account). I would not want to have to set up individual accounts at $25/year each to take care of these external forwards. But there is a way! Information about the aliased users and domains is retained in the messages as they’re forwarded by Postini. So by setting up one generic – or a “forwarder” – account at Postini, and then aliasing any such external forward accounts to it, it is possible to separate and correctly forward the emails to their intended recipients using Postfix’s rewriting rules.

Note that the in-house email server will need to be set to receive unauthenticated emails only from Postini’s IP range; that way spammers can’t bypass Postini filtering should they discover your SMTP server via port scan (and they will discover it, sooner or later). Also note that while Postini Message Continuity homepage mentions Exchange Server as the in-house mail server, they also provide instructions for how to integrate Postfix (more info can be found on the web).

While this solution does require familiarity with Postfix/Dovecot, it should offer an extremely stable email setup – and maximum amount of control over how email is routed – at a very reasonable cost.

When I get around to implementing this configuration I’ll write an article or series of articles of the process. Meanwhile I welcome any feedback, as well as suggestions for other comparable services. If a stable/high quality, reasonably priced, outsourced solution with good amount of control (equal or better than Fusemail) exists, I would consider it. But for now it seems that currently the only/best solution that won’t miss a beat while not breaking the bank is the one I’ve outlined above.

Caching inbound email on LAN with Postfix (and restricting reception of external mail only to the external mail provider)

Externalizing email reception often offers many benefits: firstly, it’s [more] worry-free than servicing email internally, especially in smaller organizations where there may not be an email administrator on-call 24/7. Or, think of a situation where there is an “IT guy” who manages internal email. Then he goes on a vacation and email goes down. Now what? And even when the IT guy is present, the budget may not allow for good redundancy for email reception. What if the email server melts down. Perhaps there is a backup plan but without a stand-by server and/or perhaps virtualization option getting mail reception back online could take a day, which potentially would be a big hindrance to business.

However, outsourced email is not without its pitfalls, too. Even with a reasonably fast network connection there is more noticeable latency when accessing a remote email server as opposed to a LAN-based solution. Then there is the issue of outsourced service quality vs. the cost. Some services like Fusemail or Rackspace offer reasonable quality and fairly customizable features. But when something does go wrong you’re dependent on their response time. You’ve essentially handed away control of your email reception, for better or for worse.

Mostly, however, reception uptime is good with most well-run outsourced mail services, and the issues that more commonly crop up are related to latency and in some cases (like with Fusemail from time to time) apparent capacity issues. And if you access Fusemail with Outlook 2010’s IMAP client, you may have noticed frequent spontaneously changing message ID’s which repeatedly pop up a notification on Outlook.

The client-side issues are easy to remedy by caching inbound email on your local server. It gives you the best of both worlds: quick access to email and safety of someone monitoring mail reception 24/7 with multiple redundancies. If your local caching mail server goes down even for an extended amount of time, all you need to do is to repoint your clients to the external provider’s IMAP or POP server and you’re back in business. You may also opt to use your own outbound SMTP service (assuming you have a static IP in use) which makes it possible to isolate your domains’ SPF records to the IPs you own (as opposed to allowing anyone with an account, for example, at Fusemail to spoof mail from your domains without an SPF penalty). And if you use Fusemail, your own SMTP server will give you a peace of mind so that your outbound emails won’t trigger suspension of your account like happened to me soon after I first signed up with them (see Fusemail auto-suspends spam-suspect accounts!). Perhaps they’ve fixed that issue since then.

Setting up a caching mail service on your LAN is fairly easy with Postfix. The following tutorial assumes you already have a functioning Postfix/Dovecot setup where you’re able to send and receive email based on your requirements.

To start with, configure and test local users that you would like to correspond to outsourced email service’s mailboxes. They do not need to have the same login name, and you can also consolidate multiple external accounts to one local account. In smaller setups it’s the easiest to simply create a flat-file for user Dovecot passwords lookups.

Assuming you like to receive all email through an outsourced service (which, if you use an outsourced service, is the preferred option), you will want to restrict mail reception from the outside world only to the sending mail servers of the external mail provider of your choice. To accomplish this some restrictions are added to the local cache server’s main.cf file. The following is the configuration I use; I’ve carefully given thought of the restrictions not being too constrictive as to unnecessarily prevent connections, but on the other hand cut off connections that would not result in a successful or desired mail transit.

smtpd_helo_restrictions =
        permit_mynetworks
        reject_invalid_helo_hostname
        permit_sasl_authenticated
        reject_non_fqdn_helo_hostname

smtpd_client_restrictions =
        permit_mynetworks
        permit_sasl_authenticated
        check_client_access hash:$config_directory/tables/smtpd_client_access
        check_client_access cidr:$config_directory/tables/smtpd_client_access.cidr
        reject

smtpd_etrn_restrictions =
        permit_mynetworks
        reject

smtpd_sender_restrictions =
        reject_non_fqdn_sender
        reject_unknown_sender_domain

smtpd_recipient_restrictions =
        reject_non_fqdn_recipient
        reject_unknown_recipient_domain
        permit_mynetworks
        reject_unlisted_recipient
        permit_sasl_authenticated
        check_recipient_access hash:$config_directory/tables/smtpd_recipient_access
        #the following also permits mynetworks!
        check_recipient_access pcre:$config_directory/tables/smtpd_recipient_access.pcre
        reject_unauth_destination

smtpd_data_restrictions =
        reject_multi_recipient_bounce
        reject_unauth_pipelining

You will notice external hash and PCRE lookup tables “smtpd_client_access”, “smtpd_client_access.cidr”, “smtpd_recipient_access”, and “smtpd_recipient_access.pcre”. Let’s look at them next.

smtpd_client_access (hash) and smtpd_client_access.cidr (example below) list the external IP addresses you allow to connect and hence relay mail to your cache server. If the external IPs are not on this list, the connection is terminated.

Here’s an example smtpd_client_access (hash, so it’s converted to smtpd_client_access.db with postmap):

# some individual external server I want to allow to connect
100.200.100.200 PERMIT

And here’s an example smtpd_client_access.cidr:

1.2.3.4/24      OK
10.20.30.40      OK
100.200.201.0/21      OK

While the sending servers of an outsourced service don’t change often, they may change at any time without a warning. Thus maintaining the above list manually would be a frustrating task. To automate the process, you can cull this information from the outsourced mail service’s SPF records with a cron-scheduled shell script (note that paths relate to FreeBSD; if you run Linux, adjust them to the taste/requirements):

#!/bin/sh

ORIGINAL=/usr/local/etc/postfix/tables/smtpd_client_access.cidr
NEW=/tmp/postfix_clients.tmp

dig +short fusemail.net TXT | grep 'v=spf1' | egrep -o 'ip4:[0-9./]+' | sed 's/^ip4://' | sed 's/$/      OK/' > $NEW

ORIGINAL_CK=`cksum $ORIGINAL | awk '{print $1}'`
NEW_CK=`cksum $NEW | awk '{print $1}'`

if [ -s $NEW ] ; then
  if [ $ORIGINAL_CK != $NEW_CK ] ; then
    cp -f $NEW $ORIGINAL
    postfix reload > /dev/null 2>&1
  fi
fi

rm $NEW

exit 0

The above example is obviously for Fusemail, but you can modify it for other providers simply by replacing the provider domain name on the dig line.

In my configuration smtpd_recipient_access (hash) lists simply the nullroute that is often required – such as in php.ini mail configuration where you might put:

sendmail_path = /usr/sbin/sendmail -t -i -f nullroute@mydomain.com

So in smtpd_recipient_access (hash) I list:

nullroute@mydomain.com  PERMIT

Meanwhile, smtpd_recipient_access.pcre lists the users who’re allowed to receive mail externally, from the IPs you defined with smtpd_client_access/smtpd_client_access.pcre:

if !/^(nullroute|abuse|postmaster|user1|user2|user3)@mydomain.com$/
/^/ internal
endif

Again, the above is sufficient for small configurations, but if you have dozens or more users whose external email you’re caching you may be better off storing the client_access table on a database server.

Finally, as you notice the ‘internal’ keyword being added above to user addresses that are not matched by smtpd_client_access.pcre, you want to add the following in main.cf:

smtpd_restriction_classes = internal, public
internal = permit_mynetworks, reject
public = permit

With these in place you will now receive email only from your external mail provider (and perhaps some other authorized servers if you defined one with smtpd_client_access hash table). This is important because you’re likely using your outsourced mail provider’s spam filtering, and you don’t want spammers contacting your cache mail server directly.

With the local server configured (and hopefully sufficiently tested 🙂 ) you can then go ahead and create mail forwarding rules at the external provider of your choice. You would simply copy any arrived email to the corresponding email address at your local cache server. I have additionally created a rule at the external provider which prunes the mailbox after given number of time since the users will not go through and delete read email there.

You may want to allow authenticated client access to your local users so that they can access their email via IMAP or POP remotely, and perhaps over the web (like via locally installed Squirrelmail). It is also a consideration that the email at the external provider will not be synced back – if users delete email from their locally cached mailbox it will not be deleted from the mailbox at the external provider, so under normal circumstances your cached email server should be the only access point for mail. But in an event of the server melt-down it is a minor inconvenience that the external provider’s mailbox would have older emails (that perhaps were deleted locally) in it – at least the users can continue to access email while you’re getting the caching server back online!

** NOTE: I’m using the current GA/Stable version of Postfix (2.7.0). If you’re using an older version, double-check that the configuration options I propose above are available before using them! This is one reason for why I prefer to use FreeBSD for mail; the ports version of Postfix is kept well up-to-date while CentOS/RHEL Postfix package is — as you would expect from an Enterprise Linux — currently at 2.3.3. You could compile it yourself, I suppose, but I’m not up to the task since I’m not a full-time Postfix admin.