Qmail, Mail toaster

Latest revision as of 02:09, 1 January 2009

[edit] What is a mail toaster, and what are we doing here?

A "mail toaster" is a silly name for a very useful and commonly needed "applicance" - a single server that can handle everything you need to do with email: receive it, send it, weed out spam, allow webmail access, allow authenticated SMTP traffic so that laptop users and other "road warriors" can send email from any network they happen to be attached to, handle multiple mail domains, allow delegation of web-based administration of individual domains to individuals responsible for those domains (and their domain only), and even allow delegation of web-based individual mailbox maintenance to owners of the individual mailboxes.

When you're done following through this article, you'll have a machine which will do exactly that - and even a few things more, like allow users to set "auto-reply" messages, forward their mail temporarily to other addresses, you name it.

(If you have a Microsoft Active Directory setup for your users, you may want to look at using qmail-ldap instead. But that's an entirely different animal.)

We'll be using the following ports:

• apache2
• qmail-tls
• ucspi-tcp
• vpopmail
• vqadmin
• qmailadmin
• sqwebmail
• dovecot
• qsheff
• safecat
• clamav
• p5-Mail-SpamAssassin

[edit] Prerequisites

First of all, you're going to need a FreeBSD server - all of the applications we're going to use are free software and available for pretty much any *nix, but we're going to be covering installing them using the ports tree... so your mileage would vary considerably were you to be trying to follow this article along on a Linux or Solaris or what have you.

Second of all, you're going to need DNS information set up to point the mail services (MX records) for any domains that you want to handle mail for at this machine. You may or may not want the same server to act as nameserver (DNS server) and mailserver, but either way, setting up DNS is beyond the scope of this article... we're just going to set up the services themselves here, not set up the internet to point the mail at us to begin with.

Finally, before we get started, be sure to synchronize your ports tree to make sure you get the newest versions of all the ports you'll be installing. Mailserver components are critical, you don't want to wind up with something outdated that could potentially have had a vulnerability disclosed in it!

ph34r# cvsup /usr/share/examples/cvsup/ports-supfile

And now that we have the latest versions of all of our ports, let's get started!

[edit] Installing Apache2

This is probably the easiest part of the whole process. Note: if you've already got your own copy of Apache up and running and you know how to care for it and feed it, that's perfectly fine, and you can skip on ahead to the next section - we aren't really going to do a whole lot of magical things with the webserver, we just need it available so that we can deliver our webmail and web-based control interfaces. It's also fine if you're using Apache 1.x instead of Apache 2.x; just make your own little adjustments and follow right along.

ph34r# cd /usr/ports/www/apache2
ph34r# make install clean

When the port's done building, rehash to update your system's PATH cache, and start Apache:

ph34r# rehash
ph34r# apachectl start

Check to make sure it's running:

ph34r# ps ax | grep httpd
91237  ??  Ss     0:00.37 /usr/local/sbin/httpd -k start
91246  ??  S      0:00.00 /usr/local/sbin/httpd -k start
91247  ??  S      0:00.00 /usr/local/sbin/httpd -k start
91248  ??  S      0:00.00 /usr/local/sbin/httpd -k start
91249  ??  S      0:00.00 /usr/local/sbin/httpd -k start
91250  ??  S      0:00.00 /usr/local/sbin/httpd -k start

And we're good! Now, on to the parts that actually handle the mail...

[edit] Installing Qmail with SMTP Authentication

Now, it's time to get the basics installed - first things first, get Qmail installed with the SMTP_AUTH and the TLS patches in place. This will allow us not only to send and receive mail, but also to authenticate SMTP sessions with a username and password so that authorized "road warriors" can use this server to send their email no matter what network they are physically attached to, and to use TLS encryption if supported on the remote end (which might be one of those road warriors, or might be another domain's mailserver) to keep potential unfriendlies from easily sniffing potentially sensitive information out of our email.

Note that we're NOT issuing make install clean as a single command here, because there are other steps we want to take after building the port before we clean out the work directory!

ph34r# cd /usr/ports/mail/qmail-tls
ph34r# make config

You'll get an NCURSES text-mode gui config screen now, and you want to make sure to check the SMTP_AUTH_PATCH and RCDLINK options before you go on. You may also want DISCBOUNCES, MAILDIRQUOTA, and LOCALTIME, at your discretion. Once that's done, we'll grab all our files and apply the FreeBSD port's patches before applying our own:

ph34r# make fetch
ph34r# make patch

Now we want to apply a couple of custom patches of our own - one to allow issuing custom SMTP 554 errors, and one to tell Qmail NOT to tell mail clients it allows CRAM-MD5 logins for authenticated SMTP (since Vpopmail's vchkpw password check program doesn't support it, so you wind up failing the first AUTH check and falling back on PLAIN every time you try to send an email otherwise).

ph34r# cd work/qmail-1.03
ph34r# fetch http://www.freebsdwiki.net/images/3/33/Qqxrc.patch.txt
ph34r# patch < Qqxrc.patch.txt

Pay careful attention to the output of the patch program: if it tells you any hunks failed, you'll need to investigate and resolve the problem or who knows what you'll wind up with. Next, we want to make sure Qmail doesn't tell mail clients that we accept CRAM-MD5 logins, since Qmail would take them but vchkpw would fail to understand them. You can use this patch if you like, but I'd recommend just using this quick-and-dirty "poor man's patch" instead:

ph34r# sed -i .bak s/CRAM-MD5\ // qmail-smtpd.c

Now we can change back into our port directory and issue a make install.

ph34r# 'cd ../..
ph34r# make install

[edit] Making or Requesting a Certificate for Qmail's TLS

Now we'll need to install a certificate for use with TLS authentication/encryption. To make a certificate request, you can type in make certificate-req - but that's all I can tell you about that process; I typically use a self-signed certificate instead (one which is not stored at Thawte or Verisign or one of the other major providers, but only on the local server). You don't have to pay for a self-signed certificate, where you would for one from Thawte or Verisign et al, but you should be aware that 1. users will be warned that they are being asked to accept a certificate not signed by a recognized authority the first time they connect to a server with a self-signed certificate, and 2. Microsoft Outlook may refuse to connect to a server with a self-signed certificate using TLS AT ALL, so you may need to forgo encryption entirely for Outlook users if you're signing your own certificate.

To use a certificate from a known-and-trusted certifying authority (Thawte/Verisign etc,) you'll need to create a CSR and send it to them when you ask them for a certificate. Be prepared to pay ~200USD for a cert. They'll send you the certificate back and you'll need to install it on your server. This is made easier by webmin, but you can do it via the commandline using OpenSSL. Certificate management is outside the scope of this article and not something to be taken lightly, as it can be more subtle and difficult than it initially appears.

Whew! With that said, actually generating the self-signed certificate is easy:

ph34r# make certificate

After entering in a bit of info at some prompts - country, state, city, that sort of thing - a certificate will be created and saved in /var/qmail/control/servercert.pem. All done. One thing worth noting - whatever you enter at the prompt that says Common Name (eg, YOUR name) []: is what's going to show up in security prompts in browsers and email clients, and frequently will trigger EXTRA "oh no something's not right!" prompting if it doesn't match the URL of your server. So in general, if you were creating this certificate for a mailserver mail.getsdeliveredhere.net, you would want to enter mail.getsdeliveredhere.net at that prompt.

[edit] Initial Qmail Configuration

Now it's time to make sure Qmail knows how to deliver the mail it receives. The way we do this is by selecting one of a large selection of possible startup scripts from /var/qmail/boot, and copying that script to /var/qmail/rc, which is symlinked to /usr/local/etc/rc.d/qmail.sh for starting and stopping Qmail when needed. We're going to be using the maildir storage format, so this is the command we'll issue:

ph34r# cp /var/qmail/boot/maildir /var/qmail/rc

Now add a line to /etc/rc.conf so that the startup script we just copied will run automatically when we boot the machine up:

ph34r# echo SENDMAIL_ENABLE="NONE" >> /etc/rc.conf

Now we'll need to generate Qmail's basic control files. Easy:

ph34r# /var/qmail/configure/config-fast
Your fully qualified host name is .
Putting  into control/me...
Putting  into control/defaultdomain...
Putting  into control/plusdomain...
Putting  into control/locals...
Putting  into control/rcpthosts...
Now qmail will refuse to accept SMTP messages except to .
Make sure to change rcpthosts if you add hosts to locals or virtualdomains!
ph34r#

Now you need to put the real name of the machine in /var/qmail/control/me - nothing fancy here, no comments, no arguments to set, just edit the file and put the name of the server in there; ie mail.getsdeliveredhere.net. Don't worry about locals, rcpthosts, or any of that good stuff - vpopmail will handle that for us, and we're going to cover that next. But first, we need to fire up qmail and make sure it's running:

ph34r# /usr/local/etc/rc.d/qmail.sh start
ph34r# ps ax | grep qmail
87877  p0  R      0:00.26 qmail-send
87878  p0  S      0:00.10 splogger qmail
87879  p0  S      0:00.00 qmail-lspawn ./Maildir/
87880  p0  S      0:00.03 qmail-rspawn
87881  p0  S      0:00.03 qmail-clean

Good deal! Now let's move on to vpopmail.

[edit] Installing vpopmail

Now, we want to make sure that we can keep our mail accounts separate from our system accounts - maybe you want every Tom, Dick, and Harry with an email account to potentially be able to use that username and password to get a shell prompt, but I certainly don't! In fact, even for those folks - like myself - who have both a mailbox and a shell account, I want to make certain that the credentials used AREN'T the same, to minimize security risks. I also want to make it easy to administer the email for multiple domains from a single machine - and that's where vpopmail comes in.

ph34r# cd /usr/ports/mail/vpopmail
ph34r# make install clean

Now we need to make a quick permissions change, so that our authenticated SMTP will work right with Vpopmail's password authentication program:

ph34r# chmod 4755 /usr/local/vpopmail/bin/vchkpw
ph34r# chown root /usr/local/vpopmail/bin/vchkpw

Mmmmm, simple. That's all we have to do with vpopmail. As a bonus, installing vpopmail also got us the ucspi-tcp port, which we'll need to use to keep a daemon listening on port 25 (and port 2525, for reasons we'll discuss later) for incoming SMTP traffic. Okay, but now that we've got a separate database for mailbox accounts from system accounts, how do we manage it? That would be two more ports - qmailadmin, and vqadmin. Vqadmin lets us add, delete, and otherwise manage entire domains we want to handle the mail for, while qmailadmin gives us a nice friendly little interface to handle individual mailboxes within individual domains. First, vqadmin:

[edit] Installing VQadmin

ph34r# cd /usr/ports/mail/vqadmin
ph34r# make install clean

Now, the vqadmin port just put its files in /usr/local/www/cgi-bin-dist/vqadmin and /usr/local/www/data-dist/images. This is probably NOT where you actually want those files to go - so you'll want to move that whole directory to wherever you actually want it served from. For the purposes of this article, we'll assume that it's a mailserver and a mailserver only, and so you're just serving everything from /usr/local/www/data and /usr/local/www/cgi-bin. So we'll move the files where we need them:

ph34r# mv /usr/local/www/cgi-bin-dist/vqadmin /usr/local/www/cgi-bin
ph34r# mv /usr/local/www/data-dist/images /usr/local/www/data

Now we'll need to add a snippet to our httpd.conf file to handle authentication for us when we use vqadmin:

# VQadmin : web interface for administering Qmail

<Directory "/usr/local/www/cgi-bin/vqadmin">
        deny from all
        Options ExecCGI
        AllowOverride AuthConfig
        Order deny,allow
</Directory>

All that does is make sure that access is allowed to our vqadmin directory, but ONLY allowed for authorized users. Next, we'll need to create a .htaccess file and a password file to tell the system what does or does not constitute an authorized user. First, create the .htaccess file in the directory you put vqadmin in (in this example /usr/local/www/cgi-bin/vqadmin):

# This is /usr/local/www/cgi-bin/vqadmin/.htaccess

AuthType Basic
AuthUserFile /usr/local/www/vqadmin.passwd
AuthName vQadmin
require valid-user
satisfy any

Now, it's time to create the password file referenced in the .htaccess file you just wrote. In this example, we're going to use the username "admin" because it's already configured in vqadmin's vqadmin.acl file to be an administrator account when present - REMEMBER, if instead you choose to use a different username, and you want that username to be able to administer vqadmin as well as look at it, you'll need to edit vqadmin.acl as well to reflect that!

ph34r# htpasswd -c /usr/local/www/vqadmin.passwd admin
New password:
Re-type new password:
Adding password for user admin

The htpasswd command, with the -c flag, will create the new file and will add the user "admin", with the password you specify twice on the command line when prompted. The screen will not echo your keystrokes or any asterisks; it will just sit there until you hit enter at each password prompt.

Okay! Now you should be ready to add your first (possibly your only) domain, using vqadmin. First, restart Apache so that the changes you made to httpd.conf will take effect:

ph34r# apachectl restart

Now fire up your web browser and check out your particular version of "www.yourdomainname.tld/cgi-bin/vqadmin/vqadmin.cgi". You should get a plain-but-functional HTML page showing you links to add/delete/otherwise maintain your domains. Add a domain here, and let's move on to qmailadmin.

[edit] Installing Qmailadmin

VQadmin handles the overall installation / deleting / privilege editing of entire domains from the system administrator's level, and it can even be used for modifying individual mailboxes, in a primitive kind of way. But for actually maintaining existing domains - and just as importantly, for delegating the handling of individual domains and individual mailboxes in those domains to the people who actually use them - what we want is Qmailadmin. First, we'll build it from the ports tree:

ph34r# cd /usr/ports/mail/qmailadmin
ph34r# make install clean

Qmailadmin dumps its files in /usr/local/www/cgi-bin.default and /usr/local/www/data.default, which is certainly not where we actually want them. Again, we'll want to put them in the correct cgi-bin directory for whatever your webserver configuration is - in our example, it looks like this:

ph34r# mv /usr/local/www/cgi-bin.default/qmailadmin /usr/local/www/cgi-bin
ph34r# mv /usr/local/www/data.default/qmailadmin /usr/local/www/data

Now the nice thing is, since Qmailadmin uses the actual mailbox usernames and passwords themselves to authenticate, you don't have to futz about any with .htaccess files and password files like VQadmin needed - once you've copied those files, you're good to go!

Go ahead and test it out by browsing to www.yourdomainname.tld/cgi-bin/qmailadmin/qmailadmin. You should get a nice little login screen asking for a username and password and domain. Remember the domain you added when you tested VQadmin earlier? Log in here as username postmaster, password whatever you set when you created the domain, and domain name as whatever domain name you created earlier. Voila! You're set up, and you can add/delete/manage/mangle mailboxes and mailing lists and what have you in this domain to your heart's content.

You can also set the postmaster password to this domain to be DIFFERENT from the password to access VQadmin, and delegate this domain's administration to someone else without worrying about them getting into something on another domain you're hosting mail for. You can even let owners of individual mailboxes login as themselves, and they'll be able to change the settings for their own mailbox, but not for others on the domain or anything domain-wide. Handy!

[edit] Installing Sqwebmail

If you have to spend much time working on other people's computers, you'll probably want to be able to use a web interface to check your email as well as administer your server. For that, we use sqwebmail - which is the webmail chunk of the Courier mailserver package, by itself. Sqwebmail assumes a maildir storage format, and it's blindingly fast compared to most other webmail packages because it does not use IMAP to communicate with the actual server - it just accesses the maildirs directly.

When we install, we'll be using the -DWITH_VCHKPW argument, to set the environment variable "WITH_VCHKPW" to true, so that the port knows we want it to build using Vpopmail authentication routines. NOTE: newer versions of this port use an NCURSES text-mode config GUI, which will let you simply check the AUTH_VCHKPW option to do this. You may also install spell-check support via the NCURSES gui.

ph34r# cd /usr/ports/mail/sqwebmail
ph34r# make -DWITH_VCHKPW
ph34r# make install
ph34r# make install-configure
ph34r# make clean

First, as usual for web-based ports, we have to copy a couple of directories from "where they got put" to "where they should be."

ph34r# mv /usr/local/www/data-dist/sqwebmail /usr/local/www/data
ph34r# mv /usr/local/www/cgi-bin-dist/sqwebmail /usr/local/www/cgi-bin

Now we'll need to add this line to root's crontab (use the command crontab -e while you are root, and check the quick docs on vi if you don't know how to use it):

0 * * * * bin /usr/local/share/sqwebmail/cleancache.pl

Now append this line to /etc/rc.conf:

sqwebmaild_enable="YES"

Now start the daemons:

ph34r# /usr/local/etc/rc.d/courier-authdaemond.sh start
ph34r# /usr/local/etc/rc.d/sqwebmail-sqwebmaild.sh start

And you should be ready to test it out! Fire up http://www.yourdomainname.tld/cgi-bin/sqwebmail/sqwebmail, and login as postmaster@yourdomainname.tld (the postmaster account @ the domain you created with vqadmin earlier) and use the same password you set for postmaster on that domain. Voila! You've got webmail.

[edit] Installing Dovecot

Okay, so now we've got Qmail running, we've got a virtual domain and its postmaster account set up and working using Vqadmin and Qmailadmin... but as of right now, the only way to check that account is via webmail. This is probably not what we want - what we really want is POP3 and/or IMAP access. There are lots of POP3 and IMAP servers available, but I've had (by far!) the best performance results (and least observed discovered vulnerabilities) from the Dovecot package, so that's what we're going to use here.

ph34r# cd /usr/ports/mail/dovecot
ph34r# make -DWITH_VPOPMAIL install

Note that we're specifying the WITH_VPOPMAIL option to make sure that Dovecot understands it should be authenticating using the Vpopmail user credentials, not system user credentials. As of the current port version of Dovecot, you actually get an ncurses GUI pop-up when you build the port; so you should make sure to manually check the VPOPMAIL box if you do. DON'T select POSTGRESQL or MYSQL or any of that other stuff - that's for advanced mail installations which use those databases instead of maildirs for delivery, and it's not what we're doing here!

At the end of the port compilation and installation process, it will ask you a couple of questions, to which you'll just answer "yes":

===>  Installing for dovecot-0.99.12.1
You need a group "dovecot".
Would you like me to create it [y]? y
Done.
You need a user "dovecot".
Would you like me to create it [y]? y
Done.

Now you'll need to fix up the dovecot.conf file to properly set Dovecot up to handle your maildirs, and to authenticate users against Vpopmail credentials.

ph34r# cd /usr/local/etc
ph34r# cp dovecot.conf.sample dovecot.conf
ph34r# edit dovecot.conf

First, look for the section which configures Dovecot's handling of mail storage. You'll find a bunch of commented-out examples, followed by this NON-commented line:

default_mail_env = mbox:/var/mail/%u

Comment that line out completely - Dovecot will handle maildirs in vpopmail's locations just fine with no config information in dovecot.conf at all, but WON'T handle them properly with that default_mail_env line specified the way it is. So:

#default_mail_env = mbox:/var/mail/%u

Next, find the two lines that look like this (but they won't be together, they will be separated by some comments):

auth_userdb = passwd
auth_passdb = passwd

And set them both to read "vpopmail" instead.

auth_userdb = vpopmail
auth_passdb = vpopmail

Finally, we'll need to change the valid_uid settings, to make sure that IMAP and POP3 logins are allowed for the vpopmail system account - and (for security reasons) ONLY the vpopmail system account! Look for the appropriate section, and - assuming that your vpopmail installation created your vpopmail system account with the uid of 89, which it should have (and if you're paranoid, cat /etc/passwd | grep vpopmail in another terminal window to check) - change it to read as following:

# Valid UID range for users, defaults to 500 and above. This is mostly
# to make sure that users can't log in as daemons or other system users.
# Note that denying root logins is hardcoded to dovecot binary and can't
# be done even if first_valid_uid is set to 0.
first_valid_uid = 89
last_valid_uid = 89

Now we need to set dovecot_enable in /etc/rc.conf:

ph34r# echo dovecot_enable="YES" >> /etc/rc.conf

Now that dovecot's configured, let's fire it up, and make sure it starts:

ph34r# /usr/local/etc/rc.d/dovecot.sh start
Starting dovecot.
ph34r# ps ax | grep dove
87484  ??  Ss     0:00.00 /usr/local/sbin/dovecot
87485  ??  S      0:00.00 dovecot-auth

Yup - we started it, and the process is running. Now we'll move on to setting up the server to actually listen for incoming SMTP traffic.

[edit] Listening for Incoming SMTP

Next, you'll want to get your server actually listening for incoming mail. You'll do that by creating a startup script to run ucspi-tcp listening on port 25 (and port 2525, for reasons we'll get into in a moment) and directing incoming connections to Qmail.

Before we can actually start tcpserver, we need to have a ruleset available to tell it what types of connections to allow from which hosts. This is MANDATORY for setting up SMTP servers, because if you don't disallow mail relay from untrusted networks, spammers will find you VERY VERY QUICKLY and use all your bandwidth irritating lots of people, some of whom will know exactly how to get hold of your ISP and tell them you're a problem child. You Do Not Want This. So, create the following as /etc/tcp.smtp:

192.168.0.:allow,RELAYCLIENT=""

Note that this assumes that your server is running on a network 192.168.0.x, which may or may not actually be the case.

Now you'll need to use the tcprules program to compile this into a cdb format ruleset. Rehash to make sure your system's PATH cache knows about tcprules, and let's do it:

ph34r# rehash
ph34r# cat /etc/tcp.smtp | tcprules /etc/tcp.smtp.cdb /etc/tcp.smtp.tmp

And finally, we'll need to generate a startup script to run tcpserver with. Some of the lines of the shell script shown below are escaped to multi-line format for readability, and while that should work fine entered as shown, I tend to recommend removing the backslashes and line breaks when you actually enter the script into your own machine, and entering those lines as single continuous lines. One less thing to worry about breaking, right?

So put the following script at /usr/local/etc/rc.d/tcpserver.sh:

#!/bin/sh

case "$1" in
start)
    /usr/local/bin/tcpserver -H -l0 -R -c 512 -x /etc/tcp.smtp.cdb -u 82 -g 81 \
    0 smtp /usr/local/bin/rblsmtpd -b -r bl.spamcop.net -r dnsbl.njabl.org \
    /var/qmail/bin/qmail-smtpd mail.isreceivedhere.net /usr/local/vpopmail/bin/vchkpw \
    /usr/bin/true 2>&1 | /var/qmail/bin/splogger rblsmtpd &

    /usr/local/bin/tcpserver -H -l0 -R -c 512 -x /etc/tcp.smtp.cdb -u 82 -g 81 \
    0 2525 /usr/local/bin/rblsmtpd -b -r bl.spamcop.net -r dnsbl.njabl.org \
    /var/qmail/bin/qmail-smtpd mail.isreceivedhere.net /usr/local/vpopmail/bin/vchkpw \
    /usr/bin/true 2>&1 | /var/qmail/bin/splogger rblsmtpd &

    ## -H tells tcpserver not to do remote DNS lookup before accepting connections
    ## -l0 tells tcpserver not to look up local host name in DNS; instead use "0" as its name
    ## -R tells tcpserver not to ask the remote server for its DNS information
    ## -c 512 tells tcpserver not to attempt to process more than 512 simultaneous connections
    ## -x specifies a rules database to control connections with
    ## -u 82 runs tcpserver under the qmaild uid
    ## -g 81 runs tcpserver under the qmaild gid
    ## 0 indicates tcpserver is running on this machine
    ## smtp (...)qmail-smtpd specifies to pass SMTP connections to qmail-smtpd
    ##       ... or ...
    ## 2525 (...)qmail-smtpd specifies to pass connections on port 2525 to qmail-smtpd
    ## rblsmtpd checks for blacklisted IP addresses before accepting SMTP
    ## -b specifies an SMTP 553 error code to return to blacklisted servers
    ## -r is specified before each successive RBL source
    ## descriptor 2 is sent to splogger to create standard log entries attributed to rblsmtpd
    ## end the line with & or the process hangs the console that starts it!
    ##
    echo "tcpserver-SMTP started"
    ;;
stop)
    ## no action needs to be taken to kill tcpserver processes
    ##
    exit 0
    ;;
*)
    echo "Usage: leave this script alone, it's for boot only."
    exit 64
    ;;
esac

So why are we running another instance of tcpserver on port 2525, as well as the standard one on port 25? Because there are a lot of ISPs these days that are blocking all traffic to destination port 25 anywhere outside their own network. So you will probably want to be able to set up your mail clients, on portable machines, to access your authenticated SMTP on a non-standard port to get around that limitation. I use 2525 because it's easy to remember, but of course you can pick whatever you like.

Whew! Now let's make sure our tcpserver.sh script is executable, then fire it up and make sure tcpserver is running.

ph34r# chmod 755 /usr/local/etc/rc.d/tcpserver.sh
ph34r# /usr/local/etc/rc.d/tcpserver.sh start
tcpserver-SMTP started.
ph34r# ps ax | grep tcpserver
87717  p0  S      0:00.00 /usr/local/bin/tcpserver -H -l0 -R -c 512 -x /etc/tcp
87719  p0  S      0:00.00 /usr/local/bin/tcpserver -H -l0 -R -c 512 -x /etc/tcp

Great - at this point we've got what appears to be a fully functional mailserver. All that's left is installing our spam and virus filtering capabilities, and then testing everything to make sure it works.

[edit] Adding Spam and Virus Filtering

WARNING: contents of this section are under extremely heavy development and what you see here is likely to be outdated.

Do not attempt to implement this filtering mechanism unless you REALLY understand what you're doing and feel confident you can fix any problems you may encounter. You may also wish to inspect the discussion page for this article for (considerably) more as-it-develops information.

We're going to want spamassassin for spam filtering and clamav for virus filtering. Install them.

ph34r# cd /usr/ports/security/clamav && make install clean

once that's done...

ph34r# cd /usr/ports/mail/p5-Mail-SpamAssassin && make install clean

and now, to allow them each to actually start up...

ph34r# echo clamav_clamd_enable=YES >> /etc/rc.conf
ph34r# echo clamav_freshclam_enable=YES >> /etc/rc.conf
ph34r# echo spamd_enable=YES >> /etc/rc.conf

Fire them up - /usr/local/etc/rc.d/clamav-clamd.sh start && /usr/local/etc/rc.d/clamav-freshclam.sh start && /usr/local/etc/rc.d/spamd.sh start - and make sure they're running. ps waux | grep clam should show you a freshclam process and a clamd process, and ps waux | grep spam should show you a spamd process.

Now it's time to install Qsheff. Qsheff is a wrapper for Qmail's queuing program, qmail-queue. It's stone axe simple - no environment variable messiness to get stepped on, it simply moves qmail-queue to qmail.orig, puts a symlink to itself where qmail-queue used to be, and then gives the message to the "real" qmail-queue after it's done. So let's install it.

ph34r# cd /usr/ports/mail/qsheff && make install clean

After the port is installed, it will give you a message telling you it isn't really installed yet, first you need to run a shell script to actually replace the original qmail-queue. This is true, but first we'll need to kill off all tcpserver and qmail processes.

ph34r# killall tcpserver
ph34r# /usr/local/etc/rc.d/qmail.sh stop
ph34r# /usr/local/etc/qsheff/install-wrapper.sh
* Moving qmail-queue to /var/qmail/bin/qmail-queue.orig... 
* Creating qmail-queue link to /var/qmail/bin/qmail-qsheff... 

! Don't forget to start qmail-send and qmail-smtpd.

Before we actually restart our qmail processes, let's go ahead and get qsheff configured the way we want it.

In its default configuration, qsheff gives you virus filtering via clamav, but no spam filtering - qsheff doesn't support that natively; you have to add your own script to handle that. Which is exactly what we're going to do. First, we'll need the port safecat to let us easily deliver mail to a maildir of our choice, so we install that.

ph34r# cd /usr/ports/sysutils/safecat && make install clean 

then create this script as /usr/local/bin/maildump.pl:

#!/usr/bin/perl 
 
############################################################################################ 
# maildump.pl - this script needs the safecat package, and is primarily intended for use   # 
# as a spamassassin / clamav wrapper for use with the qsheff QMAIL-QUEUE wrapper.          # 
#                                                                                          # 
# in this configuration it will save a copy of spam or virus messages if the appropriate   # 
# config variables are set, or save a copy of EVERYTHING if quarantine_all is set, and it  # 
# will SMTP 554 reject any messages $antispam_agent or $antivirus_agent gives non-zero     # 
# exit codes for, BEFORE the originating SMTP connection is dropped - meaning true         # 
# positives won't generate bounces that harm innocent bystanders (at least not from YOUR   # 
# server, though the originating spam gateway may not be as nice) and any false positives  # 
# WILL get immediate notification that their mail was not delivered.                       # 
#                                                                                          # 
# (c) 2006-02-16 JRS Systems.  All rights reserved under BSD license.  You may use this    # 
#                              script freely for any purpose commercial or noncommercial   # 
#                              as long as this notice remains intact.                      # 
############################################################################################ 
 
# IMPORTANT: $ServerName must match the contents of /var/qmail/control/me! 
#            (Hardcoded instead of dynamically read for efficiency concerns.)
#
$ServerName = 'mail01.advantex.net';

$quarantine_spam = 1; 
$quarantine_virus = 1; 
 
# this setting saves copies of EVERYTHING in the quarantine directory - spam, viruses, and 
# perfectly good mail alike.  Some sysadmins like to store copies of everything that crosses 
# their server for a few days this way in order to bail users out of "omg I deleted it, can 
#you get it back for me?" type incidents. 
 
$quarantine_all = 0; 
 
$antivirus_agent = "/usr/local/bin/clamdscan --quiet"; 
$antispam_agent="/usr/local/bin/spamc -E"; 
$delivery_agent="/usr/local/bin/maildir"; 
 
$quarantine_dir="/usr/local/vpopmail/domains/YOUR_DOMAIN/YOUR_MAILBOX_NAME/Maildir/.INBOX.quarantine"; 
 
################################################################################################# 
# END CONFIGURATION VARIABLE SECTION ############################################################ 
################################################################################################# 
 
# qsheff creates a temporary directory with the originating email split into several files 
# stored in the order of their occurrence in the full message.  All these files concatenated 
# = the original email. 
# 
$email_files='_headers_ textfile*'; 
print `chmod -R 755 .. `; 
 
# Spamc is the client half of the client/server implementation of SpamAssassin, and the -E 
# argument instructs it to mangle up the message if it flags positive while also giving you 
# a non-zero exit code on its way out if it flags positive. 
# 
# /usr/local/bin/maildir is an executable installed with the safecat package which simply 
# delivers an email to a maildir.  You want to point it to the directory containing the 
# cur, new, and tmp directories, NOT straight to maildir/new. 
# 
# If you choose, you can have maildump.pl deliver quarantined mails to an 
# actual retrievable-with-a-mail-client maildir.  Note that while that's convenient, 
# it does mean you're potentially exposing every single email you quarantine - or every 
# single mail, PERIOD, if you set $quarantineall - to the internet behind the dubious 
# protection of a single accountname and password.  Think carefully - if you aren't 
# SURE you want to do that, pick a location that ISN'T accessible directly to vpopmail 
# or whatever else you may have handling user accounts. 
# 
 
# We don't want to do spamassassin scanning on authenticated outbound messages.
$ourReceivedRegex='^\s*Received:\s*from\s*.*\s*(HELO .*)\s*(.*)\s*by\s*' . $ServerName;
$authReceivedRegex='^\s*Received:\s*from\s*.*\s*(HELO .*)\s*(.*\@.*@.*)\s*by\s*' . $ServerName;

@headers=`cat _headers_`;

$stop = 0;
foreach $line (@headers) {
        if ($line =~ /$ourReceivedRegex/i && $stop == 0) {
                $auth = ($line =~ /$authReceivedRegex/i);
                $stop = 1;
        }
}

# The special variable $? captures the exit code of the last code forked by the script - in
# this case, the exit code given when $filter_agent finished up.  For reasons known only to
# Larry Wall, Perl "helpfully" (?) bitwise shifts exit codes captured in $? - so we have to
# bitwise shift them BACK to get any good out of them.

if (! $auth) {
        @in=`cat $email_files | $antispam_agent`;
        $spam_found = ($? >> 8);
 } else {
        @in=`cat $email_files`;
        $spam_found = 0;
}

# Now time for anti-virus scanning.

print `$antivirus_agent`; 
$virus_found = ($? >> 8); 
 
if (  $quarantine_all!=0 || 
      ($quarantine_spam !=0 && $spam_found != 0) || 
      ($quarantine_virus !=0 && $virus_found !=0) 
   ) 
{ 
   open (FH, "| $delivery_agent $quarantine_dir"); 
   foreach $line (@in) { 
       print FH "$line"; 
   } 
   close (FH); 
} 
 
# Pass along an exit code telling qsheff whether mail is clean.  You may need or want to write 
# your own logic routine to deliver your own specific exit code instead of simply passing along 
# a plain boolean true or false to signify clean or dirty mail. 
# 
# qsheff will cause qmail to SMTP 554 reject the mail if you pass it anything other than a zero 
# here. 
 
$exit_code = ($spam_found != 0 || $virus_found != 0); 
 
exit $exit_code; 

Once you've done that, you'll need to set the following variables in /usr/local/etc/qsheff/qsheff.conf:

enable_filter_prog=1
filter_prog=/usr/local/bin/maildump.pl 
blackhole_enable=1

These variables tell Qsheff that you're using an external filtering agent, it's located at /usr/local/bin/maildump.pl (we'll be creating that momentarily), and that you don't want Qsheff to emit useless bounce messages to innocent third parties when it encounters spam or viruses. Now three more variables in qsheff.conf:

enable_virus_prog=0 
enable_body_filter=0
enable_subject_filter=0

These tell it not to use its native clamav filtering (we'll be handling that in maildump.pl), and not to use its own native spam filtering mechanism, since we'll be handling that using spamassassin (also in maildump.pl). NOTE: if you don't disable body_filter and subject_filter, you will not get quarantine copies of any messages that trigger them!

Now for a little configuration within maildump.pl itself - did you notice the configuration variable for $quarantine_dir? Set that to be wherever you want to save copies of "bad" mail. Read the comments around it - you can choose a "working" maildir which is accessible to mail clients, or you may choose a hidden one that you have to get at from the shell, or rsync, or ftp, or what have you. That's your decision.

If you specified a $quarantine_dir that does not exist yet, you'll need to create it now. If you chose to quarantine mail to a vpopmail-accessible area, first browse to your vQadmin and Qmailadmin pages to create the domain and the user account that the quarantined stuff will go in, if they don't already exist. Then you can use /var/qmail/bin/maildirmake to create an IMAP (or Sqwebmail) accessible folder underneath that, if you don't want that account to be nothing but the quarantine:

#ph34r /var/qmail/bin/maildirmake /usr/local/vpopmails/mydomain/myname/Maildir/.INBOX.quarantine

This would cause an IMAP/Sqwebmail-accessible folder named "quarantine" to be created in the account of myname@mydomain.

Now there's the question of what, exactly, we want to quarantine. You can individually choose whether to save copies of spam, viruses, or even ALL mail (virus, spam, and perfectly good mail alike) in $quarantine_dir by appropriately setting the $quarantine_spam, $quarantine_virus, and $quarantine_all variables. (By default, $quarantine_spam and $quarantine_virus are set - causing both spam and viruses to be quarantined - but $quarantine_all is not.) Note if you do enable $quarantine_all, good mail does still gets delivered where it's supposed to go - maildump just delivers an extra copy of it to the quarantine directory.

Now we'll need to make sure that our quarantine directory doesn't consume the whole hard drive. One way to do this is to set up a crontab or /etc/periodic script to clean out older stuff once per day. If you want to use /etc/periodic to handle purges automatically for you, create a new file called /etc/periodic/daily/900.purge_quarantine:

cd /usr/local/vpopmails/mydomain/myname/Maildir/.INBOX.quarantine/new
find . -ctime +30 -delete
cd /usr/local/vpopmails/mydomain/myname/Maildir/.INBOX.quarantine/cur
find . -ctime +30 -delete
cd /usr/local/vpopmails/mydomain/myname/Maildir/.INBOX.quarantine/tmp
find . -ctime +30 -delete

This will cause the server to go rooting through the quarantine folders once per day and get rid of everything in there that's more than 30 days old. NOTE: base how many days it looks back on how much traffic you're expecting to catch in here - if you set $quarantine_all, remember that this directory may grow VERY RAPIDLY, and you may want to purge things at 15 or even 7 days instead of 30!

[edit] Testing SMTP, IMAP, and POP3 via Telnet

As our first step in testing the server's basic functions, we'll want to test the tcpserver / Qmail combo by simulating some incoming traffic - which will also then give us an email to check for the presence of in our IMAP and POP3 tests later.

First, since we killed off the tcpserver and qmail processes to install qsheff, we'll need to start them back up:

ph34r# /usr/local/etc/rc.d/qmail.sh start
ph34r# /usr/local/etc/rc.d/tcpserver.sh start

Now we can use telnet to interact directly with the server and make sure it's doing what it's supposed to do:

ph34r# telnet localhost 25
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
220  ESMTP
HELO justtesting
250
MAIL FROM: me@telnettingin.com
250 ok
RCPT TO: postmaster@mail.getsdeliveredhere.net
250 ok
DATA
354 go ahead
To: postmaster@mail.getsdeliveredhere.net
From: telnetclient@mail.getsdeliveredhere.net
Subject: this is a test message
Just testing SMTP functionality by telnetting in to port 25.  I'll end this message now
by entering in a line with nothing but a period in it and hitting return.
.
250 ok 1103093638 qp 87827
QUIT
221
Connection closed by foreign host.

Okay - our server just accepted a telnet connection, responded like a mailserver, and accepted a nice little test email to the postmaster account at the domain we set up earlier. Now let's make sure that we can see that email using the IMAP protocol:

ph34r# telnet localhost 143
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
* OK dovecot ready.
A LOGIN postmaster@mail.getsdeliveredhere.net thisismypassword
A OK logged in.
A SELECT inbox
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft)
* OK [PERMANENTFLAGS (\Answered \Flagged \Deleted \Seen \Draft \*)] Flags permitted.
* 1 EXISTS
* 1 RECENT
* OK [UIDVALIDITY 1103088195] UIDs valid
* OK [UIDNEXT 1] Predicted next UID
A OK [READ-WRITE] Select completed.
A LOGOUT
* BYE Logging out
A OK Logout completed.
Connection closed by foreign host.

Great! By using telnet, we've just verified on a very direct level that our IMAP server answers incoming connections, successfully authenticates users, and successfully opens their inboxes - see the * 1 EXISTS and * 1 RECENT lines? That's the email we telnetted in earlier. So, if we have any problems trying to set up IMAP accounts on actual clients later, now we know that they're client problems or network problems - not server configuration problems. Verifying these things ahead of time will make your life MUCH easier.

Now let's test POP3 (assuming you want to allow clients to use POP3 as well):

ph34r# telnet localhost 110
Trying 127.0.0.1...
Connected to localhost.localdomain.
Escape character is '^]'.
+OK dovecot ready.
USER postmaster@mail.getsdeliveredhere.net
+OK
PASS thisismypassword
+OK Logged in.
LIST
+OK 1 messages:
1 354
.
QUIT
+OK Logging out.
Connection closed by foreign host.

Tested, confirmed, and good to go!

[edit] Configuring Email Clients

Well, we're all done now - we've got a fully functioning mailserver that can handle sending and receiving mail, authenticated SMTP (with and without TLS encryption), webmail, virtual domains, delegation of administration by domain and by individual mailbox, IMAP storage, POP3 for people who can't use IMAP, spam filtering by RBL, and more. Once you've set up your domains and your user accounts (and don't forget for every domain you set up, you will also need DNS pointing the mail services for that domain to this server, which isn't covered in this article), the only thing left is configuring your clients. Here's a few basic bullets to help you with common SNAFUs with that:

• remember to set the "username" as mailbox@domainname.tld, not just "mailbox"
• remember that in order to send email, the client either has to be on the network named in /etc/tcp.smtp (actually in the compiled version, /etc/tcp.smtp.cdb) or has to login using Authenticated SMTP
• remember that if you expect the client machine to be able to connect reliably from foreign networks, you should configure it to connect to your SMTP server on a nonstandard port
• remember that if you're using a self-signed certificate, you will get security warnings from some clients if you use TLS encryption, and some other clients (notably Microsoft Outlook) will refuse to connect at all using TLS encryption if your certificate is self-signed

And that's pretty much it - enjoy!

[edit] See also

http://www.mail-toaster.com/

http://freebsd.qmailrocks.org/

http://qmail.org

http://lifewithqmail.org