This entry describes how to set up certificates to enable sendmail to use TLS (transport-layer security). Note that for Solaris 10, the command /usr/sfw/bin/openssl and the configuration file /etc/sfw/openssl/openssl.cnf are provided by the SUNWopenssl-commands and SUNWopensslr packages respectively, which are included in the End User (and greater) package meta-clusters. So if your system is installed with the Core (or lesser) package meta-cluster, you will need to install these two packages.

1. Set up certificates. Read this section in its entirety before trying any of the steps, as there is an important note at the end of the section regarding certificate expiration. (Note that this section was derived from another page written by Greg Shapiro of Sendmail.)
   To create a certificate authority:

   1# cd /etc/mail
   2# mkdir -p certs/CA
   3# cd certs/CA
   4# mkdir certs crl newcerts private
   5# echo "01" > serial
   6# cp /dev/null index.txt
   7# cp /etc/sfw/openssl/openssl.cnf .
   

   Note that the above source path is for Solaris 10; if anyone happens to be doing the same on say, Mac OS X, the source path is /System/Library/OpenSSL/openssl.cnf; FreeBSD 4.11 puts the file in /etc/ssl/openssl.cnf.

   8# vi openssl.cnf
   

   Set values: I changed the dir value from /etc/sfw/openssl to /etc/mail/certs/CA and the stateOrProvinceName_default value from Some-State to California; the former was important but the latter was not.)

   9# openssl req -new -x509 -keyout private/cakey.pem -out cacert.pem -days 365 -c
   onfig openssl.cnf
   

   Notes on how to answer the questions: for Organization Name, I answer Sun Microsystems, for Organizational Unit Name, I answer Solaris, for Common Name, I answer with the fully-qualified host-name (FQHN) of the machine in question (see check-hostname(1M) for help on this), and for Email Address, I answer with my e-mail address.
   To make a new certificate:

   10# openssl req -nodes -new -x509 -keyout newreq.pem -out newreq.pem -days 365 -
   config openssl.cnf
   

   (The certificate and private key are in the file newreq.pem.) To sign the new certificate with the certificate authority:

   11# openssl x509 -x509toreq -in newreq.pem -signkey newreq.pem -out tmp.pem
   12# openssl ca -config openssl.cnf -policy policy_anything -days 365 -out newcer
   t.pem -infiles tmp.pem
   13# rm -f tmp.pem
   

   (The file newcert.pem contains the signed certificate; newreq.pem still contains the unsigned certificate and private key.)

   Note that commands 9, 10 and 12 have a -days 365 argument, which specifies how long until the certificate expires. There does not appear to be a reliable way to extend certificates, and going back to every machine and reapplying this process a year later is a major pain, as I found out the hard way. So the second time thru, I calculated how many days it would be until January 18, 2038, and used that value.

2. Tell sendmail.cf about the certificates. (Note that this section was derived from another page written by Claus Aßmann, also of Sendmail.) Add:

   define(`confCACERT_PATH', `/etc/mail/certs')dnl
   define(`confCACERT', `/etc/mail/certs/CAcert.pem')dnl
   define(`confSERVER_CERT', `/etc/mail/certs/MYcert.pem')dnl
   define(`confSERVER_KEY', `/etc/mail/certs/MYkey.pem')dnl
   define(`confCLIENT_CERT', `/etc/mail/certs/MYcert.pem')dnl
   define(`confCLIENT_KEY', `/etc/mail/certs/MYkey.pem')dnl
   

   to your .mc file and rebuild your sendmail.cf file and install it in /etc/mail.
3. General sym-links:

   # cd /etc/mail/certs
   # ln -s CA/cacert.pem CAcert.pem
   # ln -s CA/newcert.pem MYcert.pem
   # ln -s CA/newreq.pem MYkey.pem
   

4. Fix permissions:

   # chmod go-r MYkey.pem
   

5. Specific sym-link (note that this section was also derived from Claus' STARTTLS page):

   # C=CAcert.pem
   # ln -s $C `openssl x509 -noout -hash < $C`.0
   

6. Install other host(s) certificates: For any hosts you wish to exchange secure mail with, grab that host's copy of /etc/mail/certs/CAcert.pem (or whatever file sendmail.cf's CACertFile option points to on that host) and copy it to /etc/mail/certs/host.domain.cert.pem on your host (where host.domain is the other host's FQHN), and repeat step 5, except using C=host.domain.cert.pem.

By doing the above, I am able to send mail between hosts which are both running a version of sendmail compiled with STARTTLS defined and linked with both libssl and libcrypto, and the Received: header shows e.g.:

Received: from jurassic.eng.sun.com ([IPv6:2002:8192:56bb:9259::8192:5932])
        by opal.eng.sun.com (8.13.4+Sun/8.13.4) with ESMTP id j2TNUB8i242496
        (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK)
        for <jbeck@opal.eng.sun.com>; Tue, 29 Mar 2005 15:30:11 -0800 (PST
)
Received: from opal.eng.sun.com (opal.SFBay.Sun.COM [129.146.89.54])
        by jurassic.eng.sun.com (8.13.4+Sun/8.13.4) with ESMTP id j2TNU7cl571102
        (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK)
        for <jbeck@jurassic.eng.sun.com>; Tue, 29 Mar 2005 15:30:07 -0800 
(PST)

where the verify=OK is the key part (see Claus' STARTTLS page for an explanation of what the various verify= values correspond to).

I've spent a lot of time over the past couple of months trying out some new (and some not so new) anti-spam techniques. Note that this article assumes some familiarity with sendmail m4 macros; see $CFDIR/README for background and all sorts of details on these, where $CFDIR is one of:

• /etc/mail/cf on Solaris 10
• /usr/lib/mail on Solaris 7, 8 or 9
• the cf sub-directory of the sendmail distribution for people "rolling their own"

These techniques are in the form of FEATURE and HACK m4 macros (the difference being that the former are provided and blessed by sendmail.org / Solaris whereas the latter are not, though a HACK may evolve into a FEATURE in a future release). For a HACK, one would use

        HACK(`hack-name')dnl

in one's .mc file, likewise

        FEATURE(`feature-name')dnl

When installing hacks, one must create $CFDIR/hack (if it does not already exist) and place hack-name.m4 in that directory. Note that the sendmail distribution comes with such a sub-directory but Solaris does not.

Also, to explain some terms used below: the access list is enabled by the FEATURE(`access_db') macro; details on this are in $CFDIR/README, both in its sub-section in the FEATURES section, and in the ANTI-SPAM CONFIGURATION CONTROL section. And FEATURE(`delay_checks') is strongly recommended, as it is needed to enable the overrule by an OK entry in the access list that I mention in a few places; this feature is also described in its subsection in the FEATURES section, as well as in the "Delay all checks" sub-section of the ANTI-SPAM CONFIGURATION CONTROL section.

Anyway, onto the details. In the order I started deploying them:

• The first is HACK(`block _bad_helo'), written by Neil Rickert, a professor in the Computer Science Department at Northern Illinois University and a volunteer sendmail.org contributor. SMTP clients are supposed to send the client FQHN (fully qualified host name) as the HELO/EHLO parameter, but many broken clients send the server FQHN (or IP address) instead, or something without a ".". This rejects any such transmissions. The upside is that I have found it to block a good amount of spam, with no false positives for me. A couple of users of my personal domain have had some small number of false positives with it, though. And the down side is that it cannot be overruled by an OK entry in the access list. Bart has had a lot of troubles with this rule; apparently old versions of Netscape and early version of Mac's Mail.App got this wrong.
• The second is a regular feature: DNS-based black-lists. I use (note: line wrapped for readability)

    FEATURE(`enhdnsbl', `bl.spamcop.net',
          `"Spam blocked see: http://spamcop.net/bl.shtml?"$&{client_addr}',
          `t')dnl
    

  while Bart uses

    FEATURE(`dnsbl', `sbl-xbl.spamhaus.org')dnl
    

  Both have proven extremely effective with very few false positives, and this feature, using whichever list, has the added virtue of allowing an OK override in the access list.
• The last is HACK(`require_rdns'), also written by Neil Rickert. I enhanced Neil's original version so that it would allow an OK override in the access list. This enhanced version has been unbelievably effective, in sheer numbers, while also so far amazingly accurate (I estimated a false positive rate of 0.5% after a few weeks, but I think that may have gone even lower since I white-listed a few sites). As the name suggests, it requires that the SMTP client's IP address reverse map to some name, and also that the name forward map to the same address. (An IP address can have multiple A records; this merely requires that the original IP address is one of them.) This is the single most effective anti-spam rule I have ever deployed.

Overall, spam getting thru my personal domain's mail server to my users (including myself, my wife, my siblings, our mom, etc.) has dropped about 90% since I started using these techniques, despite the ever-increasing spam trends on the rest of the Internet.

I just answered this in comp.unix.solaris for what seems like the millionth time, so I thought I might as well record the answer here for others who might be interested...

People who run mail servers need to be careful not to be an "open relay", or they will get added to various black-lists, and all sorts of mail delivery problems will ensue, as spammers take advantage of open relays when and wherever they find them.

Basically, a message has to either "start here" or "finish here" or it is considered a relay. There is a test and a parameter for each condition. The parameter for "finish here" is the domain part (i.e., right-hand side) of each recipient's e-mail address, and the test is "is that my host name?", though it is often extended to "is that host in my domain?". The parameter for "start here" is the IP address of the client side of the SMTP connection, and the test is "is this the loopback address" (i.e., "is this the local host?"), though that is also often extended to "does this IP address reverse-map to a name in my domain?"

Note that sendmail has disabled relaying by default since version 8.9, which first shipped with Solaris 7, and that the .mc file used to generate the default sendmail.cf on Solaris contains this macro:

DOMAIN(`solaris-generic')

and that domain file contains:

FEATURE(`relay_entire_domain')

which enables the extensions to which I alluded above. Details can be found in the README file, which is in /usr/lib/mail/ on Solaris 9 and earlier, but in /etc/mail/cf/ on Solaris 10; see the ANTI-SPAM CONFIGURATION CONTROL section of that file in particular.

A common gotcha when setting up a Solaris box on the Internet and trying to send mail has been:

Host unknown (Name server: mailhost.example.com: host not found)

This is because Solaris has for many years shipped two sendmail config files: one for the "real" Internet (main.cf) and one for inside a firewall or elsewhere where DNS may not be configured properly, and average "dumb" hosts need to punt to some "smart" host (subsidiary.cf). As you might guess from the above message, that "smart" host is assumed to have the name mailhost.$m (where $m is the macro for the local domain, "example.com" in the above example) . For backwards compatibility reasons, sendmail.cf has always been a copy of subsidiary.cf rather than main.cf .

Well, starting in Solaris 10 build 63, this problem is fixed. More accurately, the fix is in sendmail version 8.13, which was introduced in s10_63; hopefully, 8.13 will be back-ported to Solaris 9 before too long. I should have thought of this years ago, but better late than never: by inventing a new FallbackSmartHost option, sendmail can handle either scenario in a single configuration:

If specified, the FallBackSmartHost will be used in a last-ditch effort for each host. This is intended to be used by sites with "fake internal DNS", e.g., a company whose DNS accurately reflects the world inside that company's domain but not outside.

This option is specified, with the value mailhost.$m as subsidiary.cf used to use, in the default sendmail.cf, and there is no longer a main.cf or a subsidiary.cf . (Well, technically, they both can still be found, but only as sym-links to sendmail.cf for backwards compatibility.)

Finally, after all these years, a grand unified sendmail configuration! :-)

Well, it's not all that new, as sendmail 8.12 was released over three years ago, but since few people pay attention to sendmail more than they have to, the major changes in 8.12 and their side effects may have gone unnoticed.

First, some background: sendmail has some historical notoriety from a security point of view, since it for many years was a setuid-root application, and thus a buffer overflow for example could make sendmail a vector for an unprivileged user to take over the system. Starting with version 8.12, however, sendmail lost its setuid bit in favor of becoming setgid to a new group.

To make queueing work, a split-queue model was created: /var/spool/mqueue as the traditional "server" queue with the same 750 root, bin permissions, and the new /var/spool/clientmqueue as the new (you guessed it) "client" queue with 770 smmsp, smmsp permissions. For the curious, "smmsp" is the new user and group invented for these purposes, having UID and GID values of 25 (chosen to match the SMTP port value in /etc/services), and it stands for "SendMail Message Submission Program".

The default configuration involving all "direct submission" messages [meaning messages submitted via sendmail invoked from the command line or comparably by programs such as mailx(1)] will go to the client queue. Messages in the client queue are then transferred to the SMTP server on the local host, which then proceeds in the traditional way. Thus running a sendmail daemon has become critical even for out-bound mail, at least from certain sources [such as cron(1M)]. See entry 3.44 of the sendmail FAQ for more details on this, as well as how to configure around this for hosts where accepting in-bound from other hosts is undesirable.

Many people with whom I regularly correspond know that I have a strong preference against mail in "pure" HTML format. Here are my reasons, along with recommended alternatives.

1. Spammers can hide tiny images: just a single pixel, small enough that your eye would likely miss it anyway, and the more insidious ones are the same color as the background. The URLs provided for your mail reader to fetch these images can contain encoded data which spammers can use to confirm your identity. I.e., each message they send can contain a slightly different URL, and when a mail reader fetches an image, it serves as confirmation for the spammers that the address corresponding to that particular URL is "live". This will in turn increase the likeliness of getting even more spam.
2. Not all mail readers handle HTML well. Most modern mail readers do, especially those with a GUI, but many older mail readers, especially those which are screen-based, handle HTML badly or not at all. Some people may use a GUI mail reader at work but a screen-based one at home or when traveling.
3. HTML takes up more bandwidth than plain text. Although this is not an issue in many environments (such as a high-speed LAN), in other environments (such a when traveling and stuck with a low-speed dial-up line) the extra bandwidth can be quite inconvenient.

The ideal alternative, when possible, is simply to send plain text, as it is sufficient for the vast majority of e-mail conversations.

When richer mark-up is needed, however, most mail programs which can generate HTML can also generate mixed text and HTML: the message's primary MIME type is multipart/alternative, with the first part being text/plain and the second part being text/html. Conforming mail readers will display HTML if they understand it, or plain text if they don't grok HTML (unless the user has configured it to display plain text by default).

For the curious, I use exmh at work and home but nmh (the CLI-based mail reading set of programs on which exmh is based) when on the road or in any other low-bandwidth environment. Exmh can display HTML, but it is much slower than displaying plain text, and not all constructs are well supported. I have mine configured to display plain text by default for multipart/alternative messages. I also have SpamAssassin configured to score "pure" HTML mail very highly, as the vast majority of such mail which I receive is indeed spam.