This entry describes how to set up certificates to enable sendmail to use TLS (transport-layer security). Note that for Solaris 10, the command /usr/sfw/bin/openssl and the configuration file /etc/sfw/openssl/openssl.cnf are provided by the SUNWopenssl-commands and SUNWopensslr packages respectively, which are included in the End User (and greater) package meta-clusters. So if your system is installed with the Core (or lesser) package meta-cluster, you will need to install these two packages.

1. Set up certificates. Read this section in its entirety before trying any of the steps, as there is an important note at the end of the section regarding certificate expiration. (Note that this section was derived from another page written by Greg Shapiro of Sendmail.)
   To create a certificate authority:

   1# cd /etc/mail
   2# mkdir -p certs/CA
   3# cd certs/CA
   4# mkdir certs crl newcerts private
   5# echo "01" > serial
   6# cp /dev/null index.txt
   7# cp /etc/sfw/openssl/openssl.cnf .
   

   Note that the above source path is for Solaris 10; if anyone happens to be doing the same on say, Mac OS X, the source path is /System/Library/OpenSSL/openssl.cnf; FreeBSD 4.11 puts the file in /etc/ssl/openssl.cnf.

   8# vi openssl.cnf
   

   Set values: I changed the dir value from /etc/sfw/openssl to /etc/mail/certs/CA and the stateOrProvinceName_default value from Some-State to California; the former was important but the latter was not.)

   9# openssl req -new -x509 -keyout private/cakey.pem -out cacert.pem -days 365 -c
   onfig openssl.cnf
   

   Notes on how to answer the questions: for Organization Name, I answer Sun Microsystems, for Organizational Unit Name, I answer Solaris, for Common Name, I answer with the fully-qualified host-name (FQHN) of the machine in question (see check-hostname(1M) for help on this), and for Email Address, I answer with my e-mail address.
   To make a new certificate:

   10# openssl req -nodes -new -x509 -keyout newreq.pem -out newreq.pem -days 365 -
   config openssl.cnf
   

   (The certificate and private key are in the file newreq.pem.) To sign the new certificate with the certificate authority:

   11# openssl x509 -x509toreq -in newreq.pem -signkey newreq.pem -out tmp.pem
   12# openssl ca -config openssl.cnf -policy policy_anything -days 365 -out newcer
   t.pem -infiles tmp.pem
   13# rm -f tmp.pem
   

   (The file newcert.pem contains the signed certificate; newreq.pem still contains the unsigned certificate and private key.)

   Note that commands 9, 10 and 12 have a -days 365 argument, which specifies how long until the certificate expires. There does not appear to be a reliable way to extend certificates, and going back to every machine and reapplying this process a year later is a major pain, as I found out the hard way. So the second time thru, I calculated how many days it would be until January 18, 2038, and used that value.

2. Tell sendmail.cf about the certificates. (Note that this section was derived from another page written by Claus Aßmann, also of Sendmail.) Add:

   define(`confCACERT_PATH', `/etc/mail/certs')dnl
   define(`confCACERT', `/etc/mail/certs/CAcert.pem')dnl
   define(`confSERVER_CERT', `/etc/mail/certs/MYcert.pem')dnl
   define(`confSERVER_KEY', `/etc/mail/certs/MYkey.pem')dnl
   define(`confCLIENT_CERT', `/etc/mail/certs/MYcert.pem')dnl
   define(`confCLIENT_KEY', `/etc/mail/certs/MYkey.pem')dnl
   

   to your .mc file and rebuild your sendmail.cf file and install it in /etc/mail.
3. General sym-links:

   # cd /etc/mail/certs
   # ln -s CA/cacert.pem CAcert.pem
   # ln -s CA/newcert.pem MYcert.pem
   # ln -s CA/newreq.pem MYkey.pem
   

4. Fix permissions:

   # chmod go-r MYkey.pem
   

5. Specific sym-link (note that this section was also derived from Claus' STARTTLS page):

   # C=CAcert.pem
   # ln -s $C `openssl x509 -noout -hash < $C`.0
   

6. Install other host(s) certificates: For any hosts you wish to exchange secure mail with, grab that host's copy of /etc/mail/certs/CAcert.pem (or whatever file sendmail.cf's CACertFile option points to on that host) and copy it to /etc/mail/certs/host.domain.cert.pem on your host (where host.domain is the other host's FQHN), and repeat step 5, except using C=host.domain.cert.pem.

By doing the above, I am able to send mail between hosts which are both running a version of sendmail compiled with STARTTLS defined and linked with both libssl and libcrypto, and the Received: header shows e.g.:

Received: from jurassic.eng.sun.com ([IPv6:2002:8192:56bb:9259::8192:5932])
        by opal.eng.sun.com (8.13.4+Sun/8.13.4) with ESMTP id j2TNUB8i242496
        (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK)
        for <jbeck@opal.eng.sun.com>; Tue, 29 Mar 2005 15:30:11 -0800 (PST
)
Received: from opal.eng.sun.com (opal.SFBay.Sun.COM [129.146.89.54])
        by jurassic.eng.sun.com (8.13.4+Sun/8.13.4) with ESMTP id j2TNU7cl571102
        (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK)
        for <jbeck@jurassic.eng.sun.com>; Tue, 29 Mar 2005 15:30:07 -0800 
(PST)

where the verify=OK is the key part (see Claus' STARTTLS page for an explanation of what the various verify= values correspond to).