We put a lot of thought into planning for failure when we setup our sites (like www.sun.com, blogs.sun.com and so on). Every component is redundant, from border firewalls to load-balancers to front end web servers to root disks. We even put the gear in separate racks on separate power, just in case someone accidentally knocks both power cables out. This is arranged in odd and even sides, and servers are placed in the corresponding side, i.e. blogs1.sun.com is placed on the odd side and blogs2.sun.com is placed on the even side. If we use more than two servers they are added to the respective side.

But the chain is only as strong as its weakest link: if I screw up when I update the puppet profile for our base server class, things will quickly go south.

No matter how carefully I test things before I commit my changes to the master mercurial repository and on to the puppetmaster (we only ran one per site before), there still is a chance things go boink! There are always some servers which were setup a few years ago, long before we started using puppet, that aren't installed and configured the way I expect, and when they are modified by puppet - they break!

So it doesn't matter that we are running multiple systems, they all get changed by puppet within 30 minutes.

To work around this problem I've set up two puppetmasters, and they serve the corresponding side (odd or even). This lets me push changes to the one side first, let it stew for a while, before I push it to the other side.

I'm back configuring Yubikeys but this time on Solaris 10 as it is what the majority of our servers run.

Here are are the steps required to get it working on Solaris 10 update 6:

1. Install curl
   pkgadd SFWcurl
2. Configure libyubico-client
   configure CPPFLAGS=-I/opt/sfw/include CFLAGS-std=c99 --prefix=/usr
3. Compile and install
   gmake install
4. Configure pam_yubico
   configure --prefix=/usr --without-ldap
5. Compile and install
   gmake install
6. Setup a user to key mapping file (e.g. /etc/yubikeys)
   

   martin:ulbtvceblvrb

7. Configure /etc/pam.conf
   

   other   auth requisite          pam_authtok_get.so.1
   other   auth required           pam_unix_cred.so.1
   other   auth required           pam_unix_auth.so.1
   other   auth required           pam_yubico.so id=16 authfile=/etc/yubikeys ignorepass

Then a ssh login will look like this:

martin@workstation$ ssh server
Password: 
Yubikey for `martin': 
martin@server$ 

You might have noticed the ignorepass option which I have added, this is to prevent pam_yubico from trying to (re)use the password I typed, nd instead force pam_yubico to prompt me for it. I have sent Simon the diff so he can add it to the next release.

I'm running puppet on the production servers I manage at Sun, and for Solaris 10 I've had to compile Ruby and create my own package (for easy distribution). I've also created my own puppet and facter packages, as I didn't want to setup rubygems.

Now on OpenSolaris this is much easier, as you can just run:

# pkg install -q SUNWruby18
# gem install -y puppet
Bulk updating Gem source index for: http://gems.rubyforge.org
Successfully installed puppet-0.24.7
Successfully installed facter-1.5.4
Installing ri documentation for puppet-0.24.7...
Installing RDoc documentation for puppet-0.24.7...

I've been looking at different solutions for two-factor authentication (as in something you have) to use as a backup to what Sun IT provides us. Since we run two data centers outside of Sun, and require two-factor authentication to log on to all our external servers, we are often prevented from logging on as the network path back to the Sun IT verification servers is down. So we need a backup solution that allows us to do the verification in our data center when the network is down.

The top contender for this is Yubico's yubikey which I think is a very cool device. And the best part of it, is that all software needed to do the verification is open sourced!

I've compiled libykclient.so and pam_yubico.so on OpenSolaris with the help of Simon as we had to make some minor adjustments to get it compiled on Solaris.

I've made some additional minor modifications to pam_yubico.so to let me use it for two-factor authentication (I'll post the diffs later).

This is how the authentication looks now:

martin@mbp$ ssh puppet-tst2
Password: my normal UNIX passphrase
Yubikey: the output from the yubikey
martin@puppet-tst2 $ 

I'm very pleased with the results of my tests so far, and if you are looking at a two-factor authentication, buy a few of them and git it a try...

As I wrote before, I've written the audit chapter for an upcoming Solaris Security book. The chapter is now available on Safari Rough Cuts and feedback is very welcome...

For the last few months I've been spending my evenings tapping away on the keyboard - but not producing code or managing Solaris servers like I usually do. I've been writing two chapters for an upcoming Solaris security book! It has been fun, but it has also been hard - not hard because I didn't know what to write, but hard to constrain myself from wanting to include too much.

The book is not intended to cover every nitty gritty detail of every security feature in Solaris - that would make it a real brick of a book! So I've had to think hard about what to include, and the level of detail of the included parts.

Parts of the book is already available on Safari Rough Cuts for review before we publish. Please leave comments about on the Safari site so that nothing gets lost.

The chapter about File System Security is mine, and I've also authored the chapter about auditing (not very surprising), though it hasn't been processed for publication yet, but when it is - I'll post a blog entry with a link to it.

Yesterday we started using an Alteon VIP to load balance SMTP traffic to our two mail servers, and everything was fine and dandy, but when I took a look in /var/log/syslog I found loads of entries like this:

Dec 11 18:17:14 prod-git1 sendmail[20899]: [ID 801593 mail.info] j93FHDNX020899: alteon1.sun.com [192.168.10.1]
did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA

The Alteon health check connects and then just issue a QUIT which sendmail finds suspicious, and hence feels obliged to let me know about it. This becomes very annoying when you have two Alteons doing the check every other second!

After scratching my head for a while and searching for a solution, I came across this patch to sendmail, which lets you select systems which shouldn't generate the above log entry. The only caveat was that I'd have to build my own sendmail, and I really don't want to roll my own stuff as it require more job to support, so I continued to look for a another solution.

I finally figured out (after reading the sendmail sourcode) that if I in /etc/mail/sendmail.cf set

O PrivacyOption=authwarnings,needexpnhelo,needvrfyhelo

sendmail would be quiet if the Alteon changed the health check to doing the equivalent of this:

mconnect localhost
connecting to host localhost (127.0.0.1), port 25
connection open
220 prod-git1.sun.com ESMTP Sendmail 8.13.8+Sun/8.13.8; Thu, 11 Dec 2008 13:58:48 +0100 (CET)
VRFY root
503 5.0.0 I demand that you introduce yourself first
QUIT
221 2.0.0 prod-git1.sun.com closing connection

So we changed the health check from being smtp to a custom script (note that you need the double backslashes):

open 25,tcp
expect "ESMTP"
send "VRFY root\\n"
expect "503"
send "QUIT\\n"
expect "221"
close

And after pushing this change out, sendmail stopped filling the log with messages I don't want to see.

I just read an article which scared the hell out of me! There is an EU proposal to require a government controlled registry of blogs. No more anonymous blogging!

"there is a need to clarify their status, and to create legal safeguards for use in the event of lawsuits as well as to establish a right to reply" in other words each blog needs to have an publisher, just like a newspaper, and this will require you to register with the local authorities. No more whistle blowing through blogs...

I wonder how they plan to enforce this? Especially in a case like mine where the server resides outside the EU.

I just read a good blog entry about the decay of the Swedish model which touched on many subjects discussed during dinner today.

People here (in Sweden) seem to think the someone else will take care of it. It is not my problem, but someone ought to do something about it. Why isn't the government doing anything about it?

Why not do it yourself?

I've come a fair bit in my puppet testing now, but one thing I lack is a user_attr type. I.e. a way to update the /etc/user_attr file using puppet.

This is what I have in mind for the syntax:

user_attr { "martin":
    type => normal,
    roles => [
        "root",
        "admin"
    ],
    profiles => "Zone Management",
    auths => [
        "solaris.mail.mailq",
        "solaris.system.shutdown"
    ]
}

One thing I haven't figured out yet is how if the definitions should be absolute, i.e. if the entry must be exactly like the definition, or if it is enough that the listed values are present. In the above example, should the role list be exactly root,admin or should it just make sure that those two roles are in the list and you can have the role audit too. Perhaps it would be good to be able to use the absent/present syntax on individual items?

I haven't decided if I'm going to manage the other user attributes too, e.g. project, defaultpriv, limitpriv and lock_after_retries. I will probably leave that for a later release...

[Technorati Tags: OpenSolaris Solaris Security Puppet ]

I've set up a puppet environment which uses mercurial to store the configuration and manifests. Now I'm trying to build an environment to be able to test changes before I commit them to the repository, and they propagate to all our 400 servers - but I encountered a problem.

You can use a separate configuration directory with the --confdir option for both puppetd and puppetmasterd, and run everything on localhost, but the problem is the source parameter

file { "/etc/profile":
    owner => root,
    group => root,
    mode => 644,
    source => "puppet://server/base/profile"
}

The above source parameter contains the hostname, so when I want to test it on my local mercurial repository, it still connects to the server instead of localhost when it fetches the files.

Luckily there is a solution! If you leave out the server part, puppetd will insert the name of the server it is connecting to.

I'm looking for ways to better manage our servers, and right now I'm playing with puppet.

I immediately ran in to a problem: it picked the wrong domain name. Internally at Sun we use NIS (yes, I know it is insecure and sucks in almost all aspects, but I'm not in position to change it - and believe me I have tried) and our NIS domain name doesn't match the DNS domain name.

This is something puppet (facter to be exact) doesn't figure out, at least not on Solaris. Instead of picking the correct fqdn for a host, e.g. puppetd.sfbay.sun.com, it picks puppetd.mpklab.sfbay.sun.com, since that is what the domainname command returns.

They tried to fix this, but unfortunately it doesn't work for Solaris, as it relies on the dnsdomainname which we don't have.

I've worked around it by creating my own /usr/bin/dnsdomainname which gets called before domainname.

#!/bin/sh
DOMAIN="`/usr/bin/domainname 2> /dev/null`"
if [ ! -z "$DOMAIN" ]; then
    echo $DOMAIN | sed 's/^[^.]*.//'
fi

So now I can continue to test my puppet configurations...

I'm headed to California and Menlo Park on Friday, and my wife has as usual given me a shopping list :)

With the dollar as lows as it is, I'm going to do some shopping myself. I'm going to buy a Time Capsule, not that I really need an extra 1 TB disk, but the rest of my family (whom I've converted to Mac) never remember to turn on the external disk I've attached to their computer - so Time Machine is useless!

And for myself, I'm going to get two 1 TB disks for my Drobo, which is 97% full at the moment. I'm squirreling away too much, but it is hard to throw away stuff... I've even got things stashed away on other external disks, but that data isn't mirrored which I don't like.

Since I got a digital video camera, I never seem to have enough disk space. I can't wait until our house is built and I can set up my U40 as a file server - 8 * 1 TB should last at least until the end of 2008 ;)

I've checked up on how my friends are progressing with the AuditAnalyzer and they have gotten quite far!

I've played with some pre-alpha stuff off and on, and the main problem have been importing audit data into the database - it has been too slow. It has managed to import about 150 records/second, which may sound much but if you are like me and get audit trails from 300+ systems, it is not enough to keep up with the stream of inbound records.

Luckily they worked on the import speed now, and have two possible solutions. One yields around 1500 records/second and the other a whopping 4500 records/second!

I can't wait until they have a new version available for me to try out :)

[Technorati Tags: Solaris ] Security ]

Out esteemed director has pushed us too far too long - he requires us to rack 'em and stack 'em all day long, and after the last spree of installing alpha hardware he got from engineering (the new 4 way, 16-core Rock based systems, code name lurad) for the www.sun.com cluster we now have such a big mess in our server room that I thought I'd share it with you:

Picture by: VespaGT

We have added 72 of these little monsters since the beginning of last week and haven't had time to clean up the cables - so now it is time to bring out the dymo and start labeling...

[Technorati Tags: Sun Solaris ]