On some systems where we are really paranoid, we restrict who can execute most setuid and setgid binaries just in case of a zero day exploit. I.e. we only let the administrators can run those commands.

Previously if you wanted to create this kind of restriction, you had to first chmod 4750 the files, and then grant all but the user(s) you wanted to restrict to a group and chgrp the binaries. This was very messy, and you could not have two different sets of restrictions.

Note: if you change file owner, group or permissions for files in a package, you must use installf to update the software installation database.

# chmod 4750 /usr/bin/su
# chgrp sysadmin /usr/bin/su
# installf SUNWcsu /usr/bin/su 4750 root sysadmin

Then came UFS ACLs. It allowed you to add multiple ACLs to the same file, so now you could have multiple sets of files with execute restrictions, but you still had to chmod 4750 all involved files.

Now that we have ZFS it is possible to use the ZFS ACLs to revoke permissions, so if the user danny should not be able to execute /usr/bin/su you can just add an ACL to remove the execute permission for that user.

# chmod A+user:danny:execute:deny /usr/bin/su

Now when danny tries to execute su it'll look like this:

$ ls -l /usr/bin/su
-r-sr-xr-x+  1 root     sys        34624 Feb 26  2007 /usr/bin/su
$ su -
bash: su: Permission denied

As with UFS ACLs the way to spot that a file has an ACL is the + sign at the end of the permissions when you execute ls -l. If you want to see the full ACL use this

$ ls -v /usr/bin/su
-r-sr-xr-x+  1 root     sys        34624 Feb 26  2007 /usr/bin/su
     0:user:danny:execute:deny
     1:owner@:write_data/append_data:deny
     2:owner@:read_data/write_xattr/execute/write_attributes/write_acl
         /write_owner:allow
     3:group@:write_data/append_data:deny
     4:group@:read_data/execute:allow
     5:everyone@:write_data/append_data/write_xattr/write_attributes
         /write_acl/write_owner:deny
     6:everyone@:read_data/read_xattr/execute/read_attributes/read_acl
         /synchronize:allow

[Technorati Tags: OpenSolaris Security ]

While talking to Tomas about measuring the impact of auditing, he gave me a nice call flow tree which I thought I'd share.

This is how it syscall auditing looks (for intel):

  + dosyscall()
    |
    + syscall_entry()
    | |
    | + pre_syscall() (if t_pre_sys set)
    |   |
    |   + audit_start() (if audit_active set)
    |     |
    |     + au_init
    |     | |
    |     | + aui_*()
    |     + auditme() (to audit or not to audit)
    |     + au_start
    |       |
    |       + aus_*()
    |
   ...
    |
    |
    + syscall_exit()
    | |
    | + post_syscall()
    |   |
    |   + audit_finish() (if audit_active set)
    |     |
    |     + au_finish
    |       |
    |       + auf_*()
   ...

Update: the ASCII graph was hand crafted

[Technorati Tags: OpenSolaris Security ]

This has bothered me since I first ran bsmconv on Solaris 2.3. People are using performance as a reason not to enable auditing on their systems, and it has been hard to convince them as there are no hard metrics to look at.

With Solaris 10 I was given the solution to the problem - DTrace. Since the day I first read about it I thought that I'd write this script, but since it never have been something I've truly needed in my job, until now, I've put it off.

Yesterday I started to write on a script, and it was easier than I thought it would, but on the other hand, I know much more about DTrace and the gory details of the auditing magic in the kernel than I did when first looked at this about two years ago.

Once I've fine-tuned the script I'll publish it, but until I've worked out all the quirks I'll just post some results for the execve system call:
the average execution time for the 1066 execve syscalls I measured was 839 µs, of those calls 419 were audited, and the average time spent in the audit_start() function in the kernel was 1.8 µs, and the average time spent in the audit_finish() function in the kernel was 13.5 µs.

The numbers above comes from using the vtimestamp to measure the time spent in the audit_start() and audit_finish() functions in the c2audit kernel module. As you can see only about 1.8% of the time is spent in the auditing code, not very much!

Two things to note:
First the execve syscall is one of the slower, so things will probably look different when I look at some of the faster ones.
Second this is not the whole truth. When the c2audit kernel module is loaded, it sets t_pre_sys and and t_post_sys on klwp_t and therefore pre_syscall() and post_syscall() are executed in addition to audit_start() and audit_finish().

The above number does not include the other code in pre_syscall() and post_syscall() in the calculation, which I need to include if I am going to measure the full impact of enabling auditing.

I just read a news article in a Swedish newspaper about CIA and the Vatican modifying information in Wikipedia. It was discovered when the folks at Wikipedia analyzed who has modified what, and discovered some interesting things.

Someone from the Vatican have modified the entry on Gerry Adams (the leader of Sinn Fein) and removed a link relating to his connection to a double homicide in 1971.

The person from CIA had only made minor adjustments, like adding the comment "Wahhhhhhh!" to the page about Mahmoud Ahmadinejad (the president of Iran).

So why am I amused by this? Because I'm an audit freak, and this is an excellent example of why you want auditing. You catch people with their hand in the cookie jar :)

Update: I've found a BBC news article about this too, so it will be easier for you read about this.

[Technorati Tags: Security ]

It has been hard to work today. Yesterday I had oral surgery because my jaw bone grew out of the gum! It started out as a sore spot in my mouth, which grew to a bulge, and a week ago the gum ruptured and the jaw bone shone like a bright white spot...

It turned out that when I pulled a molar many years ago the Swedish dentist I went to didn't do a very good job, so yesterday they had to "fix" it. The fix was to cut open my gum and using a bone file to get rid of the outgrowth. When he was filing away on my jaw the sound resonated in my skull - not a very nice sound!

The Brazilian dentist who performed the surgery was excellent! He is a 4th generation dentist working with his father, which is very uncommon in Sweden. They are both exceptionally good and if you ever need to fix your teeth while in Rio drop me a line and I'll give you their name.

Eight stitches later we were sent home with a list of medications and procedures to follow, my wife had to pull a tooth - she have (had) 34 teeth. The one that got pulled was some strange pre-historical tooth appearing behind the molars. The only good thing about all this was that we were told to eat ice cream, lots of ice cream!

What amazed me the most with the dentist was that he gave me the number to his home and to his mobile and told me to call if there was any problem or if I had any questions! That never happens in Sweden. He even called in the evening to check up on us. Talk about good service!

So today I've had a throbbing pain in my jaw which no amount of ice cream could get rid of. It has been hard concentrating on the work at hand: writing a set of script to run bart on all our Solaris 10 systems and generating alerts when there is a discrepancy. Not something which is very hard, but today I can't focus very well...

This weekend we went to Braz which is a typical Paulista (someone from São Paulo) pizza restaurant with two friends. As we went there on a Friday evening around 2000 (8 pm for the 24-hour challenged people) the place was full and we had to wait almost an hour to get a table, but it was a beautiful evening so we didn't mind waiting outside, and you could order drinks and appetizers so we didn't starve.

Inside it was a bustle, over 50 tables with guests and waiters everywhere catering to everyones needs. Lots of well dressed and elegant people, and since it is winter now you see men in suits, which are painful to wear when it is over 35C in the evenings during the summer. We even spotted Camila Pitanga an actress from the most popular novela (Brazilian TV show) Paraíso Tropical.

You have no idea how big the novelas are here until you have seen and heard the Brazilians engage in lively discussions about the characters and the plots. They talk about them as if they lived in their building. Even the fashion follows the novelas! Bebel, Camila's character, wears clothes which usually are quite revealing, and when we are shopping you can see that they sell very similar clothes in the stores.

Back to the dinner: the pizzas were fantastic. A crisp and tasty crust, with fresh and savory toppings. We had a nice bottle of Los Vascos, produced in Chile by an estate owned by Château Lafite-Rothschild. The combination was so good that I could not help ordering a second pizza, even though I was so full I could not eat more than two pieces. Luckily you could get the remaining part pra viagem - to go :)

I hope we can make a second visit there before we leave in September!

A week ago my wife Paula and I went to an 80s part in Catete (an old part of Rio de Janeiro) together with some of her friends. At the beginning of the evening the DJ played US and UK music (and even Roxette who are Swedish), but at a certain point he switched to Brazilian music from the 80s - and then I really feel like a gringo :)

As soon as the intro was playing people would scream and the dance floor got flooded, while I would try to recognize the song (Paula has subjected me to a lot of local music), and then they started singing along. The music he played has no memories attached to them for me, while they got a misty look in their eyes of teenage flashbacks - sort of how I would feel if he had played Tainted Love by Soft Cell.

The last evenings I've been reading a book I gave Paula some time ago, called Almanaque Anos 80 It even has its own community on Orkut. The book has a chapter on music, so when I read it last night I decided to listen to some of the golden oldies today.

At the moment I'm blasting the office with 80s music:

• Close to you
  The Cure
• Build
  The Housemartins
• Broken Wings
  Mr. Mister
• Oh l'amour
  Erasure
• Alive and Kicking
  Simple Minds
• Kiss
  Prince
• 99 Luftballons
  Nena

While watching the old music videos above, I made an interesting discovery. Nena, who I thought was very beautiful when I was a teenager, looks very much like my wife! I'll let you be the judge of the likeness:

Those of you who have to deal with audit trails from busy systems know that they can get really big, and when you need to store them for a couple of years they consume a considerable amount of disk.

To minimize the disk usage that you can compress the audit trails, and it works really well (I've adjusted the output to right-align the size):

root@warlord# ls -l
total 19447638
-rw-r--r--   1 root     other    1353214369 Aug  9 09:27 20070728065900.20070730163539.warlord
-rw-------   1 root     other      62209268 Aug  7 08:25 20070728065900.20070730163539.warlord.gz
-rw-r--r--   1 root     other    1965073391 Aug  7 09:35 20070728065900.20070730163539.warlord.txt
-rw-r--r--   1 root     other      71460194 Aug  7 09:35 20070728065900.20070730163539.warlord.txt.gz

As you can see the compression ratio is really good (over 90%), but one problem still remains: if you need to work with the files you have to uncompress them before you can run your scripts to go an find who edited /etc/passwd. Uncompressing one file doesn't take that long, but when you don't know exactly in which file the audit records you are looking for are, things start to take time.

Enter ZFS: with on-the-fly disk compression it is the perfect file system to store audit trails. First of all you have to enable compression:

root@warlord# zfs set compression on pool/audit

That only makes future writes compressed, so the files you already have needs to be rewritten to be compressed. After having done that it is time to look at the compression ratio:

root@warlord# ls -l
total 19447638
-rw-r--r--   1 root     other    1353214369 Aug  9 09:27 20070728065900.20070730163539.warlord
-rw-r--r--   1 root     other    1965073391 Aug  7 09:35 20070728065900.20070730163539.warlord.txt

Wait a minute! That didn't compress anything, or did it? ls -l shows the size of the uncompressed file, so you have to use du to see the compressed size:

root@warlord# du -k
554067  20070728065900.20070730163539.warlord
732399  20070728065900.20070730163539.warlord.txt

Much better! (Note that it displays the size in kilo bytes while ls -l displays it in bytes) But it still is not as good as when I ran gzip on them. Why? ZFS uses the LZJB compression algorithm which isn't as space effecient as the gzip algorithm, but it is much faster. If I had been running Nevada I could have used:

root@warlord# zfs set compression gzip pool/audit

And have gotten the same compression ratio as when I "hand compress" my audit trails. This thanks to Adam who integrated gzip support in ZFS in build 62 of Nevada.

[Technorati Tags: OpenSolaris ZFS Security ]

One of my friends is now a published author! Mags has written a book about all you need to know to get by as a foreigner in Slovakia. It deals with everything from the tourist basics to information which is invaluable if you are going to live and work there.

It is called The Foreigner's Guide to Living in Slovakia and is available on Amazon but if you want to buy it you should do it from the web site for the book. Amazon uses on-demand printing and the pictures doesn't look as nice as they do in the real thing.

And yes, I've actually read the book. I got an advance copy of it when I visited in May :)

Today I started to ponder a problem which I can't be alone to have encountered. When you administer over 300 systems and want to perform bulk operations over ssh, there are always one or two systems which are down or unreachable, so your nifty little scripts which log on to each system to install a package, apply a patch, change a configuration setting, tweak a variable or just pull statistics from the system will fail.

So I started toying with the idea of an ssh job queue which helps you keep track of bulk operations, so you can see the on which systems the operation has successfully completed. Once I started to try this out I figured that I can't be the first one to face this problem, so i thought I'd ask you for input.

How do you deal with this problem? And "pen and paper" isn't the answer I'm looking for :)

You might have noticed some strange files in /var/spool/cron/crontabs ending with .au. These are not µlaw audit files, but auxiliary audit files for crontab, which are created when auditing have been enabled and you edit your crontab entry.

# cd /var/spool/cron/crontabs
# ls -l
total 19
-rw-------   1 root     sys         1010 Feb 25 18:04 adm
-r--------   1 root     root        1371 Feb 25 18:06 lp
-rw-------   1 root     martin        38 Jun 21 00:20 martin
-r--------   1 root     martin        45 Jun 21 00:20 martin.au
-rw-------   1 root     sys         1401 Mar 13 04:28 root
-rw-------   1 root     sys         1128 Feb 25 18:09 sys

Looking closer at what is in my .au file we find the following:

# cat martin.au
300
0
0
7ff81600
4
1dad35c9 0 0 0
2441309132

This is quite cryptic, especially as it isn't documented anywhere but in the source! Using it you can discern what the above settings are.

The first number (300) is the audit id, i.e. my user id. The second and third rows are the pre-selection mask split up in two parts, first the audit on success and then audit on failure. The next three rows are the terminal id, starting with the port, address type and last the address. The port number (5f81600) is made up of two parts (major and minor) which are joined together. After that follows the address type (4) which represents IPv4, as defined in audit.h. Note that the address is made up of 4 numbers to fit IPv6 addresses, but since I logged from a system using IPv4 it is only the first part which is filled. There is a gotcha here, the number is written depends on the architecture, the example is from my X2200 M2, so the 1dad35c9 needs to be changed to network byte order to map correctly to an IP address. The last row is the session id (2441309132).

This file is created (and updated) when you edit crontab, which can cause a lot of confusion. The pre-selection mask used by cron is calculated by logically ORing the entry in the .au file with the user entry from audit_user and the global flags in audit_control. So if you reduce the auditing for a particular user in audit_user, you expect that the audit trail from the user's cron jobs would change too, but if the .au file have already been created the pre-selection masks are frozen.

To fix this you need to update the .au file too when you change the audit flags or edit the crontab so that the .au file gets rewritten.

[Technorati Tags: OpenSolaris Security ]

It has been a rainy Sunday, so my wife and I spent the day indoors. We ended up discussing the difference between growing up in Sweden and Brazil (during the 70s and 80s), especially children's shows on TV. It was a huge difference, until 1985 Brazil was a dictatorship influenced by USA, while Sweden was far out on the left side with the socialists and communists running the country (by mandate of the voters though).

When I watched TV in Sweden the only thing available was, either socialist propaganda for kids or educational shows, but every once in a blue moon something normal appeared. I recall preferring to play with LEGO over watching TV, as most of the shows didn't stimulate me, but since they only had one or two things for kids per day, it wasn't a too long break from my constructions. Below are some samples of things I watched as a kid.

An educational series about food and the body called Maten och Kroppen

I love the collar of his shirt and his sweater, soooo 70s!

Here is the opening of socialistic propaganda show called Vilse i pankakan, where one of the main characters Storpotäten (the big potato) clearly is a "capitalist pig". This show is said to have ruined a whole generation who all had nightmares about Storpotäten who would "own you" (put you in his pocket).

While I was bombarded with propaganda and educational programs to make me a good citizen, my wife in Brazil got to watch foreign series, like Land of the Lost or the Japanese the Spectre men, all dubbed in Portuguese. They also had a bunch of shows produced in Brazil like, Xuxa and Picapau amarelo, where the first one is and the second one. One thing that struck me when we watched clips of Xuxa on YouTube was that she was very scantily clad for a children's show, at least with my brain-washed socialist eyes.

The opening of Picapau amarelo

Xuxa - compare the clothes of the first clip and the second, when they tried to launch her show in the US.

While they got to watch tons of cartoons in Brazil, we got to watch animated clay figures from Czechoslovakia. There was only two times a year they showed "real" cartoons on TV in Sweden, during Christmas and during the summer vacation, but only once a day mind you. I guess that didn't want us to become Americanized!

The opening of Trazan Apanson the show that aired cartoons once every morning during the summer. It was the only thing that could get me up at 0900 am!

One of the TV series I clearly remember is Galaxer i mina braxer, sa Kapten Zoom which depicts our post-industrial society with phenomenon like trade, corporations, environmental pollution and democracy. It had an anti-capitalist message which became very clear in the last episode. Unfortunately it isn't available on YouTube.

One of my favorite shows were Sant och sånt, which actually was recorded in the city I grew up, but that was not the reason I liked it. They managed to combine science with humor to show how things work without taking the seriousness out of science and without making the jokes boring.

I think this show is one of the reasons I grew up to be an engineer...

As of this week I am no longer working in the Java SE Security Team. I got tired of fixing other peoples' mistakes - I have a hard enough time fixing my own! It was a great place to work at, lots of clever, nice and cool people, but my job wasn't all that I thought it would be...

So what do I do now? I'm the security geek for .Sun engineering (DSE - Dot Sun Engineering), the group who run and operate most of Sun's external systems - like this one (blogs.sun.com).

My main focus is Solaris security, but I will also help the development teams to review the application security. I'm responsible for the security of the Solaris servers which run all the external sites - so that'll keep me busy :)

This mean that I now have legitimate reasons to play with Solaris Auditing again, so I've already started to help Tomas to alfa test the Remote Audit Trail Storage project. DSE will be a good test bed to see how well the project scales.

[Technorati Tags: OpenSolaris Security ]