Alan Hargreaves' Weblog

html enabled in comments

Just to let folks know, that HTML has been enabled by default for comments on this site now; so hopefully we may get a few more people responding to us.

Posted at 05:52PM Jul 29, 2004 by Alan Hargreaves in General  |  Comments[1]

New On#Sun Article on Solaris on x86

My latest just made it to the web copy of the magazine. I've updated the list to the right. It's available here as well.

Posted at 02:59PM Jul 29, 2004 by Alan Hargreaves in Solaris  | 

Fixing the car

I've been away from the blog for a bit due to various things. Mainly I've had to deal with a sinus type bug that's been running around. Hopefully that's now in the past. I've also had a lot of work related stuff on my plate for those days in which I've been in.

Didn't get to see Jake play soccer on the weekend as my father and I had to do some work on my wife's car while she took him to soccer with his nan. I didn't get to hear very much about what happened except that I think they lost and Jake saying "I didn't get anything".

Anyway, the car ...

The right hand side front wing had ben sideswiped. As a result the bumper was sitting low and the indicator housing was gone. We were also looking at replacing the radiator. Thought we had all of the parts from a local wreckers. Not quite.

1. Apparantly on leaving the wreckers my wife left the replacement indicator on the sales bench.
2. On removing the current radiator we realised that the replacement was too big.

Oh well, we still had work we could do while they were out.

1. Removed and replaced the apron (the part between the bumper and the grill).
2. After a whole lot of "how the hell does this hold together" we managed to straighten out the bumper.

When everyone else got home we went for a quick trip to the wreckers to pick up the indicator housing and take the radiator back. We took the old one along as well so they could measure it and see if they had another (they didn't). They are ordering one in for us.

Anyway, we managed to pull out the wing sufficiently for us to get the new indicators in and (wonder of wonders) it all worked first go. We put the old radiator back in after giving it a low pressure reverse flush (stand it upside down and drop the hose into it for a while).

We managed to be done just in time for a lovely lunch.

Posted at 01:05PM Jul 29, 2004 by Alan Hargreaves in General  | 

Gary's Solaris Quiz

Gary Little has posted a a question on his blog with a prize of a blueprints cd. It's not a difficult one and should not take a lot of thinking to work it out.

Good luck.

Posted at 07:22PM Jul 21, 2004 by Alan Hargreaves in Solaris  | 

Jake's Soccer on the weekend

OK I'm late putting this up (if anyone is reading it at all ;-).

On Saturday his team (Tuggerah United - Scorpions) took on Wyong. Unfortunately for Wyong, most of their under sixes didn't show up so they had to field the majority of the under fives who had just come off.

The result was predictable.

11-0.

Wyong changed keepers during the first half and I must say that this little guy was really very good. He stopped it being a much larger score, and his kicks out to his players in the field were great.

While Jake didn't score any of these, he was instrumental in setting up quite a few of them with some great passing. Indeed, there were a few very good team goals.

The coach commented that we have the seeds of a really good team here if we can keep them together for next year (there are only around five or six games left in this season).

Posted at 11:46AM Jul 20, 2004 by Alan Hargreaves in Kids  | 

Playing with dtrace in user space

In chapter 30 of the dtrace reference guid we look at user level tracing. There is only one example there, so I thought I'd write a few more, as I see this as being an extremely useful tool in user-space as well as the obvious kernel use.

Now, in the current build of Solaris Express, we cannot directly run a process from the dtrace command line, so we need to do it with truss. The sample command I will be using is a simple "ls -ls". You'll probably need two terminal sessions to do this. One to deal with the command and the other the dtrace script.

On running the truss command you'll get something like

$ truss -f -t\!all -U a.out:main ls -ls
3470/1@1:       -> main(0x2, 0xffbfe784, 0xffbfe790, 0x26000)

To restart the command again after you have the dtrace running, simply use prun.

$ prun 3470

For the dtrace commands that use aggregations you need to ctrl-c the command once the process has finished.

OK, to some commands.

1. Let's look how often we call each function within the lifetime of this process.

# dtrace -n 'pid3470:::entry { @funcs[probefunc] = count(); } END { printa(@funcs); }'
dtrace: description 'pid3470:::entry ' matched 2921 probes

CPU     ID                    FUNCTION:NAME
  2      2                             :END 
  pthread_rwlock_unlock                                             1
  _fflush_u                                                         1
  rwlock_lock                                                       1
  ...
  iflush_range                                                     90
  callable                                                        136
  elf_find_sym                                                    139
  _ti_bind_clear                                                  140
  rt_bind_clear                                                   140
  strcmp                                                          146

This shows us the "hot" functions in our code.

2. We might also be interested in how long we spend in these functions per call.

# dtrace -n 'BEGIN { depth = 1; } pid3497:::entry { self->start[depth++] = vtimestamp; } \
  pid3497:::return { @funcs[probefunc] = quantize(vtimestamp - self->start[depth-1]); \
  self->depth[depth--] = 0;}END { printa(@funcs);}'
dtrace: description 'BEGIN ' matched 5816 probes

This gives us some histograms of how long we spend in various functions.

e.g.

  strcmp                                            
           value  ------------- Distribution ------------- count    
            1024 |                                         0        
            2048 |@@@@@@@@@@@@@                            47       
            4096 |@@@@@@@@@@@@@@@@@@@@@@@@@@@              98       
            8192 |                                         0        
           16384 |                                         0        
           32768 |                                         1        
           65536 |                                         0        

We could just have easily specified the functions we were interested in.

3. OK, suppose for arguments sake we were interested in strcmp (since I listed it already). How about we look at the codepath that we take through it?

# dtrace -n 'pid3486::strcmp: { trace(probename);}'
dtrace: description 'pid3511::strcmp: ' matched 256 probes

We get a really long list as we are looking at all calls to it (and we have a few). This may not be very useful. The last call looks like:

  2  47463                     strcmp:entry   entry                            
  2  47464                         strcmp:0   0                                
  2  47465                         strcmp:4   4                                
  2  47466                         strcmp:8   8                                
  2  47508                        strcmp:b0   b0                               
  2  47509                        strcmp:b4   b4                               
  2  47462                    strcmp:return   return

This list is actually a call flow through strcmp for all calls to that function. Looking at the full list could give us ideas about where we might optimise. This could be even more useful if we knew where the hot instructions within this code flow were.

4. We can do this by turning on probes for all instructions within strcmp (like above), but aggregating on the probename, which will be the function offset.

# dtrace -n 'pid3517::strcmp: { @hot[probename] = count();} END {printa(@hot);}'
dtrace: description 'pid3517::strcmp: ' matched 257 probes

The end of this list looks like

  b0                                                               74
  b4                                                               74
  c                                                                77
  18                                                               77
  14                                                               77
  10                                                               77
  2c                                                               82
  24                                                               82
  28                                                               82
  20                                                               82
  30                                                               82
  0                                                               146
  entry                                                           146
  8                                                               146
  return                                                          146
  4                                                               146

We can ignore the entry and returns as we already account for those. We can tell from them (however) that in this run we called strcmp 146 times.

4. Anyway, if we run up mdb on /lib/libc.so.1 we can find out what these instructions are.

# mdb /lib/libc.so.1 -
Loading modules: [ libc.so.1 ]
> strcmp::dis
strcmp:                         subcc     %o0, %o1, %o2
strcmp+4:                       be        +0xac         
strcmp+8:                       sethi     %hi(0x1010000), %o5
...

OK we would expect to execute these on each call, so what about the ones we hit 82 times?

strcmp+0x20:                    ldub      [%o1 + %o2], %o0
strcmp+0x24:                    ldub      [%o1], %g1
strcmp+0x28:                    subcc     %o0, %g1, %o0
strcmp+0x2c:                    bne       +0x1c4        
strcmp+0x30:                    addcc     %o0, %g1, %g0

strcmp is probably not the best example to use as it is a call that has already been very highly optimised, but I hope you get the idea. This is going to be very useful in finding bottlenecks and suboptimal codepaths in userspace.

Posted at 08:01PM Jul 14, 2004 by Alan Hargreaves in Solaris Express  |  Comments[2]

Solaris Express 7/04 out shortly

Just received the notification that the July release of Solaris Express should be available for download by July 14 and will be s10_60. When I say July 14, I do mean July 14 in the US, out here in Asia/Pacific the vagaries of timezones mean that it should be available by July 15.

Some of the new features for this release are:

• Streams Control Transmission Protocol
• New Functionality Introduced in Solaris Zones Software Partitioning Technology
• New Solaris Project and Resource Management Command Functionality
• New Functions for Converting Strings
• Java Support for pstack Command
• New Solaris Unicode locales
• USB End-User Device Support Enhancements (revised)

As has been the case for a while now, all documentation is available at http://docs.sun.com with no password.

Please also remember that the distribution is now three cd images, (1of3, 2of3 and 3of3) and that you will need them all to do an installation. The old CD0 is no more.

Alan Coopersmith also lists some things that are in the release that he has an interest in.

Update

From comments I have gotten back from the Solaris on x86 mailing list, looks like it is available NOW. Certainly beats last month when it became available after the date I said.

Posted at 02:50PM Jul 13, 2004 by Alan Hargreaves in Solaris Express  | 

More folks finding out about Dtrace

It's great that we are all seeing folks get just what is possible with Dtrace. Bryan emailed me in response to my last blog entry. He is also seeing this. He sent me a few urls of folks who are writing about it. One in particular stands out.

Daniel Berrange has written a well thought out entry on what he sees Dtrace is, how it compares with some other tools, and expresses a desire that the folks coding Linux take note of the functionality.

I for one hope that Linux community (& vendors supporting it) realize the value of a polished tool like DTrace and take prompt steps to close the gap to Solaris.

He also lists some resources for finding out about Dtrace.

This type of posting is great. As was noted in my previous entry, the detractors appear to be those who have not tried it. Get out and have a good look at it before you start with the "If it's in Solaris, the linux stuff must be better". Sure, Linux has done some great stuff; but it would be arrogant to believe that it is the only Operating Environment that is showing innovation and great advances.

As someone who has seen some of what is yet to come into Solaris Express, there is still a lot of great stuff coming.

Posted at 07:11PM Jul 10, 2004 by Alan Hargreaves in Solaris Express  | 

Slashdot points to The Register's Dtrace Article, and it's generally favourable

and I'm moderately impressed by the mostly informed comment associated with the article and the comments.

As one reader notes

What strikes me most about the commentary here is that the raves are coming from people who have actually used it, not from Sun (or not *only* from Sun; some people there seem justifiably proud of their work:-). The snarky comments are exclusively from people who haven't used DTrace ("gee, sounds like ____; what's new about that?"), and are being soundly rebutted by those who have.

Unfortunately, as it was posted anonymously, it started at mod level 0 and no-one has modded it up.

I think that this person has hit the nail on the head. Pretty much all of the disparaging remarks are coming from those who have not tried it. For goodness sakes folks, Solaris Express is a free download for non-commercial use. As noted in the slashdot comments and many other places, real admins are starting to use this for real work.

There was another analogy made, which is close too.

That's kind of like saying perl is an all round text processing tool, then asking why using perl is better than using cut, sort, and tr.

You can do a lot with cut sort and tr. Often they're all you need, but perl lets you solve problems those three tools can't even address.

I also saw the question asked a lot "Isn't this a lot like the functionality that X provides in Y?". It was comforting to see this question almost invariably answered with something along the lines of 'To see how X compares with Dtrace have a read of the Usenix paper that was presented in Boston' this year'. I can certainly recommend the paper. It will address a lot of the questions about Dtrace that people have. It will also fill in the blanks for those who do not understand just what we are doing with Dtrace.

Posted at 11:25AM Jul 10, 2004 by Alan Hargreaves in Solaris Express  | 

Man-Ching's ksh scripts

Interesting scripts, but I have another solution to the splitting problem using IFS.

I offer the following, which should work in vanilla bourne shell, with the using the same delimeters as Man-Ching used.

pl.sh

#!/bin/sh

line="$@"       # Save the argument list
IFS="|"         # Make '|' the field seperator

set -- $line    # reset the argument list

for column      # loop through $*
do      echo $column
done

Giving the same result

./pl.sh `head -1 /etc/passwd | sed 's/:/|/g'`
root
x
0
1
Super-User
/
/sbin/sh

Indeed you could do the whole part in grave accents in a single sed command.

./pl.sh `sed   -e 's/:/|/g' -e 1q /etc/passwd`
root
x
0
1
Super-User
/
/sbin/sh

Which actually goes to demonstrate a bit of unix philosopy. That is, there are many ways to achieve the same end; none is necesarily the best, or more simply, there is more than one right way to do anything.

Posted at 06:42PM Jul 09, 2004 by Alan Hargreaves in Solaris  | 

Great Dtrace article on The Register

Fired up The Register this morning to find the first article that I come across is an interview with Bryan and Adam titled Sun delivers Unix shocker with DTrace

There are also a real uses in the article from Brendon Gregg and Aeysis' Jenson.

Good interview guys!

Posted at 08:59AM Jul 09, 2004 by Alan Hargreaves in Solaris Express  | 

Jacob's Big Soccer game today

Today Jake's team (the Tuggerah Scorpions) took on Toukley.

Jake got to try out being goalkeeper for the first half. It as a new experience for him and he had some trouble drop kicking the ball, but by the end of the half he was getting the hang of being goalkeeper.

In the second half he got to play out on the field. We've been teaching him all about getting in and having a go; and if you lose the ball, get in and have another go. He got in and did that today and really picked up on his attacking. Indeed he was constantly beating players who were physically much bigger than him. Clever beats bigger.

His coach was impressed with how much he has improved in attack since he started and he got the encouragement award for exactly that.

Jake's friend Callum got the player of the match for "Putting his body on the line to protect the goals". Callum ended up on the ground between the ball and the goals at one point in defence on top of his usual great attacking game.

This team is really coming on well. One thing they do have to do is to remember to listen to what their coach is telling them as they do tend to go off on their own a bit.

They're into school holidays this week so there is no game next week; but they'll be back the following week.

This team is starting to show a lot of team spirit!

Oh yes, the result? ... 0-0 - a scoreless draw.

Posted at 12:23PM Jul 03, 2004 by Alan Hargreaves in Kids  | 

School Holidays and what's going on with the kids

My goodness the term has flown. I didn't realise until this morning that my 5 year old twins start their mid year school holidays (two weeks) on Monday. I Managed to get agreement¹ from my manager to take some time off so I am at home for the time that they are not visiting their grandparents.

Anyway, I'm really looking forward to being able to spend some time with them. We've got a great little park with climbing frames, swings and somewhere to kick a ball around so we should have a good time.

Jake has to be at Soccer early tomorrow for his team photo. He just lost his second front tooth the other night, so this should be a good one :-). He is currently having awful trouble pronouncing some words. It looks like Lucy may not be far behind him in the tooth department either.

Posted at 09:10PM Jul 02, 2004 by Alan Hargreaves in Kids  | 

Open Sourcing Solaris in the news during June

As well as the discussions going on in Andy Tucker's blog, the following is a list of news items that have shown up with regard to Sun Open Sourcing Solaris.

June 29

Sources: Sun Plans to Open Nearly All of Solaris Source Code

June 14

CRN - Sun To Open Solaris?

eWeek - Sun Should Expedite an Open-Source Solaris

June 9

The Register - SCO trumps Sun's open source Solaris bid

June 7

ComputerWeekly.comLukewarm response for 'open-source' Solaris

June 3

ComputerWire - Sun Execs Spill the Beans on Open Source Solaris

eWeek - Sun's Long Road to Open-Source Solaris

eWeek - Sun Working Out Open-Source Solaris Details

Web Host Industry News - Sun to Make Solaris Open Source

p2pnet.net - Will Sun's Solaris go open source?

June 2

InternetNews.com - Open Sourcing Ahead for Solaris

Posted at 08:52PM Jul 02, 2004 by Alan Hargreaves in Solaris  |