Interconnectedness of all things

Nant Peris Horseshoe according to Garmin

Friday Sep 11, 2009

Like this post? del.icio.us | furl | slashdot | technorati | digg

Nant Peris Horseshoe Round 2

Sunday Sep 06, 2009

The Nant Peris Horseshoe starts in Llanberis, takes in the summits of Elidir, Y Garn, Glyder Fawr, Lliwedd, Snowdon and Moel Cynghorion. It is considered by many runners to be the hardest fell race in the Welsh calender at 17.5 miles and 8500ft of ascent. If you think different, I would be interested to know what fell race is harder apart from maybe the Welsh 1000m race and the Tryfan Downhill Dash, do add a comment.

I was very pleased indeed to finish in 5 hours and 13 minutes, which was in the region of 1 hour 20 mins faster than last years epic. I had a target time of 5.30, so in general the race went well. The weather was no where near as hot and apart from the rock being slippery, it was really good running weather with a stiff wind and cloud on the tops, but not so much as to make it cold or navigation difficult. It is the 1st race I have run where felt I achieved my potential on the day. The ascents up Snowdon and Moel Cynghorion were very tough(be something wrong if they were not) and it was cheering for the 3 marshals on top of Snowdon to comment that I was in a much better state than last year.

Dilwen, if you read this, it is now one all for the only race that matters series, next year is the decider!

Mike Blake and friends did a excellent job of organization and marshaling getting the combination of freedom, safety and challenge just right. The contribution of the 3 sponsors(Vic Hotel in Llanberis, 1st Hydro and the Snowdon Mountain Railway) makes a huge difference to the race in terms of access to land in the Llanberis slate quarries to run over, getting the 5 gallons of water to the top of Snowdon on the train and the accommodation for the race HQ. This is one aspect that I was not aware of. It did make me and others chuckle that the prize for the 1st man(I think this is right) was a pair of tickets on the Snowdon Railway to the summit. If you can run that race in 3 hours 12, you do not need the train to get to the submit. Be a nice treat for his grandparents maybe.

Results will end up here and some photos of the race can be found here. This pic

is taken at about a mile or so from the start, hence the rather fresh look, this pic

is take about a mile from the end, hence the general focus on putting one foot in front of the other.

Next race is much closer to home (about a mile as the Kite flies) is the Pumlumon Challenge which is good value at 27 miles and 5500ft which is part of the Ultra Running Championships. 3 of the races in this series in a year is enough for me.

[3] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg

DTrace for breakfast

Friday Sep 04, 2009

The UK Corporate Solaris Users Group on Tuesday 29th September has a breakfast meeting where we are going to sprinkle a little DTrace on your cereal. DTrace was a requested topic at the last meeting, so in keeping with my preferred style of delivery, it will be demo only. Diagnosis and performance analysis is a full contact sport, so best to show it as it really is.

More details can be found are here

Like this post? del.icio.us | furl | slashdot | technorati | digg

Beautiful Days Festival No. 7

Monday Aug 31, 2009

Compared to the last 2 years of mud, it was a relief that the worst the weather did was give a serious threat of drizzle, but backed off before going through with it. The 2 previous years where we spent 3 nights sleeping in a VW Transporter van with 2 children under 5 who spent the day playing in the mud did test the resolve.

The Beautiful Days festival at Escot Park in Devon in now in its 7th year and it has taken me a week to get round to writing it up. The Guardian describe the festival as a family based folk punk hoedown with give a high level flavor. It is a middle sized festival at around 10,000 people with a mix of bands drawn from folk, punk, reggae and rock spanning the last 40 years of music, for example Hawkwind played the same stage and evening as The King Blues. With a couple of stages, you can't see every band and we also spent some time in the kids area. My highlights were

  • Hawkwind : 40 years and still delivering the nearest thing to space travel through the medium of guitar based rock. Great laser show and they really deliver a wall of sound. Probably my top act of the weekend aided only by the products of the Otter Brewery which may have put me in a minority.
  • The highlight in terms of exposure to new bands was The King Blues which was a folk punk mash up with some punchy lyrics.
  • An other highlight of the past for me was The Blockheads. I am proud that the 1st record I bought at the age of 10 was "What a waste" giving me some punk credentials 31 years later. Really nice balance between remembering Ian Dury and being a current band.
  • I have seen the Levellers so many times, that I can't judge if they were better or worse than previous times or other bands. I enjoyed it and playing the whole of "A Weapon called the Word" panned out well.
  • Its hard to say it, but I found The Pogues a partial disappointment. There is something special about Shane MacGowan which still comes out, but he came across as being in a bit of a state(read hell of a state). Still, the Pogues were a band I had wanted to see for a long time and I am please I stayed around.
  • Pronghorn, the worlds premier Cow Punk band was good fun. Suspect the are in a Gene pool of one, but still a good act for a open air stage in the afternoon.
  • It should not work, but it did. Les Tribute are all dressed in red, mashing up covers of disco, rock, new romantic and every other type of chart hit from the 1980's with serious showmanship and humor. Went down really well. I am curious how they find bands like that.
  • I had not heard of Lamb, Gong or the Subhumans, but all were a good way to spent an hour or so.
  • I have seen Dreadzone a couple of times and they were on form. Small people really got into the spirit of it during their set.
  • Howard Marks was interesting. I disagree with almost everything he says, but he is still good value listen to
  • Mitch Benn who does the music bit on Radio 4 "The Now Show" was very good indeed.
  • There is a lot of other stuff going on including full contact Morris dancing where they hit each other with 2 inch wooden sticks, various modern sculptures, a golf carts dressed up as a pirate ship and the option of Confession with Nuns who have whips (it would be quite hard to explain).

So, much fun had by the King family once again, made much easier by the weather and we hope to do it again next year.

An assortment of images.

Next race is the Nant Peris Horseshoe on Saturday at 18 miles and 8500ft of ascent where we have something to prove after last year's rather poor showing.

Like this post? del.icio.us | furl | slashdot | technorati | digg

UKUUG summer conference

Friday Aug 07, 2009

Tomorrow, I am talking at the UKUUG conference which I am really looking forward to. Well not actually my talk, but the event itself. I still have a very fond memory of the 1st UKUUG conference I went to in Herriot Watt University in I think 1992 and it influenced all sorts of things down the line including the series of Sun events I termed Mashups held at various UK Universities in the last year. Just technical, no marketing.

[2] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg

How deep a recurse?

Saturday Aug 01, 2009

Chris has been exploring various limits of a lab M8000. Inspired (well, umm, also maybe board on a conf call) by this and prompted by a Twitter update on Google and recursion from Alec (don't recall if I read it first on his blog or Twitter) got me thinking about how deep you can recurse on a modern system? So wrote some code. The marginally tricky bit was setting up the alternate stack to handle the signal on with sigaltstack.

#include < unistd.h >
#include < stdio.h >
#include < signal.h >
#include < stdlib.h >
#include < sys/resource.h >
#include < sys/mman.h >

static void handler();
static void recurse(void);

static int depth = 0;

int main()
{
  struct sigaction act;

  struct rlimit rlp;
  stack_t ss;

  getrlimit(RLIMIT_STACK, &rlp);
  printf("RLIMIT_STACK = %u:%u\n", rlp.rlim_cur, rlp.rlim_max);

  act.sa_handler = handler;
  sigemptyset(&act.sa_mask);
  act.sa_flags = 0;
  act.sa_flags |= SA_RESETHAND|SA_SIGINFO|SA_ONSTACK;

  if (sigaction(SIGSEGV, &act, NULL) < 0) {
    perror("sigaction failed");
    exit(1);
  }

  if ((ss.ss_sp = mmap(NULL, SIGSTKSZ, PROT_READ | PROT_WRITE,
		      MAP_PRIVATE | MAP_ANON, -1, 0)) == MAP_FAILED) {
    perror("mmap failed");
    exit(1);
  }

  ss.ss_size = SIGSTKSZ;
  ss.ss_flags = 0;

  if (sigaltstack(&ss, NULL) < 0) {
    perror("sigaltstack failed");
    exit(1);
  }

  recurse();
}

static void recurse(void)
{
  depth++;
  recurse();
}

void handler(void)
{
  printf("depth = %u\n", depth);
  exit(0);
}

1st attempt on a Macbook with OSX gave a number of 524030. We then moved to Solaris Nevada 110 running on one of our x86 lab system. Also tried the S10 Sparc stable server. The Sparc numbers are a lot smaller than the x86 numbers. The Sparc numbers are similar on Solaris 10 or Nevada. What a great microbenchmark this would make to base purchases on, how deep can a system recurse with no function arguments passed. Many purchasing decisions have been made on the results of benchmarks of similar relevance to the business problem in hand so lets not dismiss totally. Anyway, back to reality.

On a Solaris 10 Sparc box we get

ebusy(5.10)$ cc -o recurse recurse.c                               
ebusy(5.10)$ ./recurse
RLIMIT_STACK = 1000000000:1000000000
depth = 10416627
ebusy(5.10)$ cc  -m64 -o recurse recurse.c   
ebusy(5.10)$ ./recurse                       
RLIMIT_STACK = 1000000000:1000000000
depth = 5681792
ebusy(5.10)$ uname -a
SunOS ebusy 5.10 Generic_137137-09 sun4u sparc SUNW,Sun-Fire

and on the Nevada x86 lab system we get

exdev(5.11)$ cc  -o recurse recurse.c          
exdev(5.11)$ ./recurse
RLIMIT_STACK = 2147483647:2147483647
depth = 16812966
exdev(5.11)$ cc  -m64 -o recurse recurse.c     
exdev(5.11)$ ./recurse                         
RLIMIT_STACK = 1000000000:1000000000
depth = 62499512
exdev(5.11)$ uname -a
SunOS exdev 5.11 snv_110 i86pc i386 i86pc

I am sure there are some games to play with increasing the hard stack limit or allocating the alternate stack a huge segment of memory and recursing through that as well. However, over 62 million stack frames is adequate for most recursive situations which will complete.

Interesting that compilation with -x02 or higher leads to assembler that does nothing and the code just sits in a loop without ever calling the function below.

exdev(5.11)$ pstack `pgrep recur`
17659:  ./recurse
 0000000000400f80 recurse ()
exdev(5.11)$ 

So a few interesting questions that will have to wait for the next conf call where I don't need to pay too much attention.

[4] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg

Give your sysadmin a hug

Friday Jul 31, 2009

A virtual hug of respect to all members of the noble profession on SysAdmin day.

The multitude of skills I needed to develop as a Systems Administrator, even though 15 years ago, are still essential to the job I do today.

When I visit a customer in a crisis, the Systems Administrator is typically the calmest, have the best grasp of the underlying problem and best placed to bring the various parties involved together.

So go on, hug your systems administrator. A systems administrator is for life not just for the 31st of July.

Like this post? del.icio.us | furl | slashdot | technorati | digg

Why large ISM pages are not as large as I expected.

Wednesday Jul 29, 2009

I was pondering why a large SGA segment was made up of 4M pages rather than 256M pages and decided to experiment. A simple as can be bit of code to create an ism segment

#include < sys/types.h >
#include < sys/ipc.h >
#include < sys/shm.h >
#include < stdlib.h >
#include < unistd.h >
#include < stdio.h >

int main(int argc, char **argv)
{
  int sz;
  int sid;
  void *a;

  sz = atoi(argv[1]);

  if ((sid = shmget(getpid(), sz * (1024 * 1024), IPC_CREAT)) == -1)
    {
      perror("shmget failed");
      exit(1);
    }

  if ((a = shmat(sid, (void *)0, SHM_SHARE_MMU)) == -1) 
    {
      perror("shmat failed");
      exit(1);
    }

  sleep(60);
}

In a system with UltraSparc VI+ cpu's (panther) I found by default asking for a 1G ISM segment, we were still producing 4M pages according to pmap -xs. A little bit of kernel code reading and we found the decision is made in map_pgszism which looks like this

map_pgszism(caddr_t addr, size_t len)
    591 {
    592 	uint_t szc;
    593 	size_t pgsz;
    594 
    595 	for (szc = mmu_page_sizes - 1; szc >= TTE4M; szc--) {
    596 		if (disable_ism_large_pages & (1 << szc))
    597 			continue;
    598 
    599 		pgsz = hw_page_array[szc].hp_size;
    600 		if ((len >= pgsz) && IS_P2ALIGNED(addr, pgsz))
    601 			return (pgsz);
    602 	}
    603 
    604 	return (DEFAULT_ISM_PAGESIZE);
    605 }
    606 
A little poking around with mdb shows the value of disable_ism_large_pages to be 0x36. In the common code it is set to 0x2, so must be some platform specific code resetting this value. Poking disable_ism_large_pages to 0x2 with mdb meant the pages for the ISM segment were now 256M in size as reported by pmap. No recommended as a spur of the moment action for your production E25K running Oracle.

disable_ism_large_pages gets set in hat_init_pagesize as an or of disable_large_pages which is set to a shifting and bitmasking perturbation of mmu_exported_pagesize_mask. So a few more hops leads to bugid 6313025 which describes why 32M and 256M pages were turned off for the Panther cpu. Executing application code from the larger (>4M) pages caused nasty thing to happen. The bug is dated 2005 and I had a very distant memory of it, but it was worth tracking down the specifics.

Like this post? del.icio.us | furl | slashdot | technorati | digg

ssd_max_throttle vs IT governance

Wednesday Jul 29, 2009

Chris and I had a short IM exchange yesterday regarding a customer visit I made on monday, its a customer we have both worked with a lot over the years. One of the significant contributory factors of the reported problems is that a line of the form

set ssd:ssd_max_throttle=32
is missing from /etc/system across the estate attached to a particular SAN. Common problem, easy diagnosis, etc.

I made the comment that a co-worker of ours in a different part of the organization would have picked up on the need to address the underlying IT governance issue.

I liked this definition from Wikipedia :

   Specifying the decision rights and accountability framework to encourage desirable 
   behaviour in the use of IT.
and the cause of the cause of the cause of the cause of ssd_max_throttle not being set was in the structure of the established decision rights and accountability framework.

Still, far easier to stick ssd_max_thottle=32 in /etc/system and leave these battles to others this time round. However, an awareness has been sparked.

Like this post? del.icio.us | furl | slashdot | technorati | digg

Snowdon race

Tuesday Jul 21, 2009

The Snowdon Fell Race is the best known Fell race in Wales, probably on account of S4C covering it on television for as long as I can remember. It has a bit of a carnival atmosphere and the reception each runner gets as they pass the reasonably intoxicated crowd outside the Victoria Hotel is very lifting indeed.

I was a quite a bit slower than I had hoped on the ascent, but had a good decent in 35 minutes. Heart rate monitor slipped down on the decent hence the heart rate of less than 110, I can assure you it was somewhat higher.

It is quite a fast race, so having done events like the Highland Fling which are much longer, but far less intense, it seems your body adapts as you train it. I also had not been training hard enough in the 6 weeks before, in part down to a ankle problem. Good as water running is, it is not a full substitute.

Which ever way you cut it, it is sad when a fellow runner dies in a race. It appears to have been heart related, talking to a paramedic after the race who suggested if today is your day, it is your day. Still just as sad and makes you reflect that you don't know how long you have down here left.

Like this post? del.icio.us | furl | slashdot | technorati | digg

The Twitter Revolution – a time for democratic renewal

Thursday Jul 02, 2009

This has been interesting to follow on Twitter from the point of view of the instigator.

Wales has a legacy of a sizable subset of politicians who

  • Don't use technology more advanced than a hand driven egg whisk
  • Don't understand why other people might find technology more advanced than an early 1990's tractor (and nothing wrong with early 1990's tractors, but electrics rather than electronics) useful or necessary

A position which persists at all levels to this day. I can demonstrate the above in the evening of any 3rd tuesday of the month which will leave you in no doubt a lack of technology awareness is inhibiting economic and social progress in Wales.

So it is really nice to see a politician enthusiastic about demonstrating the utility of technology to engage with the wider population. Engaging with the wider community who are not paid up Plaid Cymru members is still a bridge the party has to cross, but appear to be a few pages further forward than the big 3 in the UK.

Like this post? del.icio.us | furl | slashdot | technorati | digg

Stephen Fry and the KT rational Process : new iPhone App idea

Tuesday Jun 30, 2009

I am not a car buff(who would be with a Volvo), but we enjoy the moments of madness on Top Gear such as turning a Robin Reliant into a space shuttle!

Though shalt not question Stephen Fry who was the star in a reasonably priced car on Top Gear which was shown in the UK on sunday night.

From Mr Fry's twitter feed........

Oh lor. So I gather from your tweets that the Top Gear people
didn't edit out the Grindr stuff ... Gracious me.

I don't have an iPhone and am not in the target demographic for Grindr, but it did get spark some lateral thinking about the utility of a iPhone Mobile App which would find the nearest person able to apply rational problem solving to your urgent issue where:-

  • Cause is unknown?
  • Need to know cause?
  • Deviation of actual from should?

Maybe the App could be called ResolveR and if you need process help in

  • Situation appraisal (working out what needs to be done)
  • Solving a problem
  • Making a decision
  • Avoiding future problems or taking advantage of future opportunities
  • Thinking beyond the fix
  • Incident mapping (looking forward to the Leader training in 2 weeks time for this)
  • Performance system engineering

then you bring up the IPhone ResolveR app and you find you find the location of a KT certified Program Leader or Red Belt 20 meters away who can help you.

Should this great idea be stolen and ResolveR make it to market, probably best not to get GrindR and ResolveR mixed up.

Like this post? del.icio.us | furl | slashdot | technorati | digg

Hitchhikers blog posting to Pension Provision

Monday Jun 29, 2009

As a trustee of the Sun UK Pension scheme I am unable, unwilling and not qualified to offer any type of investment advice. The views are mine alone. If you work for Sun UK, Aon can help with Pension specific questions or consult an Financial Advisor (who may or may not be independent) if you are unsure on any of the issues raised in this blog posting and how they effect you.

The Hitchhikers Guide to the Galaxy suggested that there were 3 stages of evolution

  • How do we eat?
  • Why do were eat?
  • Where do we eat?
In Pension planning terms for those of us in our 30's and 40's, the panic striken journey goes something like

  • Why do I need to invest more than the minimum?
  • ****, how much can I afford to invest?
  • Where shall I invest? I have no idea which fund to pick
  • Who can I trust to advise me?

I have spoken to a number of people so scared that they will get bad advice in the 4th step in pension panic evolution, that they got no advice and do nothing for the 1st 3 steps.

Lets do some futurology. As the mist in my pension futures crystal ball clears I see

  • The basic state pension will keep you alive, but little more
  • Only saving at the default rate for a Defined Contribution Pension Scheme will leave us dissapointed and poorer than we think we should be
  • It is going to take at least 10 years the political minerals to develop to tackle the unfunded time bomb which is Public Sector Defined Benefit (Final Salary scheme), estimated at 65% to 85% of GDP and all off balance sheet to be paid for by future taxation
  • Pension tax relief for higher wage earner will be nibbled away at as the threshold is reduced.

Not a very ambitious crystal ball I grant, it was quite cheap because it is unregulated.

I was very fortunate to meet a retired R.A.F. officer while on a diving holiday in the Maldives around 8 years ago. He managed to explain that the easiest way to understand how big a investment pot to aim to accumulate when you retire is to work backwards. The spirit of example he gave me goes like this:-

Lets assume I am a member of a Defined Contribution Pension Scheme and want 20,000 pension income per year when I retire.

Assume predicted annuity rates at 5%. Could be more or less in 2040, but lets assume 5% for easy sums.

So to give us 20,000 pension income a year we would need to have squirreled away 400,000 pounds available to buy a pension at the time of retirement.

For an individual who is 35 and intends to retire at 65 (makes the sums easy). If to date they have saved a pension pot of 100,000, they would need to save an additional 300,000 pounds more to meet their 400,000 pounds target. That is a need to save around 10,000 in 2009 pounds (remember inflation) each year over the next 30 years.

The default on many DC pension schemes is around 10% including company contribution so if you earn 40k, that means you would be saving around 4k a year into a pension which is about 40% of what might be needed to keep you in the beer to which you wish to become accustomed post retirement with an income of 20k a year (must remember to adjust for inflation of course).

We really should put a margin of error in for the inherent complexity of inflation, investment return and annuity rates when one retires. Even a basic spreadsheet user who can use Google can factor in some element of inflation vs investment return over 20 years. Of course, it will be wrong, but it should scare you into the right ball park.

Apart from Alan the retired RAF officer, no one I have talked, professional or otherwise, has suggested setting a broad goal to work towards. If you are a project manager or product developer, you will know the best way for a project to fail is not to know have a clear idea of the intended outcome. Things might change on the way as the future is unknowable, but you have a goal to work towards which can be modified as knowledge about the future unfolds.

This is the real problem I have with the current and proposed future FSA policy, it makes no provision for setting a intended outcome of the project( of having a secure retirement), instead suggests a save and hope for the best approach. No clear goal setting make it very hard to reach a goal.

If you think you have just read any advice above, you have not. Note the title as a blog posting not a guide.

I advise you to get a Scottish Wife, they don't spend much money That is an example of what financial advice looks like.

So some useful links which I am not advising you to think about looking at include

Some Pension Policy changes that I woud like to see made include

  • Put all MP's into a Defined Contribution Pension scheme. This would help focus their minds on the underlying pension crisis.
  • Any consumer financial advisor must give at least 1 hour of free financial goal setting (as above, hopefully better and more complete) before they can start charging.
  • Empower the purchase of Corporate Bonds in demoninations of less than 50k (5k or 1k) within a SIPP Pension Plan. Some European rubbish I need to fully understand the background to, but is a real pain is trying to build a risk adjusted portfolio.
  • Require financial journalists to be regulated and accountable for the advice they give in print.

Water running : seems to work?

Friday Jun 26, 2009

Since a very pleasant morning run up Ben Klibreck while on Holiday in May, I have had a bit of ankle pain.

It came to a fore during the last 7 miles of the Man v Horse 10 days ago. So a bit of time off running, what else can we do?

The mountain bike is obvious, we have some great local trails very close including Nant-y-Arian, one large objective hazard to be aware of is 4 stone of dog cutting across your path. I have also been out Water Running in a local lake where a wet suit is mandatory. I do get odd looks from mountain bikes passing the lake.

Seems to work in that you can get quite tired in 45 minutes. Should be back running next week, but think I will continue with the water running at least once a week.

Like this post? del.icio.us | furl | slashdot | technorati | digg

GCHQ steps up strategy to combat cyber-attacks

Thursday Jun 25, 2009

Some more money for GCHQ as part of the war on Cyber crime.

Sounds like Solaris Trusted Extensions (or just normal Solaris) should have a part to play here. It is good to be proactive, but sounds no where near as exciting as "offensive Cyber Weapons".

Maybe something along the lines adding a header to each email suggesting that the parents of the intended recipient were siblings, or do they mean something else by "offensive Cyber Weapons".

Like this post? del.icio.us | furl | slashdot | technorati | digg

Thanks to The Times, more Fuzz blogs bites the dust

Monday Jun 22, 2009

I don't have a lot of interaction with the Police, and may it remain so. I do read a couple of blogs which are from serving Police officers and it comes across that they live in the interactions of madness between bureaucracy and the failings of society. The world is much more complex, but as a 40,000ft view, it works for me.

Most of the Police are human and try to do a good job. Some officers are "unsuited for the role" and get it very wrong, but that is true of any walk of life. On the odd occasion I have been in a UK town late on a Friday or Saturday night, they really do have some quality idiots to deal with.

It really bothers me that "The Times", a paper which I had some respect for and bought on occasions, sought to expose the identity of an anonymous Police Blogger. The side effect of the action of The Times is that a number of the better Police bloggers have stopped writing or are having such thoughts including Stressed Out Cop and The Plastic Fuzz.

The Times has moved off my "buy" list of papers. If their editorial judgement is this bad, then can you trust what you are reading? The worst of tabloid journalism having a negative impact on freedom of expression and informing the public on what is really happening if they have to dial 999.

Like this post? del.icio.us | furl | slashdot | technorati | digg

Fly tipping : If something really annoys you, complain, then do something about it

Saturday Jun 20, 2009

While I have quite liberal politics, I would vote for making fly tipping the only crime to carry a manadatory captial offense. Harsh, but fair.

I had reported a caravan dumped near Nant-y-Moch a few times, once via the web site linked from here. It is on a very frequently used mountain bike trail and I have heard a lot of people complain that it looks a mess.

The caravan which had been dumped about 18 months ago was now being distributed around a valley which has a Welsh name which translates into "the nose of the bellows", so your can imagine how windy it gets up there in a storm. I would have though the local council was responsible for clearing it up, but seems not. The Ceredigion highways department do not have a lorry which could get to the site, so I offered to turn up with tractor and trailer and move it to the road for the Highways Department to dispose of which was arranged by a Ceregidion Environmental Health Officer.

Some by a round about route a collective of the farmer, myself, two local police off duty and 2 council officials, also in their own time, decided it was time for "JFDI".

The before

and the after with the 2 police officers and the farmer.

The Council Officers turned up after the remains were loaded on the trailer, so missed Tea and a stove cooked bacon butty, but they should take much credit for getting all the parties together.

[1] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg

Where does the SLP in prstat -m come from?

Thursday Jun 18, 2009

So we have some prstat output for a single process like this

exdev(5.11)$ prstat -c -m -p 2312 1            
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   1   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.24, 8.79, 8.84
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   2   0  19   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.29, 8.80, 8.84
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.34, 8.80, 8.84
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.38, 8.81, 8.84
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.42, 8.82, 8.85
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.38, 8.81, 8.84
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   2   0  19   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.33, 8.80, 8.84
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.29, 8.79, 8.84
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.24, 8.78, 8.83
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.19, 8.77, 8.83
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   0   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.15, 8.75, 8.82
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/NLWP  
  2312 clivek   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   2   1  19   0 gnome-netsta/2
Total: 1 processes, 2 lwps, load averages: 8.10, 8.74, 8.82
exdev(5.11)$ 

If performance is your worry and this process is on your business problem critical path, the obvious question is where is the significant amount of SLP time coming from. SLP, the man page defines as

     SLP

         The percentage of time the process has spent sleeping.

is not the most helpful definition, but means that the process is waiting for events such as network or disk I/O, inside poll or some other waiting state where the cpu has been given up.

So if we really want to know what the process is doing while it is sleeping, we can use this bit of D

#pragma D option quiet 

sched:::off-cpu
/curlwpsinfo->pr_state == SSLEEP && curpsinfo->pr_pid == $1/
{
	self->ts = timestamp;
}

sched:::on-cpu
/self->ts/
{
	@[execname,stack()] = sum(timestamp - self->ts);
	self->ts = 0;
}

tick-60s
{
	normalize(@, 1000000);
	printa("%20s %k %@u\n", @);
	exit(0);
}

which gives us output along the lines of

exdev(5.11)$ pfexec dtrace -s mstate_sleep.d 2312   
    gnome-netstatus- 
genunix`cv_wait_sig_swap_core+0x170
genunix`cv_wait_sig_swap+0x18
fifofs`fifo_read+0xca
genunix`fop_read+0x6b
genunix`read+0x2b8
genunix`read32+0x22
unix`_sys_sysenter_post_swapgs+0x14b
 2
    gnome-netstatus- 
genunix`cv_timedwait_sig_internal+0x1d6
genunix`cv_waituntil_sig+0xba
genunix`poll_common+0x461
genunix`pollsys+0xe4
unix`_sys_sysenter_post_swapgs+0x14b
 55101

exdev(5.11)$ 
So now we have the evidence that the process is waiting in poll. We could have infered the same from truss, but for more complex cases where there are a more diverse set of system events generated by the application, the picture will not be so clear.

Not that in a 60 second sample time, we have only 55101ms of SLP time accounted for. This is because the application (gnome-netstaus-) wakes up from its poll loop every 5 seconds so because of the way the D is constructed, we will miss any period before the 1st off-cpu and after the last on-cpu. Most application, this will be a few ms missed, but worth being aware of.

Off at a complete tangent, I learned this evening that the UK peak power consumption is around 50 Gigawatts. A additional wind farm is proposed in our area to come on stream in 2012 after 2 years of fighting its way through planning. Its output will be 140 Megawatts, so 1/350 of the UK peak requirement. Sounds a bit optimistic in terms of contribution, so going to have to check the numbers out.

Like this post? del.icio.us | furl | slashdot | technorati | digg

If I was a blogging copper, I would not be pleased.

Thursday Jun 18, 2009

If I was a blogging serving police officer, I would be rather annoyed by this. The NightJack blog has now been deleted. While not unique in terms of a blogging serving police officers, it was one of the better police blogs. Thanks to Mr Finkelstein, we are no longer able to read it. Can't he do something useful like hassle M.P's, rather than trying to identify someone who does not want to be identified, but provides a public service which gives an insight into the chain of causality as to why the police are unable to respond to crimes such as burglary in a timely manner.

Like this post? del.icio.us | furl | slashdot | technorati | digg

Patch 141414-01 : U7 sustaining gate

Tuesday Jun 16, 2009

Patch 141414-01 which was released yesterday (Update 7 sustaining gate for SPARC) has a couple of interesting performance related fixes for larger systems. I came across a case of 6682267 at a customer last week. Lockstat -C outputs looks something like this

root@xxxxxxxx-M9000 # lockstat -C -s 50 -n 1000000000 sleep 10 | more

Adaptive mutex spin: 1097781 events in 16.337 seconds (67195 events/sec)

-------------------------------------------------------------------------------
Count indv cuml rcnt     nsec Lock                   Caller                  
99062   9%   9% 0.00   232883 pcf+0x8                page_create_putback+0x64

      nsec ------ Time Distribution ------ count     Stack                   
       512 |                               62        page_alloc_pages+0x16c  
      1024 |                               675       anon_map_getpages+0x348 
      2048 |                               2162      segvn_fault_anonpages+0x384
      4096 |@                              6125      segvn_fault+0x530       
      8192 |@@@                            11233     as_fault+0x4c8          
     16384 |@@                             9625      pagefault+0xac          
     32768 |@@@@                           13437     trap+0xd50              
     65536 |@@@@@@                         22915     utl0+0x4c               
    131072 |@@@@                           13585     
    262144 |@@                             7181      
    524288 |@                              3778      
   1048576 |                               2975      
   2097152 |                               2439      
   4194304 |                               1898      
   8388608 |                               891       
  16777216 |                               78        
  33554432 |                               2         
  67108864 |                               1         
-------------------------------------------------------------------------------
Count indv cuml rcnt     nsec Lock                   Caller                  
90004   8%  17% 0.00   306645 pcf+0x8                page_create_wait+0xa0   

      nsec ------ Time Distribution ------ count     Stack                   
       512 |                               74        page_alloc_pages+0x68   
      1024 |                               804       anon_map_getpages+0x348 
      2048 |                               1554      segvn_fault_anonpages+0x384
      4096 |@                              3909      segvn_fault+0x530       
      8192 |@@                             8361      as_fault+0x4c8          
     16384 |@@@@                           12636     pagefault+0xac          
     32768 |@@@@@                          16311     trap+0xd50              
     65536 |@@@@@                          15513     utl0+0x4c               
    131072 |@@@                            10583     
    262144 |@@                             6068      
    524288 |@                              3897      
   1048576 |@                              3335      
   2097152 |@                              3077      
   4194304 |                               2652      
   8388608 |                               1108      
  16777216 |                               112       
  33554432 |                               7         
  67108864 |                               3         

Worth considering this patch under load if you see a similar stack or drop me a mail as I have a few bits of D and MDB which can help confirm the initial analysis.

Like this post? del.icio.us | furl | slashdot | technorati | digg