Thoughts on Thin Computing, Latency, Minimization, Solaris and Multiplicity ...of...it...all Isaac's musings

 

New York City on the 1st day of November ...aaah, yet another data point reminding us that the flavor of autumn has once again made its appearance.  I love autumn!  As if last night's stroll with the kids trick-or-treating wasn't enough, this morning's cold, crisp air (augmented with randomly dancing yellowish leaves occasionally painted with traces of red and orange) created such a wonderful & unique experience. (Had it rained, the experience would not only have been  unique but a wet one, too!)

The constant hustle and bustle of the city was no different today.  Brooklyn's Marriott Hotel (by the the historic Brooklyn Bridge) hosted this year's New York City Government Technology Forum.  Sun was one of the sponsors. There was an exhibit with many interesting (and well-known) vendors catering to the City, State and Federal Government industries.  I had an honor of speaking at this event; topic - Server Virtualization.

Together with Christopher Theon (Practice Manager, GTSI) and Marcos Merced (an MIS Director from DoITT) we had just over an hour to define and discuss server virtualization.  We divided the time frame into, roughly, three 15 to 20-minute timeslots (to allow opportunity for Q&A). Since my topic dealt with introducing virtualization and discussing hardware and software components, I spoke first. My goal was to remain vendor-neutral and provide an objective overview of server virtualization technologies. (I hope I've succeeded in meeting that goal).  Here are my presentation slides.   Chris extended the discussion by picking up on the business drivers that I had introduced.  He took the discussion further into cost analysis and other business-centric elements that are critical in evaluating the approach and success of a virtualization project.  Marcos talked about a case study of deploying VMware solutions over the last 3 years, and how it has helped his organization  improve overall agility and importantly - reduce costs.  While each of our topics could well take an hour, each on their own, I would love to hear what you thought (if you were among those attending).   

And if you weren't in the audience, I would still love to hear about creative ways that you've used to employ virtualization technologies to address business drivers (even if you did not [or do not] use an ounce of Sun's technology to accomplish that)...

 

Have you caught the OpenSolaris wave yet?  Back in the Summer, Brian Gupta and I kicked off a New York OpenSolaris User Group  for the purposes of educating the community and soliciting input from the community on how people use OpenSolaris and Solaris technologies. This is meant as a two-way street. Check out some of our previous meeting pages, as well as an upcoming meeting on November 15th.   Why is all of this important? If you are not yet aware of how Solaris development works, you may be pleasantly surprised that all of the development of new technologies goes into the next release of Solaris that is currently being developed in the open - dubbed Nevada, i.e. SunOS 5.11. You can actually get a taste for some of the technologies that have been opensourced as part of that process and experience them before SunOS 5.11 is released commercially.  If you look back at the calendar, eventhough Solaris 10 came out commercially in January of 2005, our engineers had already been running builds of it back in 2002.  That means enhancements to (or new) features like CIFS, ZFS, FMA, enhancements to crypto technologies, DTrace, Intel-specific enhancements ....are all available today, via opensolaris distributions, such as SXDE 9/07 release.  Check out the free SXDE release...

Last week I had an opportunity to meet with and present to one of our interesting financial customers (who will remain nameless for non-disclosure reasons). Subject of the meetings had been Solaris, virtualization, container management, processor roadmap discussions (and so forth).

 During the meeting we discussed how Solaris Zones are being used with a real business strategy in mind. Although Zones are a nice feature of Solaris, when taken at face value it doesn't really stand out until you do something creative with it, something so customer-specific (yet amazing!) that vendors typically do not offer out of the box.   What I am referring to here is facility that's been developed [in-house] that focuses on empowering lines of business units to self-sufficiently create Solaris Zone environments for themselves, via  a webtool. 

Not only that, but what had been demo'ed to me actually appeared as a working, really thought through, solution that, at its core, has business drivers in mind.  One of the key deliverables of this webtool is that it leverages inexpensive technology to enable the consumer to request an instantiation of a pre-defined zone profile (and there are a few), with storage, processing power, memory, IP allocation, etc. - self-sufficiently via the intranet web! On top of that, a Zone gets provisioned and becomes available for use in less then a minute!  

How does this exponentiate the coolness factor ?  It allows the customer to expedite the time to requisite hardware/processing space in the datacenter and shrink that from what it would otherwise be (and you know how long that may take) ...down to 20 seconds!

I was amazed.   The idea is really straight forward - there are a number of servers, SAN storage and IP address pools that are allocated for LOB's as a co-operative. Leveraging Solaris Zones (often referred to as Containers), the tool offers up an opportunity for end-users to pick a pre-defined Zone profile (there's a set, reflecting typical types of applications that are of utmost interest, held in a cpio archive).  An end-user access the internal website, chooses an available profile, fills out some basic information about a zone and submits a form. Within seconds, the work is done. The user sees progress of zone creation, gets notified by email when the process is complete. Because it is a shared environment, a zone has an expiration date (think: leasing) that can be adjusted (if needed).

[BRILLIANT, MIKE!]
 

---

 

8/12/07 - Update and a response to comments -

 

I understand the tools were pretty much  PHP and CGI-like scripts on the backend to call things like zonecfg(1M) and zoneadm(1M).  I do not have specifics as to the effort, however I inquired as to the timeframe for getting this sort of thing done and I understand it was done in a matter of a few months, by one key person working on this often on their own time - as a pet project.


Since this is based on the Solaris 10 11/06 release (which backported* a ton of Zone/ZFS integration code from OpenSolaris/Nevada), the zone clone feature is leveraged, so is the ZFS filesystem, which is what zones get deployed to and how reservation and quotas are enforced. They also provide storage for data on ZFS so its easier to manage and delegate complete control of ZFS filesystems to the Zone administrator - voila!

*For those that may not be aware of this fact, a currently available commercial release of Solaris does get features from the next release currently in development. As such, Solaris 10 does get certain features from the next release of Solaris  being developed in the OpenSolaris community. The codename for the SunOS 5.11 kernel is Nevada and that is where new features appear first, prior to getting (if appropriate) backported/integrated into the Solaris 10 codebase. See http://www.opensolaris.org for more.

 

These past few weeks have been interestingly eventful.

 

In June Brian Gupta and I kicked off an OpenSolaris User Group for the New York area. 

In July we had Ian Murdock and Sara Dornsife visit us while in New York.
As a quick reference, Ian Murdock is a founder of Debian distribution and a former Chief Technology Officer at the Linux Standards Base, whilst Sara runs OpenSolaris Marketing.

Details and some initial responses from the July meeting have been interesting, to say the least. We were very pleased to have had a number of members of the New York Linux User Group and UNIGROUP communities with us. This is what communities are all about. Thank you to all who make events such as this educational and fun.

A day later we had an all-day Sun/Solaris marketing event in our New York office, followed by a pipe explosion a block away from our office. NOT fun!

Earlier that day, I had a 40 minute time slot to present to a room full of customers on the features and futures of Solaris. For obvious reasons I can not distribute content in writing that alludes to future-available features and timeframes, but I have sanitized it to the extent that, I think, allows me to confidently portray the types of things that we are expecting in a Solaris update really-soon-now. What's interesting is that a good number of features that will be showing up in Solaris have actually been out there in the OpenSolaris world for a while. This is how we develop Solaris and if you're not yet an OpenSolaris community member, what is stopping you?

You're welcome to take a peak at the slides if you missed it when I presented it in New York (and Somerset, NJ the next day).

Let me know what you think.

Last week I had been in Mexico City, presenting at Sun's Immersion Week 2007 conference.  Speaking on 2 topics, twice each in one day, does pose its challenges.  As one example from this presentation, I recall catching myself uttering phrases  like "Unplanned Uptime" (in the context of Solaris features like SMF, FMA, Zones).  Of course, my intent was to elaborate on how Solaris 10 helps minimize unplanned downtime.... but jinxing myself with phrases like unplanned uptime was certainly a jaw-dropper, to say the least.  

Another, as shared over diner with Scott, Antonio, Mauricio, Jazmin and Enrique later in the day, was to maximize unplanned uptime of non-global (loco) zones.  But you'll haveta buy me a drink to get the full story.

 

 

Bah! A short stroll on the blogroll ...

I must've missed  this when it was announced just 2 weeks ago! 

This is really taking Project Blackbox much further. More importantly, its taking it OUTSIDE from the Datacenter and into the great white open.

 
 

Solaris 10, as well as OS minimization in general, still seems like a hot topic these days.  Following my Customer Engineering Conference '06 blog entry from San Francisco, I've uploaded  a Solaris 10 Adoption and Minimization presentation [pdf] that I gave there last October.

Would love to hear how you're doing in this space and if you need help, would love to come and talk with you and your organization about lessons learned in the field on this very interesting subject.

Ah, April 1st ...

Who would like to be teased? 

Ever wonder why so many blogs are about cats and dogs that people have and adore? Well, I don't have a dog, nor do I have a cat, but.... my sister has a Rottweiller so I guess that qualifies for a blog entry related to dogs.   And yes, he has been to my datacenter.  Sniffed quite a bit actually, all without causing any downtime.

    Last month I was invited to help kick-off a monthly set of OpenSolaris User Group meetings in Florida. We had started with Ft. Lauderdale and Tampa, and in the months ahead are planning to be adding Orlando. We had a number of participants and, as promised, I wanted to post the presentation that I had given. The topic was OpenSolaris and, as we'd say in Russian, "what would you eat it with", or more precisely: What it is, how it works and why it may be of interest to you - the community.

 The slides for Ft. Lauderdale's discussion are here whilst the presentation deck for Tampa is located here. The content was pretty much the same. Both are in .PDF.   You can use Evince or gpdf to view these (in OpenSolaris).

Big kudos to Forsythe, Inc. for helping sponsor this event, and in particular to John McLaughlin, who runs SystemNews e-newsletter.

 

If you're into Russian pop, like me,  but may not have  yet have had a chance to hear Valeriya perform, then you may be in for a treat. Check out her myspace link and get a glimpse of her cool videos.  She has a wonderful voice, and a totally charismatic personality.

Today was the conclusion of yet another successful OS Ambassadors conference in Menlo Park, CA. OS Ambassadors is a Sun internal program that exists to foster cross-pollination of ideas and trends between R&D and field/customer-facing engineers. Over the years, there have been many different Ambassadorship programs, focusing on topics such as Operating Systems, Security, Data Center, Commercial Frameworks & Tools, Networks, Cluster, Campus (universities), etc.
For those that may not know what this is, the OS Ambassador program consists of roughly 100 deeply skilled technical architects and customer-facing engineers with membership spanning the entire globe and representation from various organizations such as Globals Sales, SunService and Professional Services Delivery. We help shape the trends in Solaris by being a consistent conduit between our engineering organisation and the customers we work with on daily basis. There are many interesting perspectives that get voiced during such meetings from people facing a myriad of unique customers' business problems and technology challenges.
Topics discussed this week included:

• modernization and usability of Solaris
  surely you've heard about Ian Murdock, founder of Debian, coming to work with us
  
• ZFS
• FMA
• what's in the upcoming Update to Solaris 10
• what's on the horizon
• patching
• what's getting integrated in OpenSolaris
• ZFS boot, yay!!
  
• direction in management tools
• storage technologies
• database strategies
• lots of other nice little surprises

...And I had a relatively seamless installation of Solaris Express Developer Release (based on Solaris Next, or as we call it "Nevada" build 60). I did find and file a bug with GNOME, but a day later thought about it some more and figured out what was causing the failure. (The failure, by the way, is apparently present in some of the previous builds as well).
So stay tuned. And if you're not on OpenSolaris yet, please take a look and become a member of this rapidly growing community. That is the one place where you can see and experience new features in the land of Solaris development and get a sense for some future directions!

Ever feel like your actions are setting off a number of events in motion ?

We do this every day, I suppose -but w/out getting too philosophical (or geeky) here's a way I think of this now:

If I were dtrace, I'd be witnessing probes getting fired.

Ha!

A note I had written to...someone...very...close... and - thought would be worthwhile to convert to a blog

---

Ok, so I remembered our conversation from last week about binding processors to CPUs and seeing which processors these processes are actually bound to.

I wanted to refresh this in my memory and went through this last night and here are my notes.

I am doing this on a Solaris 10 server with 2 CPUs, but am assuming that this should work on Solaris 8 as well. The only caution I have is around not being completely sure whether the 'ps' command was actually enhanced to take '-o psr' as one of the 'format' arguments to it back in Solaris 8 or not. (You can validate this just by running '/usr/bin/ps -o' alone on the command line and looking for a line that includes 'psr' and/or 'pset' in the output.

Before I start, the output should look like the following:

root@ultra2cd:/export/home/isaac: 23:35 > /usr/bin/ps -o
/usr/bin/ps: option requires an argument -- o
usage: ps [ -aAdeflcjLPyZ ] [ -o format ] [ -t termlist ]
        [ -u userlist ] [ -U userlist ] [ -G grouplist ]
        [ -p proclist ] [ -g pgrplist ] [ -s sidlist ] [ -z zonelist ]
  'format' is one or more of:
        user ruser group rgroup uid ruid gid rgid pid ppid pgid sid taskid ctid
        pri opri pcpu pmem vsz rss osz nice class time etime stime zone zoneid
        f s c lwp nlwp psr tty addr wchan fname comm args projid project pset

Now, onto the experiment. The goal is to find out whether, given a certain process that is running (and you knowing its PID) we need to understand which CPU it is bound to (if any, in fact) ?

On my system, I have 2 CPUs: CPU 0 and CPU 1, as seen from the output of 'mpstat 1' below:

root@ultra2cd:/export/home/isaac: 23:29 > mpstat 1
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0    0   0  112   401  301   49   11    2    0    0    16   34   0   0  65
  1    0   0    0   114  102   54   12    2    0    0    15   34   0   0  65
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  140   0  273   623  524   90    2   10    5    0   374    2   4   0  95
  1  489   3  215   112  106   57    4    4    6    0   364    2  14   0  84
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  381  11  183   590  490  106    5    8   63    0  1318   14  21   0  66
  1   83   0  186   110  105   98    2   12   64    0   365    1  20   0  79
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0  209   7  366   596  490  110    8   10  121    0   864    5  10   0  85
  1  601  11  100   116  111   99    4   11  122    0  2132   23  23   0  54
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0   41   1  290   638  535   99    2    9   73    0   263    1   5   0  94
  1  407   8  103   117  108   59    7    6   68    0  2467   18  34   0  49
^C

Now, in my environment, I don't YET have a process bound to a CPU, so I will pick a random process and bind it to CPU #0. I pick my 'tcsh' process, and note its PID (16029).

root@ultra2cd:/export/home/isaac: 23:29 > ps
   PID TTY         TIME CMD
 16014 pts/3       0:00 sh
 16029 pts/3       0:00 tcsh
 16130 pts/3       0:00 ps

Then, I run 'pbind -b 0 16029' (after glancing over 'man pbind')

root@ultra2cd:/export/home/isaac: 23:30 > pbind -b 0 16029
process id 16029: was not bound, now 0

This tells me that process was not bound and now is bound to CPU 0.

I can check that with 'pbind -Q', which will tell me all bindings.

root@ultra2cd:/export/home/isaac: 23:30 > pbind -Q
process id 16029: 0
process id 16331: 0

Notice that now I have another process, 16331. What is that ?

That's the 'pbind -Q' itself, which the shell forks to execute the query.

Notice if I keep doing it, the PID of this "mysterious" process will change. That is because there's a new process ID being created every time I invoke 'pbind -Q'.

root@ultra2cd:/export/home/isaac: 23:30 > pbind -Q
process id 16029: 0
process id 16353: 0

So now I have a process bound to a CPU, so this should be somewhat equal given your original question of 'how can I see what CPU a process is bound to?'.
Armed with a PID of a process of interest, I feed it to 'ps' and ask 'ps' to shape its output format to include the actual processor, and I get my answer below:

root@ultra2cd:/export/home/isaac: 23:31 > ps -p 16029 -o psr
PSR
  0

Looking at the various additional formats that 'ps' is capable of arranging, I can include things like the pid itself, or the fname, or even the memory address of the process (see '/usr/bin/ps -o' for the additional types of formats supported).

root@ultra2cd:/export/home/isaac: 23:31 > ps -p 16029 -o psr,pid,fname
PSR   PID COMMAND
  0 16029 tcsh

root@ultra2cd:/export/home/isaac: 23:31 > ps -p 16029 -o psr,pid,fname,pset,addr
PSR   PID COMMAND  PSET             ADDR
  0 16029 tcsh       -      300021ab7f8

Now, what if I had a number of processes that I kicked off from the current shell, and the list was too numerous to provide ? I could simply omit the '-p PID' option to 'ps' and here's what I'd get.

root@ultra2cd:/export/home/isaac: 23:31 > ps -o psr,pid,fname,pset
PSR   PID COMMAND  PSET
  - 16014 sh         -
  0 16029 tcsh       -
  0 16527 ps         -

(Note that 'pset' is a processor set, something that I have not yet configured on this system, hence the corresponding output is simply a hyphen)
What if I wanted the address ?

root@ultra2cd:/export/home/isaac: 23:31 > ps -o psr,pid,fname,pset,addr
PSR   PID COMMAND  PSET             ADDR
  - 16014 sh         -      3000219afc0
  0 16029 tcsh       -      300021ab7f8
  0 16558 ps         -      300021bb350
root@ultra2cd:/export/home/isaac: 23:32 >

What if I wanted more things, including user, time, scheduling class, priority ?

root@ultra2cd:/export/home/isaac: 23:48 > ps -p 16029 -o psr,pid,fname,pset,addr,time,class,pri,user
PSR   PID COMMAND  PSET             ADDR        TIME  CLS PRI     USER
  0 16029 tcsh       -      300021ab7f8       00:00  FSS  59     root
root@ultra2cd:/export/home/isaac: 23:48 >

root@ultra2cd:/export/home/isaac: 23:48 > ps  -o psr,pid,fname,pset,addr,time,class,pri,user
PSR   PID COMMAND  PSET             ADDR        TIME  CLS PRI     USER
  - 16014 sh         -      3000219afc0       00:00  FSS  11     root
  0 16029 tcsh       -      300021ab7f8       00:00  FSS  52     root
  0 18453 ps         -      300021c1348       00:00  FSS  59     root
root@ultra2cd:/export/home/isaac: 23:49 >

I think you get the point.

One last comment: notice how the 'sh' process has a '-' hyphen in the PSR column. That is because the 'sh' process is a parent of 'tcsh' and, since I only bound 'tcsh' itself, only tcsh's children will be bound, not the parents. Processor bindings are inherited from the parent.

Hope this was interesting and helpful.

Please check out some of the videos that give you a closer in-depth look into what Project Blackbox is all about. Thanks to Dave Douglas and everyone on the team who helped make this happen!!

Today I am acting as a point person at Sun's exhibit table at the United Nations in midtown New York, where AIT Global is holding its 18th Annual Conference, Workshops & Exhibit.

This year's topics include: Best Practices & Solutions for SOA, IT Infrastructure, InfoSec and Web4Development.
Glenn Brunette (Sun's Distinguished Engineer) and Michelle Dennedy(Sun's Chief Privacy Officer) are both speaking today.

With only 3 hours into the event, I've already seen so much interest in this that I've almost ran out of most of the collateral I brought to give out.

The picture boards for the exhibit are quite indicative of the phenomenon.
See for yourself! (please pardon the resolution - I am doing this from the event itself!)

 

 

 

 

 

 

 

 

 

This is the setup.... and this is I with Glenn Brunette 

Well, here we are again, about to surprise the industry with... [things we can't blog about yet!]. We'll just have to leave your appetite wet for now

Can you say: Can't blog this ?
Can you say: Can't blog this ?
Can you say: Can't blog this ?
Can you say: Can't blog this ?
Can you say: Can't blog this ?
Can you say: Can't blog this ?
Can you say: Can't blog this ?
Can you say: Can't blog this ?
Can you say: Can't blog this ?
Can you say: Can't blog this ?