Sanjay Nadkarni's Weblog: Sanjay Nadkarni's Weblog

 

A couple of  months ago, I got  a new workstation.  A quad core Xeon box from Dell. It came with preinstalled Ubuntu.  I decide to play with that for while.  After a while I noticed that the system would randomly reset. Ah the darn unstable linux is what I thought!!

I reinstalled the system with OpenSolaris 2008.11 and it ran for a day or so. The system reset again.  Annoyed I started looking at /var/adm/messages and I found that there was a hardware fault detected by FMA and taken appropriate action.  Now it was nice to call Dell Support  and tell them definatively the cause, diagnosis and that I needed a new motherboard.  

The rep asked me what test was running.  I said nothing. This functionality is built into OpenSolaris...and it's free!!  One cannot get this running linux.

Here's what fmadm faulty printed

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Feb 20 01:29:08 152a7687-c256-40dd-80b1-83c1f4ed74c7  INTEL-8001-43  Critical 

Fault class : fault.cpu.intel.nb.ie
FRU         : "MB" (hc://:product-id=Precision-WorkStation-T3400:chassis-id=65QLTH1:server-id=opensolaris/motherboard=0)
                  faulty

Description : Northbridge has detected an internal error  Refer to
              http://sun.com/msg/INTEL-8001-43 for more information.

Response    : System panic or reset by BIOS

Impact      : System may be unexpectedly reset

Action      : Replace motherboard

--------------- ------------------------------------  -------------- ---------
TIME            EVENT-ID                              MSG-ID         SEVERITY
--------------- ------------------------------------  -------------- ---------
Feb 20 01:29:08 50ba84aa-3f12-c2c5-9c0b-8fdec9454104  INTEL-8000-LE  Major    

Fault class : fault.cpu.intel.l1dcache
Affects     : hc://:product-id=Precision-WorkStation-T3400:chassis-id=65QLTH1:server-id=opensolaris/motherboard=0/chip=0/core=3/strand=0
                  faulted and taken out of service
FRU         : hc://:product-id=Precision-WorkStation-T3400:chassis-id=65QLTH1:server-id=opensolaris/motherboard=0/chip=0
                  faulty

Description : A level 1 Data Cache on this cpu is faulty.  Refer to
              http://sun.com/msg/INTEL-8000-LE for more information.

Response    : The system will attempt to offline this cpu to remove it from
              service.

Impact      : Performance of this system may be affected.

Action      : Schedule a repair procedure to replace the affected CPU.  Use
              'fmadm faulty' to identify the module.

 

We recently got a few Nehalem systems from pogolinux.  When ordering we noticed that they offered to preinstall the systems with OpenSolaris 2008.11.  Naturally we chose it.

The systems arrived, we plugged them in and there was the login screen for OpenSolaris.

Nice!!

Itś been a long time but its finally nice to have acroread for Solaris x86.   Thank you very much Adobe folks

http://www.adobe.com/go/getreader

As a developer, I often have to build modified images to test out my changes.  Using DC one can do this very easily. I imagine others might have similar  needs, hence here is a quick tutorial.

The distro_constructor lists the set of steps required to build an ISO. One can see this by running:

distro_const build -l <config_file.xml>. Here's an output from one of my runs.

distro_const build -l slim_cd.xml

Step           Resumable Description
-------------- --------- -------------
im-pop             X     Populate the image with packages
im-mod             X     Image area modifications
slim-im-mod        X     Slim CD Image area Modifications
br-init            X     Boot root initialization
slim-br-config     X     Slim CD boot root configuration
br-config          X     Boot root configuration
br-arch            X     Boot root archiving
slim-post-mod      X     Slim CD post bootroot image area modification
grub-setup         X     Grub menu setup
post-mod           X     Post bootroot image area modification
iso                X     ISO image creation
usb                X     USB image creation

Modifying an image in DC is achieved by addding an extra step. In the example below, I had to test the install changes to support booting off a multi-terabyte disk. My SUNWinstall, an SVR4 pkg, had the updated changes. Since the installer is part of the microroot, I decided that changes had to be applied after "Image area modification".

In the slim_cd.xml file these steps are called out in the finalizer class. Each step is a script sub-class which has a step name, a message and the script to call. Invoking my favorite editor (emacs!!) I searched for "<finalizer>". The following lines were added to create a new step.

 <script name="/export/bld_images/mtb-pkgadd">
         <checkpoint name="mtb-pkgs" message="Add MultiTB pkgs"/>
 </script>

These lines were inserted after the "im-mod" step. Next, I created my script mtb-pkg. When the script is called from the distro constructor, stdin and stdout are not available. For a pkgadd to run in this mode, a private version of the admin file is required. My mtb-pkgadd script was:

 #!/usr/bin/ksh93

ADMIN_FILE=/tmp/admin.$$
cat << \ADMIN_EOF > $ADMIN_FILE
mail=
instance=unique
partial=nocheck
runlevel=nocheck
idepend=nocheck
rdepend=nocheck
space=nocheck
setuid=nocheck
conflict=nocheck
action=nocheck
networktimeout=60
networkretries=3
authentication=quit
keystore=/var/sadm/security
proxy=
basedir=default
ADMIN_EOF

pkgadd  -a ${ADMIN_FILE}  -d /brmnas/nadkarni/slim_wses/gui-changes/packages/i386/nightly-nd -R /export/bld_images/mtb/build_data/pkg_image SUNWinstall

That is really all that is need to apply the changes. Since I wanted to verify that the pkg was applied correctly, I decided to stop after mtb-pkgs step.

distro_const build -p slim-im-mod mtb.xml

DC stops at the start of the step. Running

distro_const build -l mtb.xml
 
Step           Resumable Description
-------------- --------- -------------
im-pop             X     Populate the image with packages
im-mod             X     Image area modifications
mtb-pkgs           X     Add MultiTB pkgs
slim-im-mod              Slim CD Image area Modifications
br-init                  Boot root initialization
slim-br-config           Slim CD boot root configuration
br-config                Boot root configuration
br-arch                  Boot root archiving
slim-post-mod            Slim CD post bootroot image area modification
grub-setup               Grub menu setup
post-mod                 Post bootroot image area modification
iso                      ISO image creation
usb                      USB image creation

I verified the bits and then resumed the build process by running:

distro_const build -R mtb.xml

 

Call me the "late Latif".  Most people blog about a release when it's released.  I blog about it when it's aged a bit.   After months of hard work it was great to finally deliver 2008.11.  A functionality that has not been discussed in the blogs thus far is  the support for booting off a multi-terabyte disk.   While support for multi-terabyte non boot disk has been available for quite a while, boot disks were limited to under 1TB. Boot disks can now be upto 2TB. This feature along with the prototype version of the Automated Installer should interest users in the enterprise space who have been waiting for jumpstart type functionality from the Caiman project for a while.

Another component delivered by the Caiman project is the Distro Constructor(DC).  This tool enables one to create liveCD or Automated Install (AI) images.  It is preferable to run this tool on a reasonbly powerful machine as it does
require a fair bit of compute power.

 When I first tried to build it on my Toshiba M9, I noticed that I could keep my coffee warm by the heat it was generating. My desktop, a W2100z was no better. Building images on this
system upped the decibel level of fan noise to that of a jet engine.  Luckily, I was able to snag an Ultra 24.  Its quiet and efficient. I can build images in 20-30 minutes on this puppy.
Often its even faster since I use DC's snapshoting capability

I view building ISOs as BDC (Before DC) and ADC (After DC). In the BDC world, one used "cdkit". It required a complex setup and assummed a very specific environment. Even users inside Sun would have to struggle to create an ISO.  Distro Constructor was designed to eradicate the "fear" of trying to build an ISO and I feel this goal has been met.

So go ahead fire up those fans, use  DC and send us your feedback either on caiman-discuss@opensolaris.org
or by filing bugs for new features.

Over the last couple of months I have been mentoring Chris Walsh and monitoring the sparc port.  It  was a lot of fun.  Chris's dogged persistence on getting this to work is remarkable.  He has blogged his trials and tribulations at
http://blogs.sun.com/cwalsh/entry/porting_opensolaris_project_indiana_to  and http://blogs.sun.com/cwalsh/entry/porting_opensolaris_project_indiana_to1

Chris will be giving a talk on this at the FROSUG meeting in August where you can see a demo.

Six easy steps to  recover your scrozzed system in  OpenSolaris Developer Preview 2.[Read More]

OpenSolaris Developer Preview - an halloween event!!

 

[Read More]

The long overdue and promised Solaris Volume Manager Internals is finally available  here.
The contents were developed as a tutorial for internal engineers and it was a week long course. 

Enjoy.

[ T:OpenSolaris  Solaris]

Work on the new installer for OpenSolaris has begun.  See how you can participate.[Read More]

The single malt Ledaig project was putback recently. [Read More]

A while back we finally started a project to enable friendly names for SVM volumes.  As users of SVM know, the volume names until now could only be dxxx and the names for hotspares had to be hspxxx.  We needed to fix that.  So as  is typical for our group the project that delivered it was called  bunnahabhain (prounced as bunahavan).  Yes another single malt scotch.

This project was putback in Nevada build 37.  For more info look here.

[ T:OpenSolaris  Solaris]

I am really excited to announce that frkit is now available on the opensolaris laptop community. We are still fighting the legalities of getting things out.   But that is improving significantly.  Over the next few months, expect to see many more modules, including the NDIS wrapper.

Network configurator for laptop

I have followed the trend of many nascent bloggers by being silent for a long time.  So what have I been doing.  Aside from doing the normal work, I have been trying to get things organized so that we can start putting tools and utilities that make a Solaris laptop a bit more real.    Internally we have been using a number of tools delivered by frkit.  Now the plan is to make frkit available on laptop community for OpenSolaris.  The community is now open and a number of people are contibuting.

Inetmenu is one of the tools used internally to configure network interfaces.  Out of the box, Solaris and OpenSolaris don't deal well with nomadic systems like laptops.   One typically has install Solaris as a stand alone system.  Then after booting up, the user has to manually plumb the interface,  start DHCP  and then fix up the files required  for  DNS or NIS.   Inetmenu is a script that does all that.  It handles, wired, wireless and  dial up interfaces.   One can define profiles for use.   All these make life a bit nicer for laptop users.

However Inetmenu is just one of the network configurators.   There is yet another one that is used internally called netprof and hopefully we should be able to have that available in not too distant future. 

Technorati Tag: OpenSolaris

Optimized Resyncs in Solaris Volume Manager

Over the last couple of months, a number of people wanted to know about optimized resyncs. People familiar with VxVM, might know this as DRL (Dirty Region Logging).  In Solaris Volume Manager this functionality is called  optimized resyncs.  Optimized resync in Solaris Volume Manager is implemented using resync regions(RR).

The function of DRL or RR is to ensure that a mirror is consistent in event of a crash.   Consistency does not mean that the mirror will contain up to date information.   What is guaranteed is that  a read request of the same block from any side of a mirror returns the same data. For example,  if block 10 is read from a 2 sided mirror, the data returned must be identical whether it is supplied from side 1 or side 2 of that mirror.

When parallel writes to a mirror are enabled, a window exists where a system may die before writes to all sides of the mirror are completed.   To ensure the mirror is consistent in event of a crash, a simple implementation might be to choose one side of a mirror and copy its contents to all the other sides when the system boots up.  This obviously is not very efficient.   A smarter approach is to track regions in which writes occurred and resync only those regions.   SVM uses this technique.     SVM divides a mirror into 1001 regions (max). This is maintained as an incore bitmap and in the mddb. When a write request arrives at the mirror strategy routine, it has the block number and length. From this information the  impacted regions are computed.  Prior to issuing a write the incore bitmap region is checked to see if the region has already been marked dirty.  If not the incore bitmap is updated.   An asynchronous resync kernel daemon thread monitors this bitmap every few seconds and writes it out to the mddb if required. After the RR bitmap is flushed to the mddb, the bitmap is reset. On boot up  svc:/system/metainit:default  starts the resync kernel threads.  There is one thread per mirror.   The resync thread scans the mddb and only the regions that are marked dirty are resynced.  When a machine is shutdown cleanly,  the bitmap is zeroed out and  no resync occurs when starting up.

 In the mddb, the resync bitmap is called the resync record.  Every mirror has two resync records associated with it.  To reduce hot spots, the resync records are spread across multiple mddbs.  That is, if one has 2 mirrors and 4 mddbs, then the resync record for one mirror will be on mddb1 and mddb2. For the second mirror the resync record will be on mddb3 and mddb4.  The actual algorithm for resync record placement is a bit more sophisticated.

One can get metastat to display the location of the resync regions for the mirror.

   
#metadb
        flags           first blk       block count
     a        u         16              8192            /dev/dsk/c1t1d0s7
     a        u         16              8192            /dev/dsk/c1t0d0s7
     a        u         16              8192            /dev/dsk/c1t2d0s0

# export MD_DEBUG=STAT

# metastat d10

d10: Mirror
    Submirror 0: d0
      State: Okay         Wed Jun  1 19:53:10 2005
    Submirror 1: d1
      State: Okay         Wed Jun  1 19:53:10 2005
    Pass: 1
    Read option: roundrobin (default)
    Write option: parallel (default)
    Size: 67094528 blocks (31 GB)
    Regions which are dirty: 34% (blksize 67094 num 1001)
    Resync record[0]: 0 (/dev/dsk/c1t1d0s7 16 8192)
    Resync record[1]: 1 (/dev/dsk/c1t0d0s7 16 8192)

d0: Submirror of d10
    State: Okay         Wed Jun  1 19:53:10 2005
    Size: 67094528 blocks (31 GB)
    Stripe 0:
        Device                             Start Dbase State       Reloc Hot Spare Time
        /dev/dsk/c3t50020F23000100F7d9s0       0 No    Okay        Yes             Wed Jun  1 19:52:53 2005


d1: Submirror of d10
    State: Okay         Wed Jun  1 19:53:10 2005
    Size: 67094528 blocks (31 GB)
    Stripe 0:
        Device                              Start Dbase State       Reloc Hot Spare Time
        /dev/dsk/c3t50020F23000100F7d10s0       0 No    Okay        Yes             Wed Jun  1 19:53:04 2005


Device Relocation Information:
Device                            Reloc Device ID
/dev/dsk/c3t50020F23000100F7d9    Yes   id1,ssd@n60020f20000100f740336c7b00023087
/dev/dsk/c3t50020F23000100F7d10   Yes   id1,ssd@n60020f20000100f740336ca20001241b

In the output above, notice that the resync regions are spread across 2 mddbs.  I was running newfs on the mirror and therefore it  shows that 34% of the regions are dirty.   The blksize refers to the size of the resync region.    If you were monitoring the iostat output for an active mirror, you would notice that the disks that contain the mddbs are being written to.  These writes are due to the periodic updates of resync region bitmaps to the mddb.



Technorati Tag: OpenSolaris
Technorati Tag: Solaris