Consider this. Since I joined CFS and Lustre, my job stack has been unwinding.

Sun bought Cluster FileSystems (Lustre) and I returned back to Sun. Now Oracle is on the verge of buying Sun and I might be back with Oracle too! It is quite funny.

Looks like Opensolaris is all set for its official release. The package repository seems to be all loaded and ready.

I had OpenSolaris DP2 on my laptop. And upgrading it to 2008.05 was as simple as running th upgrade script - a piece of cake!

Regardless of one's take on the Opensolaris naming controversy, this distro rocks!

I had earlier posted a demo of installation of lustre on a single node.

But most, if not all production installations of lustre are on multiple nodes - tens of Object Storage Servers (OSS) and thousands of clients.

Here is a demo of a multi node setup - though on a much lesser scale - one Meta Data Server (MDS), two OSS and a single client. And I show this on VMWare. I take three virtual machines on which lustre 1.6 packages are installed. (See earlier post on how to do this.
Lustre Installation

Here is what the demo does:

A few days ago, I was putting together a demo of lustre for some teams in Sun interested in Lustre. Then yesterday I got another request for a demo, this time at a college in Tamil Nadu. This got me interested in producing a flash demo that can be reused.

So, with the help of vnc2swf and tightvnc, here is what I put together. On a virtual machine running CentOS 4.4 on my laptop, I show the setting up of an MDS, a OSS and a client on a single node. The filesystms are on top of loop devices.

Lustre Installation

Lustre Configuration
[Note: Right click and select 'Play' to play the demo.]

Here is what the demo does:

$ wget http://downloads.lustre.org/public/lustre/v1.6/latest//kernel-lustre-smp-..rpm
$ wget http://downloads.lustre.org/public/lustre/v1.6/latest//lustre-..rpm
$ wget http://downloads.lustre.org/public/lustre/v1.6/latest//lustre-ldiskfs-..rpm
$ wget http://downloads.lustre.org/public/lustre/v1.6/latest//lustre-modules-..rpm

$ mount -t lustre ${MDS}@tcp0:/lustre /mnt/lustre

As you can see, it is quite simple - a big change from the lustre 1.4.x days!
Download lustre packages from here

In my earlier post, I described how I setup Indiana on my laptop.

I had two must haves for my work - virtual machines, and support for my ethernet card. My Indiana Preview 2 installation met both; well sort of...

I installed VirtualBox and then went on to install OpenSolaris build 84 and CentOS 4.4 on virtual machines. While VirtualBox is a bit different from VMWare, the installation was a piece of cake.

Setting up NAT took care of Internet access for the VMs. The NAT interfaces use DHCP. Piece of cake again.

Then my troubles started.

I need to SSH in to my VMs. On VMWare it is as simple as adding host-only network cards to the virtual machines. The host is automatically on this network. The VMs can also talk to each other over this network.

On VirtualBox, things are a bit different. The host can talk to the guests (AFAIK) only by setting up port forwarding. This is explained in the VirtualBox User Manual.

The following commands did it for the 'osol' virtual machine. I could ssh and sftp in!

$ VBoxManage setextradata "osol" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/Protocol" TCP
$ VBoxManage setextradata "osol" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/GuestPort" 22
$ VBoxManage setextradata "osol" "VBoxInternal/Devices/pcnet/0/LUN#0/Config/guestssh/HostPort" 2222

$ ssh -p 2222 root@localhost
$ sftp -o port=2222 root@localhost

See also Renaud's post.

If you have multiple VMs, use different host ports (2222, 2223 etc).

One problem solved. However, the VMs still can't talk to each other. To do that I needed InternalNetwork. This is also explained in the UserManaual.

Setting it up is easy. Using CLI, this is what I did.

$ VBoxManage modifyvm osol -nic2 intnet
$ VBoxManage modifyvm osol -intnet2 intnet
$ VBoxManage modifyvm rhel -nic2 intnet
$ VBoxManage modifyvm rhel -intnet2 intnet

That's it. In theory this should have just worked. However it did not.

I spent a couple of days fiddling around with this. I could ping the other node at times. But once I try to ssh in, ssh fails and ping fails thereafter!

To isolate the problem, I made a copy of the linux VM and InternalNetwork worked fine between the linux VMs. That suggested, I was not goofing up with the VirtualBox setup.

Next, I tried with two OpenSolaris VMs. The same problem showed up again - I could ping but ssh failed. After this failure, ping also failed.

This suggests a bug somewhere. But hey, VirtualBox is still beta on OpenSolairs. I am hopeful that this issue will get sorted out soon.

But till then, I am going back to Ubuntu. Bye bye OpenSolaris! :'(

OpenSolaris and VirtualBox

I used to use OpenSolaris on my workstation while I was with the Sun Cluster group. When I moved to Cluster Filesystems, I got a new Dell laptop and I moved to Ubuntu out of necessity - I needed to use virtual machines (VMWare Server) for my work.

I have been looking for a way of moving to OpenSolaris and have been eying Xen as a possible replacement of VMWare. But the traffic on xen-discuss@opensolaris.org suggested enough difficulties and I kept away.

VirtualBox on OpenSolaris gave me a new reason to try things out. So, I decided to throw away Windows Vista (given to me, thanks to Dell) and decided to take OpenSolaris for a spin.

OpenSolaris Developer Preview 2 vs Solaris Express

I have used OpenSolaris and Solaris on Virtual machines and on workstations. So I am not a newbie. I had two options - Solaris Express or Indiana. Given that it is easier to upgrade Indiana (thanks to IPS a.k.a pkg(5)) and having tested it on a virtual machine, I decided on Indiana.

Installation of Indiana

So, I burnt the iso on to a CD and booted from it. Partition editing is disabled in preview 2. And certainly did not want to use the entire hard disk. So, I booted back into Ubuntu and changed the partition type of the Vista partition to Solaris and booted back to Indiana LiveCD.

Now the Solaris partition was recognized. And the rest of the installation was a piece of cake.

Grub entries for Ubuntu (and ext3 read only support)

Installation did not make entries for Ubuntu. And the default page on Firefox told me to get the entries from the Linux partition. Well, time for a resuce CD to access the Linux partitions? No! Time to install read-only support for ext3/ntfs.

# wget http://www.genunix.org/distributions/belenix_site/binfiles/FSWfsmisc.tar.gz
# gzip -cd FSWfsmisc.tar.gz | tar -xvf -
# pkgadd -d . FSWfsmisc
# mount -F ext2fs /dev/dsk/c7d0p1 /mnt/lin1
# cat /mnt/lin1/boot/grub/menu.lst

Copied the grub entries that I wanted to /rpool/boot/grub/menu.lst and I was done.

Thanks Moinak and Martin Rosenau!

Network drivers

Indiana had no trouble supplying drivers for my wireless network. But the Ethernet driver was not supported. I do have a wireless router. And nwam configured it just fine. But I did want the LAN to be functional as well. And thanks to Masayuki Murayama and Moinak Ghosh , it was as simple as

# wget http://blogs.sun.com/moinakg/resource/nicdrv.tar.bz2
# bzip2 -cd nicdrv.tar.bz2 | tar -xvf -
# cd nicdrv
# for i in `ls -d SFE*`; do yes | pkgadd -d . $i; done
# ifconfig -a plumb

VirtualBox

Installing Virtual Box was a piece of cake

# wget http://virtualbox.org/download/testcase/VirtualBox-opensolaris-x86-1.5.51-r28414-beta1.tar.gz
# gzip -cd VirtualBox-opensolaris-x86-1.5.51-r28414-beta1.tar.gz | tar -xf -
# pkgadd -d VirtualBox-1.5.51-SunOS-x86-r28414
# VirtualBox

I had some trouble launching Virtual Box though.

No 64-bit VirtualBox

I have a 64 bit system and the VirtualBox vboxdrv driver is 32 bit only. So I had to boot into 32 bit. So, I added a new grub entry:

  title OpenSolaris Developer Preview 32 bit
  kernel$ /platform/i86pc/kernel/unix -B $ZFS-BOOTFS
  module$ /platform/i86pc/boot_archive

Now the driver vboxdrv loaded just fine. 'add_drv vboxdrv' confirmed this.

But still VirtualBox failed to start a virtual machine with the message:

   VBox status code: -1909 (VERR_VM_DRIVER_NOT_ACCESSIBLE).

Need to be root user! But "su -" refused to connect to X. Setting DISPLAY, "xhost +", ForwardX11 etc did not fix the issue. Google came to the rescue again. Doing the "su" instead of "su -" did the trick.

Sound driver

My laptop has a 82801G (ICH7 family) Intel audio controller. And OpenSolaris does not support it yet. No problem, OSS has drivers for it!

# wget http://www.4front-tech.com/release/oss-solaris-v4.0-1014-i386.pkg
# pkgadd -d oss-solaris-v4.0-1014-i386.pkg

However, though osstest produces sound, the audio devices do not show up in the sound preferences. This is a problem to be investigated for another day.

Blastwave

Blastwave has a huge repository of packages for Solaris. So, I pkg-get a piece of it.

# pkgadd -d http://www.blastwave.org/pkg_get.pkg
# /opt/csw/bin/pkg-get -i xchat
# /opt/csw/bin/pkg-get -i xemacs

This should become simpler once the pkg(5) version of blastwave is unleashed.

Now, I am installing CentOS 5 on a virtual box!

After about 3 years with Sun Microsystems, I moved to work for Cluster Filesystems Inc. to port Lustre to Solaris. Half-way through the project, Sun decided to buy CFS. And that brings me back to Sun.

At Sun, they call this a boomerang.

It was Diwali. And I was brooding over an imagined slight. It was a little past 10 PM and I was walking along the SG Palaya main road - walking briskly, trying to work off the anger. Something I do quite often by the way.

It was a bit chilly; usual for this time of the year. That's when I noticed what looked like a ball of rags on the side of the road. It was actually a beggar woman sleeping with just her sari drawn over her head to protect her from the cold. I might not have recognized her, except for the fact that I had seen her a couple of times, sitting and begging at that very same spot. On both occasions, I had refused to give her money - something I always do.

I probably have passed by similar homeless folks dozens of times and not given it a second thought. Don't know why, maybe it was the way I was feeling then that made me think. Here I was, with a roof over my head, financially stable, lots of things going for me, yet brooding. And there is this lady, probably old enough to be my grand mother sleeping by the side of a road.

It was past the deadline set by the city police for bursting fire crackers. People paid no heed and were having fun all over that place. Money being burnt for fun and this lady had no place to sleep.

I could take her to my house; nothing stopping me from doing that. But I didn't. I just walked on. On my way back, I took this snap, trying to act like I was read a message on my mobile. I needing have bothered. There was no one watching...

The day after I found her begging and parted with a five rupee note. I knew I hadn't made her life better, hadn't changed a thing, and perhaps should have done better...

-Manoj

I am Manoj Paul Joseph. I am with the Sun Cluster development team. I work from Bangalore, India. I work with the Global Data and Devices (GDD) group. I work on DCS, PxFS and Sun Cluster SMF services... I work on all of that but my heart is with PxFS.

Before joining Sun, I had a short stint with Oracle and an even shorter one with iNautix Technologies.

That's me on the left.

That's all for now!

-Manoj