I have spent some times playing with the port of Linux to LDoms that Dave Miller and Fabio have done. Dave has provided us a Linux disk image and I initially had some hard time trying to boot from this image: first because it has been a long time since haven't played with Linux especially on Sparc, second because I was trying to understand the different tricks Dave and Fabio are describing to deal with disk labels.

Linux on UltraSPARC-T2

Anyway, it eventually works and it works fine. You can see the result with a demo on Ash's blog. And note that the demo and the tests have been done on a system with an UltraSPARC-T2 processor, so Linux does work with the UltraSPARC-T2. Here is a log of the boot sequence:

{0} ok boot
Boot device: rootdisk  File and args: 
SILO Version 1.4.13
boot: linux.2623
Allocated 8 Megs of memory at 0x40000000 for kernel
Loaded kernel version 2.6.23

Remapping the kernel... done.
OF stdout device is: /virtual-devices@100/console@1
Booting Linux...
[585488.953894] VIO: Adding device channel-devices
[585488.954061] VIO: Adding device vnet-port-0-0
[585488.954202] VIO: Adding device vdc-port-0-0
[585488.954350] VIO: Adding device ds-0
... snip ...
 * Running local boot scripts (/etc/rc.local)      [ OK ]

Ubuntu gutsy (development branch) t2k-linux1 ttyS0

t2k-linux1 login: root
Password:
Last login: Fri Aug 10 10:48:13 2007 on ttyS0
Linux t2k-linux1 2.6.23-rc1 #1 SMP Sun Jul 29 21:19:34 PDT 2007 sparc64

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@t2k-linux1:~# uname -a
Linux t2k-linux1 2.6.23-rc1 #1 SMP Sun Jul 29 21:19:34 PDT 2007 sparc64 GNU/Linux

root@t2k-linux1:~# grep CPU /proc/cpuinfo | wc -l
60

root@t2k-linux1:~# cat /proc/cpuinfo
cpu		: UltraSparc T1 (Niagara)
fpu		: UltraSparc T1 integrated FPU
prom		: OBP 4.27.0.build_03***PROTOTYPE BUILD*** 2007/07/27 18:48
type		: sun4v
ncpus probed	: 60
ncpus active	: 60
D$ parity tl1	: 0
I$ parity tl1	: 0
... snip ...
MMU Type	: Hypervisor (sun4v)
State:
CPU0:		online
CPU1:		online
CPU2:		online
CPU3:		online
... snip ...
CPU57:		online
CPU58:		online
CPU59:		online

Linux identifies the processor as an UltraSPARC-T1 but this is really an UltraSPARC-T2. The evidence is that the UltraSPARC-T1 has only 32 threads and here we have a Linux domain running with 60 (yes 60!) cpus. The UltraSPARC-T2 has 64 threads and this system was configured with a primary domain running Solaris with 4 cpus and a guest domain running Linux with 60 cpus. Note that we have to use a Linux 2.6.23 kernel to be able to boot the UltraSPARC-T2 processor.

Linux Domain bind/start Tricks

When you have a Linux disk image, Dave Miller and Fabio mention some tricky steps to be able to boot from that disk image because the LDoms virtual disk server mangles partition tables.

Here is a simpler procedure: let say you have a Linux disk image /ldoms/disklinux and you have configure the domain linux-domain to use that image. Then if you just do a "ldom bind" and "ldm start" of the linux-domain, Linux will not boot correctly. What you need to do is:

• Backup the disk label of the Linux disk image:

# dd if=/ldoms/disklinux of=/ldoms/labellinux count=1

• Erase the disk label of the Linux disk image:

# dd if=/dev/zero of=/ldoms/disklinux count=1 conv=notrunc

• Bind the Linux domain:

# ldm bind linux-domain

• Restore the original disk label of the Linux disk image:

# dd if=/ldoms/labellinux of=/ldoms/disklinux count=1 conv=notrunc

• Start the Linux domain:

# ldm start linux-domain

And then you can start Linux.

Why do we need to do that? On the Linux disk image, you will have a fake Sun VTOC disk label that defines 0 partition. If you directly bind and start the Linux domain with this disk label then the virtual disk server will read the label and, accordingly, it will see that no partition is defined. Then later, when Linux starts, it will request the virtual disk server to read from slice 2, but as no partition is defined the virtual disk server will return an error and Linux will be unable to read from the disk.

When we erase the disk label and bind the domain, the virtual disk server will create a default partitioning with partition 2 representing the entire disk. After the domain is bound the virtual disk server will not read the disk label again so the original label can be restored. Then when Linux will read from slice 2, there will be no problem because the virtual disk server now knows about slice 2.

This will be improved with some next version of the virtual disk server driver and probably a change in the Linux virtual disk so that none of these tricks are required to start a Linux domain.