OS Technology

Monday May 16, 2005

CTO Advisory Council

Sun's CTO, Greg Papadopolous, invited me to a customer advisory council event that he held a couple of weeks ago. This was similar to a customer advisory council that Jonathan Schwartz wrote about here. The interesting twist here was that the attendees were a mix of CTOs from Sun's business units and equivalent CTO/CIOs from some of Sun's largest customers.

It was fascinating to hear what things really keep our customers up at night, and their perceptions of our technology strengths and gaps. For our part, we discussed different aspects of our technology, including processors, platforms, operating systems, middleware and the Sun Grid.

Reactions varied - but apart from occasional surprises ("I didn't know Sun even did [this product or that service] .."), we heard the following (paraphrased) question "but why are you investing in [this component]?" This theme was particularly evident when we talked about the capabilities of each area, isolated from the others. Perhaps the way we presented our technologies reinforced the common misperception that all computer system components are now just commodities," i.e. "done," at least as far as techological innovation goes.

But by the end of the day, the customer CTOs seemed to have internalized our set of investment areas; and had a clearer idea of how each component investment related to the other, and thus improved the effectiveness of the whole offering. A great example was our CMT processor technologies, ambitious new platform designs, Solaris 10 threading and related observability capabilities like DTrace and plockstat, and so on up the stack to the Sun Grid. And then we reached one of those "Eureka!" moments, when I think the room did the synthesis step, and rediscovered the intrinsic value of a systems company - that takes an integrated approach to the coevolution of all the components of an information processing system.

Then after a brief moment basking in our little ray of sunshine, we were quickly taken to task for not communicating that message to the marketplace effectively! Sun is a company that lives and breathes innovation, and tends to focus on detailed technology messages. But for many of these CTOs and CIOs, their executive staffs need simpler, more direct, messages that make sense to less technical people.

Everyone is searching for simplicity, in particular, simple answers to complex problems, yet the levels of abstraction spanned by modern operating systems makes their implementation intrinsically complex, and it's easy to get lost in the detail. For my organization, and the other members of the OpenSolaris community, our biggest challenge may simply be communicating the full value of Solaris 10 and OpenSolaris to a broad audience - both as a set of component technologies, and as a systems technology.

One of the participants suggested that Sun needs to find a "great communicator" who can take our technology portfolio, and translate it into something more accessible to non-technical people, to really get our messages and core values across.

That's an interesting idea, and one that's certainly set us thinking.

Posted at 10:54PM May 16, 2005 by tpm in General |

Wednesday May 11, 2005

Solaris 10 on x64 Processors: Part 2 - Getting Started

Booting and Startup
Some of the trickier issues with porting Solaris to a new platform architecture originate from some of the decisions we made 15 years ago. This is a complex story, and one that we're soon to drastically improve on x86/x64 systems with newboot , (as demoed recently by Jan Setje-Eilers at the first OpenSolaris User Group meeting) but I'll try to relate the minimum needed so that you understand why we took the path that we did for Solaris on x64 processors.

The Solaris 2 booting system was originally designed back in 1991 in the context of the problems and issues we had in the world of SunOS 4.x, our desires to have the hardware and software organizations in the company execute relatively independently, as well as to support the then-blossoming clone market for SPARC workstations. We wanted to enable both ourselves and 3rd parties to deliver support for new hardware without having to change the Solaris product CD bits - except by adding new driver modules to it. Also remember that speeds and feeds were two orders of magnitude slower then, and that kernel text was a larger proportion of the minimum memory size than it is today so we had to be far more selective about which modules to load as we booted.

The design we came up with starts with the primary bootstrap loading the secondary booter, and the secondary booter putting the core kernel into memory. The kernel then turns around and invokes various services from the secondary booter using the bootops interface to allow the kernel to discover, load and assemble the drivers and filesystem modules it needs as it initializes itself and explores the system it finds itself on. Once it determines it has all the modules it needs to mount the root filesystem, it takes over IO and memory allocation, mounts the root, and (if successful) continues to boot by loading additional kernel modules from that point on.

Note that this early part of the boot process starts out with the secondary boot program being the one true resource allocator i.e. in charge of physical memory, virtual memory and all I/O, and ends with the kernel being that resource allocator at the end. While moving from the former state to the latter sounds simple in principle, it's quite complex in practice because of the incremental nature of the handoff. For example, the DDI (device driver interfaces) aren't usable until the kernel has initialized its infrastructure for managing physical and virtual memory. So we have to load the modules we might need based on the name of the root device given to us by the booter which in turn comes from OpenBoot. Kernel startup somehow has to get most of the VM system initialized and working, yet still allow the boot program and its underlying firmware to do I/O to find the drivers and filesystem modules it needs from the boot filesystem to mount the filesystem. Practically speaking, this entails repeated resynchronization between the boot program and the kernel over the layout and ownership of physical and virtual memory. In other words, the kernel tries to take over physical and virtual memory management while effectively avoiding conflicting with the secondary booter and the firmware using the same resources. It's really quite a dance.

For the x86 port, a similar approach was used, using real-mode drivers that were placed onto a boot floppy as an analogue of the OpenBoot drivers in the SPARC world to construct a primitive device tree data structure analogous to the OpenBoot device tree. (In 1995, for the PowerPC port, we implemented "virtual open firmware" which was an even closer simulation of OpenBoot to make it easier to reuse SPARC boot and configuration code).

Note that the x86 secondary boot program itself runs in protected mode like the kernel; it is responsible for switching to real-mode and back to run the real-mode drivers.

Six years go by ...

During that time, specifically in Solaris 2.5, we made things even more complicated for hardware bringup by splitting the basic kernel into separate modules: unix, genunix and the kernel linker krtld; these make bringup more difficult because genunix and krtld are not relocated until run-time, thus diagnosing hexadecimal addresses where the kernel has fallen over in genunix or in krtld becomes a significant pain, absent a debugger like kmdb or its predecessor kadb.

Now, in 1997 when we created 64-bit Solaris for SPARC, it was relatively simple to make a 64-bit boot program use the OpenBoot interfaces; there is no "mode" switch between operating a SPARC V9 processor in "32-bit" mode or "64-bit" mode - apart from the address mask, the rest of the difference was entirely about software conventions, not hardware per se. So we didn't really have to do anything to the basic boot architecture, this part of the 64-bit Solaris project on SPARC was really quite straightforward, mostly a matter of getting the boot program LP64-clean.

Meanwhile, in late 2003 ...

The Opteron architecture presented us a far more difficult challenge because the processor needed to switch to long mode via an arcane sequence of instructions including switching between different format page tables and descriptor tables. Worse still, the 64-bit Solaris kernel on Opteron would need to turn around and invoke what it would think of as a 64-bit boot program running in long mode in order to fetch modules from the disk (and as discussed above) invoking real-mode code and the BIOS to do so!

Our initial approach was to use the existing, unmodified, protected mode 32-bit booter and have it boot an interposer called vmx that used the 32-bit booter to load the 64-bit kernel into double mapped memory. In this case, "double mapped" means that there's an (up to) 4Gbyte area of physical memory that are (a) mapped by 32-bits of VA in the protected mode page tables and (b) mapped by the bottom 32-bits of VA and the top 4G of VA in the long mode page tables. The interposer then pretended to be a 64-bit booter to the 64-bit kernel. When the 64-bit kernel asked vmx for a boot service via one the bootops vector (fortunately a relatively small and well-behaved interface), vmx quickly switched back to protected mode, then after massaging the arguments appropriately, invoked the bootops of the 32-bit, protected mode booter to provide the service. That service would in turn often result in the protected mode booter switching the processor back to real mode to deliver it. Though our colleagues at AMD winced at the thought of the poor processor switching back and forth hundreds or thousands of times through so many years of x86 history, Opteron didn't mind this at all.

Finally, before we delivered the code to the Solaris 10 gate, William Kucharski, who took the half-thought-out prototype and made all this insanity actually work correctly, integrated the vmx code inside the 32-bit booter so this component is invisible in the final product.

What else could we have done?

We could've made the boot program be completely 64-bit, and thus have that program deal with the mode switching from long, to protected to real to invoke the BIOS and back again. While possible, it would've involved porting a bunch of code in the x86 boot program to LP64, and reworking both protected-mode and real-mode assembler in the boot program to fit into an ELF64 environment. The latter seemed like a lot more work than we wanted, even if we'd somehow managed to convince the assembler and linker to support it.

Another suggestion was to somehow jump into the 64-bit kernel and do magic to allow us to call back to 32-bit boot services there. But that seemed to us to be just a matter of finding a different place to put the code we had in vmx; we thought putting it into boot where it will be eventually reused by the system was better than making that ugliness live in the kernel proper.

The other option on the table was to become dependent on the newboot project which we were planning at the time to bring us into the modern world of open source kernel booting; but we were unwilling to wait, or to force that dependency to be resolved earlier because of schedule risk.

Descriptor Tables and Segmentation

It quickly becomes apparent to students of the x86 architecture that with x64, AMD tried hard to preserve the better parts of the x86 segmentation architecture, while trying to preserve compatibility sufficient to allow switching to and from long mode relatively painlessly. But to the kernel programmer, it only seems to get more complicated. In previous versions of Solaris, we used to build the descriptor tables (almost) statically, with a single pass over the tables to munge IDT and GDT structures, from a form easy to initialize in software, into the form expected by the hardware. Early on, we realized it was worth bringing this up-to-date too, so we discarded what we had (for both 32-bit and 64-bit Solaris) and used the FreeBSD version which use a series of function calls to build table entries piece by piece.

Some of our more amusing early mistakes here included copying various IDT entries for machine exceptions as "trap" type exceptions instead of interrupt-type exceptions which caused real havoc when an interrupt would sneak in before the initial swapgs instruction of the handler. All terribly obvious now, but less than obvious at the time.

One optimization we made was to exploit the fact that the data segment descriptor registers %ds and %es were effectively ignored in long mode. Further, %cs and %ss were always set correctly by the hardware, %fs was unused by the kernel, and %gs has a special instruction to change the base address underlying the segment. Taken together this lead us to a scheme of lazy update of segment registers; we only update segment registers on the way out of the kernel if we know that something needs to be changed.

Next time I'll try and touch on the rest of the work involved with the kernel port proper.

Technorati Tag: OpenSolaris
Technorati Tag: Solaris

Posted at 11:34PM May 11, 2005 by tpm in Solaris | Comments[1]

Tuesday May 10, 2005

The Xen Summit

A few weeks ago I attended the Xen summit in Cambridge, UK. Xen is an open-source hypervisor project being driven by Ian Pratt from the University's Computer Laboratory. Xen is attracting a lot of interest as the de facto open source hypervisor for commodity hardware.

Xen is designed to be a thin layer of software which allows multiple kernels to run on a single machine. Among the many cool things this new layer of virtualization allows is OS checkpoint and resume, which the Xen team have used to good effect in their workload migration experiments. (I was at a Linuxworld BOF last March where the sound of jaws dropping as Ian presented their results was quite evident!) Take a look at the papers on their website - it's pretty cool stuff.

Xen is based on paravirtualization; that is, you have to make some changes to the low-level kernel to allow the OS to run. This is both because that was easier to do on existing x86 hardware, and more importantly, it's also better performing than other approaches.

Anyhow, we've been looking at Xen for about a year now, and recently a few of us have been working on a prototype port of Solaris to Xen on the x86 architecture. We're planning to make it work on x64 machines where we can exploit the new hardware virtualization technology as it becomes available. We're also planning to make the Solaris side of things into an OpenSolaris community project too; particularly since Xen is itself an open source project. Although we're still working on the mechanics of all that, I'd like to hear from people who want to participate.

Update
Some of the comments below imply that you might think I'm only interested in help from kernel developers. I'm also interested to here from people who are already using Xen, and are prepared to experiment with, and give us feedback on, early alpha-class builds of Solaris on Xen too.

Another Update
The OpenSolaris on Xen community is now up: see the OpenSolaris web site to participate.

Technorati Tag: OpenSolaris
Technorati Tag: Solaris

Posted at 02:59PM May 10, 2005 by tpm in Solaris | Comments[12]

Solaris 10 on x64 Processors: Part 1 - Prework

SSE Support
Back in 2002, after we had resurrected Solaris on x86, we realized that we needed to get back to basics with a number of core kernel subsystems, because while we'd been slowly disinvesting in Solaris on x86, the x86 hardware world had been scampering off doing some really interesting things e.g. the Streaming SIMD Extensions (SSE) to the instruction set, and introducing fast system call mechanisms. We also knew that these were basic assumptions of the 64-bit architecture that AMD was working on, so we started work on the basic kernel support which allows the xmm and mxcsr registers to be part of the state of every Solaris lwp. At the same time, Matt Simmons helped out with the disassembler and debugger support for the new instructions. This didn't take too long, and the work was integrated into Solaris 10 in October 2003, and Solaris 9 Update 6. One of the immediate benefits was Java floating point performance which used SSE instructions on capable platforms, and on the right hardware, Solaris was now one of those platforms!

Fast System Calls
In earlier Solaris releases, system calls were implemented using the lcall instruction; we'd been faithfully doing this for years, without really noticing that the performance of call gates was falling further and further behind. For Solaris 10, we decided to make fast system calls work using the sysenter instruction that was first introduced on Pentium II processors. Because of some awkward limitations around the register usage of the sysexit instruction (in particular, dealing with system calls that return two values), plus our desire to run on older machines, we also keep the older lcall handler around too.

First, I should explain something about the way Solaris system call handlers work in general. As you can imagine, in a highly observable system like Solaris, we can, in some circumstances, end up doing a lot of work in the system call enter and system call exit path. But, most of the time, we don't actually need to do all the checks, so more than 10 years ago, one of my former colleagues restructured SPARC system calls to do a single test for all possible pre-work on the per-lwp thread variable called t_presys, and a single test for all possible post-system call handler work on another thread variable called t_postsys. The system call handler is then constructed assuming that the t_presys and t_postsys cases are rare - but if either t_presys or t_postsys is set e.g. by the current system call, previous system call, or via /proc, we handle the relevant rare case in C code, allowing us to code the fast path in a small amount of assembler. To summarize:

entry_point: if (curthread->t_presys) presys(); real_handler(); if (curthread->t_postsys) postsys(); return-from-trap sequence

Obviously the Solaris x86 architecture mirrored this to some extent, but the presys() and postsys() functions had been
partially rendered from C into assembler which was, as usual, difficult to understand, port and maintain, and wasn't even particularly fast. So the initial exercise was to turn the slow paths back into C code, and macro-ize the assembler code involved in performing the pre and post checks so that different syscall handlers could easily share code. Then I coded up a sysenter style handler, and we were pretty impressed with the results on our system call microbenchmarks.

Hardware Capability Architecture
All this kernel work was fun, but we didn't have a clear idea of how we were going to let libc use fast instructions on machines capable of handling them, and fall back to lcall on machines that couldn't. We also noted that when AMD processors are running in long mode, sysenter is not supported but syscall (similar but different) is.

Earlier attempts to introduce support for this facility had considered using either the libc_psr mechanism that we introduced in Solaris 2.5 for dealing with the fast bcopy instructions available on UltraSPARC platforms, or using the isalist mechanism. The former scheme assumes that the instruction set extensions were specific to a platform, while the latter implicitly assumes that there are a small set of instructions extensions that were additive, and acted to improve performance. However we realized that in the x86 world we weren't dealing with platform extensions so much as processor extensions, and that processor vendors were adding instruction set extensions orthogonally, so we'd be better describing each instruction set extension by close analogy to the way the vendors were describing them in the cpuid instruction i.e. via a bit value in a feature word. See getisax(3C) for programmatic access to the kernel's view; <sys/aux_386.h> contains the list of capabilities we expose.

What we ended up with is (currently) three copies of the libc binary compiled different ways; the basic version in /lib/libc.so.1 is able to run on the oldest hardware we support, the newer versions in /usr/lib/libc correspond to more modern hardware running on a 32-bit or 64-bit kernel. Thanks to Rod Evans the libraries are marked with the capabilities they require, and the system figures out which is the best library to use on the running system at boot time. Last November, Darren Moffat wrote something up about how the system configures which libc it uses; there's no point in repeating that here.

The Tool Chain
The other key piece we needed for the amd64 kernel was to make the Solaris kernel compile and run with the GNU C compiler and assembler, so I started work on that too. Note that wasn't because we didn't want to use the Sun compiler, it's just that it's easier to bring up an OS using a compiler that works, instead of debugging both the kernel and the compiler simultanously. More critically, there wasn't a Sun compiler that would build 64-bit objects at the time. GNU C is great for finding bugs, and really complimented the capabilities of the Sun compiler and lint tools. I got the 32-bit kernel working fairly easily, we were able to start the 64-bit project using this compiler, once we'd hacked up an initial configuration.

In the meantime, while we were completing and integrating some of these prerequisites into Solaris 10, we were assembling the main amd64 project team; work really started in earnest in January of 2004. Next time I'll describe some of our early problems with the porting work.

Technorati Tag: Solaris

Posted at 02:30PM May 10, 2005 by tpm in Solaris | Comments[1]

Hello, World

Picture of Tim Marsland Hello. First some preliminaries. Who am I? Well, I'm a Distinguished Engineer in the Operating Platforms Group at Sun; I'm also the Chief Technical Officer for that group. The first part of that means that I get to work on Solaris, the second part of that sentence means that it's a continuous struggle to do any engineering work in the face of too many meetings :) Among the other CTOs I've met both within Sun and outside of Sun, it's a relief to see that there's a wide interpretation of the role. Some CTOs are only concerned with abstract futures, some are solely buried in day-to-day issues. I try to be somewhere in the middle, and thus please nobody all of the time :(

I'm an engineer by calling; like my colleagues, we're in this to build software artifacts that make other peoples lives better. I've worked on Solaris for many years, on numerous subsystems and problems, from architectural direction to fixing bugs. We're very excited about Solaris 10 and how much it seems to be helping customers with their problems, and changing the way people think about Solaris, Sun, and Operating Systems technology in general. I think we're living in a time of transitions, and operating systems are relevant again, as the boundaries between software system components are shifting, hardware devices become ever more capable and complex, while new business models emerge.

During Solaris 10, my principal code contributions were around modernizing the Solaris kernel port to the x86 architectures, and on bringing 64-bit Solaris up on x64 platforms - our slightly boringly named "amd64" project. Some months ago a member of that team blogged about some of the work he'd been doing on that project, and hoped that someone would spend a bit more time talking about the other work that we did during the amd64 port. That seemed like a good topic for a blog entry, so that's what I thought I'd write about first.

Posted at 02:20PM May 10, 2005 by tpm in General | Comments[2]