If you're using our wireless drivers on Solaris today, then you probably know that we were only supporting WEP encryption in the past. There have been numerous requests for adding WPA support as well, and so I'm very happy to announce that we now have a ready-to-use solution available on OpenSolaris Laptop Community web page. Everything you need to know is documented on that page. Only Atheros (ath) driver supports WPA at this time. In order to start using WPA, you'll need to download and install ath-0.4 driver, wpa-0.1 daemon, and wificonfig-0.3 utility.

The X4100/X4200 coupled with Solaris 10U1 running Sybase IQ 12.6 has just achieved two new World Records on both 100GB and 300GB TPC-H benchmarks. This is a big step ahead considering that this is Sun's first TPC submission our x64 platform. Congratulations to everyone who was involved!

According to this blog, Asterisk PBX (a popular software PBX/VOIP solution that runs on many platforms) was found to be more scalable on Solaris 10 than on Linux. I've been playing with Asterisk a little bit and found it extremely powerful and flexible. We're trying to see if we can use it to host OpenSolaris conference calls over VOIP (for free!).

If you have a system with AMD Opteron processor(s), then try running the following program on it:

#include  <stdio.h>

inline void cpuid(unsigned int op, unsigned int *eax, unsigned int
*ebx, unsigned int *ecx, unsigned int *edx)
{
   __asm__("cpuid"
   : "=a" (*eax), "=b" (*ebx), "=c" (*ecx), "=d" (*edx)
   : "a" (op)
   : "cc");
}

int main(void)
{
   unsigned int eax,ebx,ecx,edx;
   unsigned int i=0;
   char array[17];
   char *cp=array;

   cpuid(0x8fffffff, &eax,&ebx,&ecx,&edx);

   for (i = 0; i < 4; i++)
       *cp++ = eax >> (8 * i);
   for (i = 0; i < 4; i++)
       *cp++ = ebx >> (8 * i);
   for (i = 0; i < 4; i++)
       *cp++ = ecx >> (8 * i);
   for (i = 0; i < 4; i++)
       *cp++ = edx >> (8 * i);
   *cp = 0;

   printf ("%s\n", array);
}

It can be easily compiled on Solaris with gcc:

$ /usr/sfw/bin/gcc -o opteron opteron.c

I'm very happy to announce the release of a new version of the CPU Caps project. This new version is based on Solaris Nevada build 40 and comes with a number of changes made since last release:

• CPU caps should now be more accurately enforced.
• FSS shares now work under caps (e.g., capped zone with projects in FSS class)
• Added DTrace probe for when threads are placed on wait queues (sched:::wait) for better observability
• Maximum CPU cap value was raised from 65535 to 1000000
• CPU caps now work property with hires_tick set to 1
• Added new -N option to prstat to show unnormalized CPU usage percentages (i.e, 100% = 1 CPU)

Hopefully, more people will download and try this version on their systems now. As always, please forward your comments and/or questions to cpu-caps-beta@sun.com. The feedback received so far had been very positive.

I'll be visiting my home country next week to give two presentations at the Solaris Day in Moscow on April 19th. This will be only the first day of the 3 days long Sun Tech Days event. If you're a developer who happens to live in Moscow and would like to find out more about Solaris, Java, NetBeans and other Sun products and technologies, then you can register for free for this event here. I'll be covering two new features in Solaris 10 - the Service Management Facility (SMF) and the Zones. I plan to post my slides here after the talk.

Hope to see you there!

UPDATE (April 20th): Copies of my presentations (in PDF format) are now available from here and here. It was great to see and chat with so many Solaris users in Moscow who participated in our Solaris Day sessions. I was very impressed with the solid level of Solaris knowledge that was obvious from the kinds of questions that were asked during the talks.

If you ever wanted to limit CPU usage of some applications so that other applications can run faster or even waste some CPU cycles running the idle loop instead, then keep reading. If you always try to achieve maximum possible performance for all of your applications, then this feature is probably not for you, and you should pay a visit to the Performance Community on OpenSolaris :-)

Beta preview of upcoming Solaris feature called CPU caps was released today under brand new OpenSolaris Resource Management project web page. CPU caps can be set independently on projects and zones using two new resource controls. The unit used for these resource controls is a percentage of a single CPU. Processes can still migrate between different CPUs, but their overall CPU usage will be limited. If you want more details, go to the project web page.

I'm really excited about finally getting it released externally and hoping to get your feedback on how it does (or maybe it does not for some reason) fit your specific needs on cpu-caps-beta@sun.com. It would be really interesting if somebody tried this on a SunRay server (but please, do NOT run this in production yet!) and tell me how it behaves there and whether it can be used to keep bad behaving applications (or users) under control.

If you ever wanted to limit CPU usage of some applications so that other applications can run faster or even waste some CPU cycles running the idle loop instead, then keep reading. If you always try to achieve the maximum possible performance for all of your applications, then this feature is probably not for you, and you should pay a visit to the OpenSolaris Performance Community.

Beta preview of upcoming Solaris feature called CPU caps was released today under brand new OpenSolaris Resource Management project web page. CPU caps can be set independently on projects and zones using two new resource controls. The unit used for these resource controls is a percentage of a single CPU. Processes can still migrate between different CPUs, but their overall CPU usage will be limited. If you want more details, go to the project web page.

I'm really excited about finally getting it released externally and hoping to get your feedback on how it does (or maybe it does not for some reason) fit your specific needs on cpu-caps-beta@sun.com. It would be really interesting if somebody tried this on a SunRay server (but please, do NOT run this in production yet!) and tell me how it behaves there and whether it can be used to keep bad behaving applications (or users :) under control.

Here is a tip that can be useful to people running JDS3 on Solaris 10... To add new TrueType fonts, copy all .ttf files to ".fonts" directory in your home directory and relogin. Any GTK application (e.g. Mozilla or Firefox) will be able to use them. No need to mess with mkfontdir(1) or anything else. Thanks to Alan Coopersmith for a tip.

PS: I think I've finally switched to JDS3 today after using fvwm2 forever. This is my third attempt so far to do it (old habits die hard) but I have a feeling this time it is going to work.

Hooray! We're announcing our new SunFire T2000 platform today. Being Sun's first server to incorporate the UltraSPARC T1 processor, it delivers remarkable performance in a rack-optimized 2U enclosure while drawing less than 400W of power. With prices starting at just $2995, it is probably the most affordable solution in its class on the market today. Here's a brief overview of SF T2K's main features.

• 32 simultaneous execution threads on a single chip
• 8x1.2Ghz cores (with 4 threads on each) connected through a 134Gb/sec crossbar switch to a high-bandwidth 12-way shared on-chip 3MB L2 cache (read Hugo's blog entry if you think this cache is too small)
• On-chip public key encryption support
• Up to 32GB of DDR2 memory
• 5 PCI slots (3 PCI-E and 2 PCI-X)
• 4 integrated Gigabit Ethernet interfaces
• Up to 4 73GB Small Form Factor SAS (Serial Attached SCSI) disks

SunFire T2000 is also Sun's first system with support for the Hypervisor. The Hypervisor is a small firmware layer that provides a stable virtualized machine architecture (sun4v) that runs below the operating system layer. The sun4v architecture is closely based on the legacy sun4u architecture of UltraSPARC-III systems, so as a result all sun4u applications just work on sun4v. The UltraSPARC T1 processor has new hyper-privileged mode of execution (in addition to user and privileged modes) so that the processor can distinguish between a legitimate access to various state registers by the hypervisor and illegitimate accesses by a rogue operating system. Memory is virtualized as well -- new intermediary "real" address address space is introduced so that the hypervisor can re-arrange physical memory resources without a guest OS needing to know exactly what happened. The I/O subsystem now also supports virtual devices. There's much more to talk about the Hypervisor, but that's a topic for a separate blog entry.

What I would like to talk about today is the scheduling optimizations that Eric Saxe and I have introduced for the SunFire T2000 platform. There are two main optimizations:

• Halting of idle CPUs -- when the scheduler idle loop can't find any work to do for a given hardware thread (of which there are 32), it makes a special call to the hypervisor layer to stop executing further instructions and waits for some interrupt to arrive. This is very similar to the way HALT instruction works on x86 and there are huge benefits for using it. When the idle loop is looking for work and tries to steal threads from other CPUs, it actually uses substantial amount of its core's execution pipeline bandwidth, which otherwise might have been given to various software thread doing some real work. In early development stages we saw a solid boost in performance when CPU halting was added.


• Core level load balancing -- because the processor has 8 cores, and all of them share a single L2 cache, one may think that it's not really important to choose on which core and which hardware thread within that core to run the threads. Turns out it's quite the opposite. Even though per-core L1 caches (16K I$ and 8K D$) and TLB entries (64-entry fully-associative I & D TLBs) are small and don't usually play any significant role here, the throughput that can be achieved by running two software threads simultaneously on two different cores vs. on two hardware threads of the same core can be very different. On UltraSPARC T1 hardware threads within the same core can switch on every cycle if other thread is ready. Threads also switch on all long latency instructions (loads, stores, fp ops, divisions, etc) and on various pipeline events (cache misses, traps, etc) while fair scheduling of threads within each core is done using LRU scheme. For more detailed description of VT (Vertical Threading) on UltraSPARC T1 go read Jim Laudon's blog. Given all this, threads that don't have to compete with other software threads running on the same core will get interrupted less often and as a result will get better throughput. The way the kernel finds out what CPUs belong to what cores is by analyzing the Machine Description DAG (directed acyclic graph) provided by the Hypervisor. This way, when we come up with even more complex CPU topology in the future, we'll just have to update that description and Solaris will take care of the rest.

As an example of what things looks like when simple while(1) loops run on otherwise idle SunFire T2000 system, here are two snapshots from David Powell's wonderful xlp utility (download sparc and/or x86 version) showing 8 and 16 threads getting evenly distributed across all 8 cores.

Monitoring CPU utilization with mpstat is so last century :-)

We're now working on additional scheduling optimizations which should yield even better performance on CMT hardware. This is a new and very exciting area for us to be in and we're spending quite a bit of time with our colleagues from SunLabs trying to figure out how to squeeze the last bit of performance on these systems. The tricky part of doing scheduling optimizations is that you don't want to "overdo" it as the benefits might get washed out by the extra complexity introduced.

[ Technorati: NiagaraCMT, Solaris ]

In case you haven't noticed yet, Sun Studio 11 just got released today. Sun Studio 11 software is available for Free now! This is going to have a significant impact on Solaris 10 adoption by the larger software developer community. And we're already seeing some good signs of that -- Oracle just selected the Solaris 10 Operating System as its preferred development and deployment platform.

Two more drivers were released today on the OpenSolaris Wireless Networking web page . The first one is called ipw, and supports Intel Pro/Wireless 2100 chipsets. The second is called iwi, and supports Intel Pro/Wireless 2200BG and 2915ABG chipsets. Both of these drivers are based on the source code obtained from the Native *BSD Driver project. So if your laptop has Intel Centrino wireless network adapter, you can now go wireless with Solaris on it. Have fun!

---

Technorati Tags: OpenSolaris, Solaris, WiFi

I'm very excited to announce that today we've finally made one of our wireless drivers available for download from the OpenSolaris Wireless Networking web page. This is just one of the few wireless drivers that will be pre-released to the OpenSolaris Laptop community in the future in a similar way and I think it is indicative of how we're changing our development processes by making internal projects more visible outside of Sun. We hope that you'll find new ath driver and new wificonfig tool easy to use and will provide us with valuable feedback that will help us to get these components eventually integrated into Solaris with the highest quality.

Along with this release, we've also created new OpenSolaris Laptop Community. I encourage people interested in running Solaris on their laptops to subscribe to laptop-discuss@opensolaris.org and start using it.

Note that this is still work in development, and things may change before this gets putback into Solaris. Also, you may encounter some issues with the driver, but so far it has been extremely stable and a lot of work has been put into testing it in various configurations.

Have fun!

PS: If you're looking for a replacement mini-PCI card for your laptop I recommend getting Wistron's CM9 card based on the Atheros 5213 chipset. It works flawlessly on my Ferrari 3400 and I know a few other engineers in the Solaris Kernel Group who are successfully using these or similar cards.

---

Technorati Tags: OpenSolaris, Solaris, WiFi

If you PC is still not quiet enough after installing fanless heatsink (like this one), I can recommend you to take a good look at Antec's Phantom 350 fanless power supply. This thing is absolutely silent and can provide up to 350 watts of power. The only source of any substantial noise left in my PC now is the hard drives. Turning on the acoustic mode helps a lot there, but the disks are still spinning :-)

Today is The Day. We are releasing the source code for the OpenSolaris. Welcome! It's incredible that anyone in the world can now look at what we've been working on for many years, and more importantly, choose to participate in it. Back when I was a CS student taking an OS Internals class and studying Solaris by reading the wonderful "Operating Systems Concepts" (aka "the dinasaur book") book by Abraham Silberschatz and "Unix Internals: The New Frontiers" by Uresh Valhalia, I can't imagine what a difference it would make to be able just open a browser and see the actual source code. This would've been so incredibly useful and educational. Unfortunately, our teacher at that time could only get us a copy of the "Lion's Commentary on Unix", the legendary underground classic book which covers the complete source code of an early version of Unix, and we had to study the ancient PDP-11 architecture to understand what each line of code is actually trying to do.

Now, fast forward a few years ahead, and you'll see how different the landscape is today. Being able to look at the source for the Operating System studied in the class, and even modify it in any way you want, is going to make a huge difference. Yes, I know, this has been possible with Linux or FreeBSD for a number of years, but with OpenSolaris today, we've given even more choice to developers. With all these changes happening to Solaris, many things, however, managed to stay the same, like the name of the disp() routine in disp.c¹. Function and variable names no longer have the compiler restriction to be no more than 5 characters long as they did more than 20 years ago on PDP11, but nevertheless, creat(2) system call is still called creat(), and it will never be renamed to be create(2). But I digress...

So, with this preamble, I'd like to give you a quick tour of one small subsystem of the Solaris kernel that I've been working on during Solaris 10. The subsystem, one particular aspect of which I'm going to describe here today, is called "Resource Pools". With Resource Pools, various types of resources (such as CPUs, memory, etc) can be combined together and given unique names. Solaris had processor sets for a long time, but they were not persistent across a reboot, and didn't have easy to use names. So, we've changed the administrative model (while keeping all processor set interfaces intact for compatibility reasons) by making the pools configuration persistent, and allowing different types of resources, not just CPUs, to be used in the future. One of the projects that is under development right now is Memory Sets. This project will allow chunks of memory to be put in pools the same way CPUs are put today. So, while implementing the kernel part of the Resource Pools framework, we had to keep this in mind.

There has been one particular aspect we had to deal with in Resource Pools that took more brain cells to figure out than the others. That was the problem of binding of a collection of processes to a pool. Sounds simple, right? That's what I thought in the beginning too. I'll just grab pidlock (one global lock that protects the list of PIDs) so that new processes don't get created while I'm going around modifying the existing ones, and that would be it. In reality, this had to be done in a totally different way, partially because we couldn't actually have pidlock held while changing resource pool bindings for processes. The binding operation had the following requirements all of which had to be met:

• The operation had to be atomic. That is, either all processes affected by the operation had to be moved to a new pool, or none.
• If the operation is going to fail, we have to know about that in advance before actually attempting to rebind. We can not sustain a failure while rebinding because then we may have to undo everything done so far and we may get another failure again, which would violate the first requirement.
• The target set of processes could be a single process, a task, a project, or a whole zone. For more on tasks, and projects, please read this article on docs.sun.com or read the source.
• There should be zero performance impact on system calls like fork(2), and exec(2) if no binding operation is in progress.
• When support for memory sets is added, we should be able to change processor set and memory set bindings for processes using the same mechanism.
• The operation had to be interruptible, as it can take a long time to change memory set bingings for processes with large address spaces

It took us some time to figure out how we're going to satisfy all these requirements. Obviously, grabbing pidlock and cpu_lock (to prevent processor set configuration changes) wasn't an option anymore. What we came up with was much more delicate.

First, we've came up with the idea of a pool barrier for processes. That is, when a process is in a critical section of the code which may overlap with what pool rebinding operation is supposed to do, we're putting that process inside a special barrier thus preventing this process from being rebound while it is there. There are a few places in the kernel where we do that. There are two functions, pool_barrier_enter(), and pool_barrier_exit(), which serve this purpose. These have to be called while we're holding process' p_lock, and in the common non-contended case these functions are just incrementing and decrementing reference counts.

As you can see in pool_barrier_enter(), if the process is being rebound when we're entering the barrier, we're going to hang out there and wait until the rebinding operation finishes.

   
        void
        pool_barrier_enter(void)
        {       
                proc_t *p = curproc;
                 
                ASSERT(MUTEX_HELD(&p->p_lock));
                while (p->p_poolflag & PBWAIT)
                        cv_wait(&p->p_poolcv, &p->p_lock);
                p->p_poolcnt++;
        }

In pool_barrier_exit(), we have to do a more complicated dance since it is possible that a rebinding operation began while we were inside the barrier. In this case, the thread doing rebinding will be waiting for us to acknowledge that we're done and then we will tell that thread to go change our binding and we're going to wait.

Then we needed something to protect the pools configuration itself from changing while we're doing rebinding. A mutex or a read-write lock wouldn't work, since we were dealing with possibly really long hold times (with large memory sets), and the action of acquiring this lock had to be interruptible. What we came up with was the pool_lock_intr() routine which used a combination of one condition variable, and one mutex to give us the desired behavior.

Now that we have this infrastructure in place, we can look at the rebinding operation itself. The meat of the rebinding operation can be found in the pool_do_bind() routine. There's a good sized block comment above it which describes the steps we're going to take, and covers them in some amount of details. The first major step is to find all processes of interest and set PBWAIT flag in the p_poolflag for each one of them. Once we set these flags, we're guaranteed that these processes will not disappear, even after we drop the pidlock. We save all proc_t structure pointers in a special array so that we can safely reference these processes without pidlock being held.

The second step is to wait for relevant processes to stop before they try to enter the barrier or at the exit from the barrier. One important detail here -- we can't allow a debugger, or /proc to stop us while we're here while we're holding pool_lock. So we use the special lwp_nostop flag for that which will leave us stopped in post_syscall() on our way back to userland with no locks held, and then politely ask the library to retry the whole operation again.

The third step is pick up child processes that were created by fork, but didn't exist during our first scan of the process list. Their parents will now be stopped at pool_barrier_exit in cfork(). We can also now get rid of the processes that were in the middle of the pool barrier in exit(2) when we did our first scan. One interesting detail here is that since we're holding parent processes hostage, their children cannot start running again and create more children. The place for a call to pool_barrier_exit() in cfork() was carefully selected to prevent that.

        /*
         * pool_barrier_exit() can now be called because the child process has:
         * - all identifying features cloned or set (p_pid, p_task, p_pool)
         * - all resource sets associated (p_tlist->*->t_cpupart, p_as->a_mset)
         * - any other fields set which are used in resource set binding.
         */
        mutex_enter(&p->p_lock);
        pool_barrier_exit();
        mutex_exit(&p->p_lock);

A few lines later, the newly created child process is made runnable and it would be too late. And now... (drumroll, please)... we've reached the end of our journey and we finally ready to perform the bind operation for our processes. This may include changing its processor set binding (which, by the way, we know is not going to fail), and changing its scheduling class. When we're done with all that, we insert the feeding tube into the process again and let him go whatever it wants again.

        /*
         * Okay, we've tortured this guy enough.
         * Let this poor process go now.
         */
        pool_bind_wake(p);

That's pretty much it. I thought this was a good example of what types of problems we have to deal with in Solaris sometimes. It seems like a very simple task on the surface, but as more requirements are being put on it, it forces us to come up with new and sometimes unusual ways of solving them.

In my future blogs, I will try to cover some interesting parts of the Solaris dispatcher and its scheduling classes. There's a lot of topics there that deserve individual posts. What I like about working on scheduling problems is that they are never boring. Having DTrace in our arsenal of tools has definitely made it easier to analyze various scheduling problems and also help us verify the fixes for them. I'll also need to tell you (or even warn you) about the curse of disp.c.

---

¹ While you're in swtch(), check out this interesting assertion, which, I have to admit, really confused me the very first time I looked at this code:

 
#ifdef  DEBUG
        if (t->t_state == TS_ONPROC &&
            t->t_disp_queue->disp_cpu == CPU &&
            t->t_preempt == 0) {
                thread_lock(t);
                ASSERT(t->t_state != TS_ONPROC ||
                    t->t_disp_queue->disp_cpu != CPU ||
                    t->t_preempt != 0); /* cannot migrate */
                thread_unlock_nopreempt(t);
         }
#endif  /* DEBUG */

So, if all three statements are true, we're grabbing thread's lock and testing that at least on them is false. Any ideas on what this code is trying to do? :-)

---

Technorati Tag: OpenSolaris
Technorati Tag: Solaris