Jim Connors' Weblog

Using Solaris to Burn the OpenSolaris 2008.05 CD Image

If you're using Solaris or OpenSolaris as a platform to download and burn the recently announced OpenSolaris 2008.05 image, the instructions found here may need a slight modification.  After downloading the Live CD, you are instructed to use the cdrw(1) command as follows to burn the image to CD:

cdrw -i /path/to/iso/image/os200805.iso

Here's what might happen on your system when you attempt to use the 'cdrw -i' command:

The cdrw(1) command fails because it thinks the ISO image is larger than the default capacity of the CD.  The solution to this problem is explained in these excerpts from the cdrw(1) man page:

"Most commercially available drives allow writing beyond 74 minutes as long as the media has the capacity (such as 80–minute media). However, the ability to write beyond 74 minutes might not be supported by the drive in use. If the drive supports this feature, then use the -C option to indicate that the tool should rely on the capacity indicated by the media.

"-C [option] Uses stated media capacity. Without this option, cdrw uses a default value for writable CD media, which is 74 minutes for an audio CD, 681984000 bytes for a data CD, or 4.7 Gbytes for a DVD." 

This time let's invoke 'cdrw' with the -C option.

Voila.

Posted at 12:20AM May 07, 2008 by jtc in Sun  |  Comments[0]

Modifying and Respinning a Bootable Solaris ISO Image

As an adjunct to the previous blog post, a slightly customized boot environment capable of enabling serial console kernel debugging was required to diagnose Solaris install problems.  The post itself mentioned that a nice way to accomplish this was to set up PXE network boot via Solaris jumpstart.  It is indeed flexible and enables one to experiment with modifications and quickly test whether they perform as expected.  The one downside to this environment is that an additional DHCP/TFTP boot server has to be configured.  To eliminate that service, you could, once the customizations are well understood, simply create a new version of the Solaris install DVD with your customizations.  Let's run through the process for a simple example.

1. Get a Solaris DVD.  For this example, we'll use an interim build of the upcoming Solaris 10 Update 5.

2. Extract the the entire contents of the DVD.

# lofiadm -a /export/iso/s10x_u5b10_dvd.iso
/dev/lofi/1
# mkdir -p /iso
# mount -r -F hsfs /dev/lofi/1 /iso
# cd /iso
# mkdir /export/modified-s10x_u5b10
# find . -depth -print | cpio -pudm /export/modified-s10x_u5b10
4516208 blocks

3. Modify the content contained in /export/modified-s10x_u5b10.  In this case, we'll change the boot/grub/menu.lst file found in this directory to look like:

#
#pragma ident   "@(#)install_menu       1.1     05/09/01 SMI"
#
default=0
timeout=30
serial --unit=0 --speed=9600
terminal serial
title Solaris Serial Console ttya
        kernel /boot/multiboot kernel/unix -B install_media=cdrom,console=ttya
        module /boot/x86.miniroot 

4. Issue the following magic incantation to create a bootable ISO image based on the contents of the /export/modified-s10x_u5b10 directory.

# mkisofs -R -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -R -L -r -D -U \
-joliet-long -max-iso9660-filenames -boot-info-table -o \
/export/iso/modified-s10x_u5b10.iso /export/modified-s10x_u5b10
...
Size of boot image is 4 sectors -> No emulation
  1.80% done, estimate finish Fri Apr 18 15:55:13 2008
  2.25% done, estimate finish Fri Apr 18 15:59:07 2008
  2.70% done, estimate finish Fri Apr 18 15:58:37 2008
  3.15% done, estimate finish Fri Apr 18 15:58:48 2008
...
 98.43% done, estimate finish Fri Apr 18 15:59:37 2008
 98.88% done, estimate finish Fri Apr 18 15:59:37 2008
 99.33% done, estimate finish Fri Apr 18 15:59:36 2008
 99.78% done, estimate finish Fri Apr 18 15:59:36 2008
Total translation table size: 2048
Total rockridge attributes bytes: 3075577
Total directory bytes: 18921472
Path table size(bytes): 148014
Max brk space used 1b24000
1112471 extents written (2172 MB) 

Voila!  Acknowledgements to Tom Haynes and this blog post which served as an excellent guide.

Posted at 09:28AM Apr 21, 2008 by jtc in Sun  |  Comments[1]

Enabling Remote Console Debugging of Solaris x86 Boot/Install

Our partners do a fair amount of business supplying ruggedized Solaris-powered Panasonic Toughbook computers to their US government/military customers.  As a regular part of the product cycle, Sun usually works with both the integrators and Panasonic to assure that as new models become available, Solaris runs on these systems properly.  Furthermore, when we can get our grubby little hands on the systems, we'll run them through our certification suite of tests and formally place them on the Solaris Hardware Compatibility List.  As an example, here's the certification report for one of the versions of the Panasonic Toughbook CF-29.

Panasonic recently introduced a new version of the Toughbook CF-30 (referred to as revision F) which tweaks some of the computer subsystems resulting in an all-too-familiar scenario: namely, these differences cause the current version of Solaris to fail to install.  Note: Solaris is not alone here, all Operating Systems must continually play this cat and mouse game to support the latest hardware/firmware advances.

Our initial hypothesis lead us to believe that the problem was related to the introduction of the Intel ICH8 SATA chipset.  So we called on some of our Solaris I/O experts, based out of Sun's Beijing office, to take a peek at what was going on.  As the laptop is currently in New York, we needed a way for folks half way around the world to have access to this system.  There are lots of mechanisms available to remotely diagnose systems, what's somewhat unique here is the following: (1) the diagnosis takes place very early in the boot stage, way before any windowing or networking is set up and (2) The system in question is a laptop, not a server, where things like Lights Out Management (LOM) service processors are non-existent.

The solution here was to utilize decades old RS-232 technology combined with some of features of the GRUB bootloader.  Here are two requirements needed:

• A serial connection must be established between the system to be diagnosed, which will be referred to henceforth as the target, and the system which accesses the target which we'll refer to as the remote host.   Unfortunately most laptop manufacturers have seen fit to eliminate serial port connectors in lieu of using USB to serial converters as a replacement technology.  At the early stages of boot, USB/serial capability is not yet available, so these systems are not good candidates for this type of diagnosis.  Thankfully the target in question here, the Panasonic CF-30 Toughbook, still comes with a serial port.

• A Jumpstart environment capable of installing Solaris on x86/x64 systems is strongly recommended.  As part of the process described below, we'll be modifying the target's GRUB environment  (menu.lst).  If you chose to use a DVD boot/install environment instead, you'd need to modify and burn a new DVD for each change made to the target's boot environment.  It took a bit of time to find the right incantations in the menu.lst file to get what was needed here; continually re-burning DVDs would have been excruciating.  This exercise is left to the reader, here's a good start to understanding the jumpstart setup process.

Here's how to set up the remote console environment:

1. A null modem cable must be physically connected between the remote host and target.  The most common cable required will be a DB-9 female-to-female connector.  Your configuration may vary.

2. Check the BIOS of the remote host and target and make sure serial ports are enabled.

3. Running Solaris on the remote host, we'll be using the tip(1) command to access the target via serial port.  Edit the /etc/remote file and look for the hardwire: entry.  Modify it to look like this:

hardwire:\
        :dv=/dev/term/a:br#9600:el=^C^S^Q^U^D:ie=%$:oe=^D:

4. As part of setting up a jumpstart install for the target, a series of files are created in the /tftpboot directory of the jumpstart server.  Under /tftpboot, there should be a custom menu.lst file for each managed install, suffixed by the unique network MAC address of the system in question.  For example, the network MAC address for the CF-30 in question is 0:b:97:db:c0:97.  The related /tftpboot file for the CF-30 turns out to be /tftpboot/menu.lst.01000B97DBC097. As your target will have a different MAC address, it's menu.lst  file will have a different suffix in the /tftpboot directory.  Edit that custom menu.lst file (for example, /tftpboot/menu.lst.01000B97DBC097) to look as follows:

default=0
timeout=30
serial --unit=0 --speed=9600
terminal serial
title Solaris_10 s10x_u5b10
        kernel /I86PC.Solaris_10-1/multiboot kernel/unix -B console=ttya,install_media=192.168.1.5:/export/s10x_u5b10 -vkd
        module /I86PC.Solaris_10-1/x86.miniroot

The key modifications here involve (1) inclusion of the serial --unit=0 --speed=9600 and terminal serial lines plus (2) additional arguments added to the kernel directive.  Grub is very fussy about the order and placement of arguments; playing around with these will likely change grub's behavior. 

5.  From the remote host, access the serial console of the target by issuing:

$ tip hardwire

6. Inside a terminal window, here's what the serial console looks like, after the system has been power cycled and runs through the POST sequence:

After the miniroot is loaded, you'll be presented with an mdb prompt and a screen which looks like this:

You can now issue mdb commands to diagnose.  In this scenario you should also be able to reboot the  system without any other manual intervention, like this:

 

Here's what issuing the mdb commands ':c' and '$c' look like in this environment.   From this simple  trace we can ascertain that the SATA drivers were never even loaded.  Turns out this is likely a VM problem.  Here's the filed bug report.

Posted at 04:28PM Apr 14, 2008 by jtc in Sun  |  Comments[2]

Reduced Footprint Java SE: Bringing Java Standard Edition Down to Size

A previous blog post demonstrated how you can, with minimal effort, lessen the disk footprint of a typical Java SE 5.0 runtime environment by about a third without violating the Java Standard Edition Licensing agreement. That post focused primarily on removing optional files and compressing class library jar files.  It turns out that with a little more engineering,  there is significant opportunity for further space optimization.

These additional savings involve more intimate knowledge of the inner workings of Java SE.  Sun performs this engineering work and provides a series of reduced footprint versions of Java SE, in binary form, for some of the most common embedded platforms.  They include some of these enhancements:¹

• Headless configuration - The inclusion of graphics subsystems like AWT and Swing comprise a large chunk of the Java SE footprint.  If your device has no graphics capability (i.e. is headless) why would you need to include this functionality?  Headless configurations:

1. Do not support keyboard or mouse input
2. Cannot create windows or display graphics
3. Throw a HeadlessException when a graphics API is called
4. Still support a functional Java 2D API for printing and off-screen rendering
5. Are still 100% Java SE compatible
   

• Eliminate client or server compiler - Sun's Java SE implementations include two HotSpot compilers, tuned and designed for specific environments.  The client compiler focuses on things like fast user interactivity and quick startup, while the server compiler's priority is on optimizing large, long-lived server applications.  The Java SE VM can only use one of these compilers at a time, so removing the unused one can save considerable space.

• Minimizes memory consumption - Standard Java SE allocates RAM to house things like the JIT code cache and the object heap.  By default, each one of these areas (and others) can be assigned 64 MB.  In an embedded platform where the total amount of RAM might only be 64MB, Java SE will simply not have enough memory to run.  Java SE Embedded on the other hand, will automatically adapt memory usage to the amount of available RAM.  Consequently, 64MB is a reasonable amount of RAM for a large set of embedded Java applications.

• Space vs. speed tradeoffs - (1) Java SE implements a thread lookup table, which in layman's terms, helps save a few instructions when switching between Java threads.  By eliminating this table, a couple of MBs of RAM can be spared from your application's working set. (2) Java SE also sets aside an area (mmap'ed) for loading jar files into random access memory, which as was explained to me, may help performance, but may also result in duplicate copies of jar files in memory.  Removal of this area further reduces the Resident Set Size.

An Example

Following through with the exmaple in the last post., let's start with an unmodified version of Java SE 5.0 Update 13 for Linux/x86.  By default, the static footprint is approximately 88MB.

   jimc@jalopy:/tmp> du -sk ./jre1.5.0_13/
   88185   ./jre1.5.0_13/

After following the directions in the previous post, we can pare it down to roughly 60MB.

   jimc@jalopy:/tmp> du -sk /tmp/jre1.5.0_13/
   59358   /tmp/jre1.5.0_13/ 

Downloading Sun's reduced footprint version of Java SE for x86/Linux yields:

   jimc@jalopy:/tmp> du -sk /tmp/jre1.5.0_10/
   31003   /tmp/jre1.5.0_10/

This version of the JRE is about one-third it's original size, and furthermore has been modified to use significantly less memory than the standard Java SE offerings.  Note: we are comparing slightly different updates of Java SE 1.5 (update 13 vs. update 10).  They are indeed not identical but their differences should be considered, for the sake of argument, negligible. 

[1] Many thanks to Bob Vandette, who though presentation and conversation, supplied this information.  One of many sources comes from Bob's Java ONE 2007 session called, Deploying Java Platform Standard Edition (Java SE Platform) in Today's Embedded Devices (TS-2602)

Posted at 02:50PM Mar 26, 2008 by jtc in Sun  |  Comments[1]

Reducing Your Java SE Runtime Environment Footprint (Legally)

Because the resource requirements for Java Standard Edition are by no means insignificant, developers interested in utilizing Java on smaller platforms have traditionally opted for one of the Java Micro Edition configurations.   It should therefore come as no surprise that some of the Standard Edition functionality has to be sacrificed in these constrained Java ME environments.  However, as the trend towards more capable devices continues, it becomes reasonable to once again consider the advantages of utilizing Java SE.  Unfortunately, with a static footprint that could easily exceed 100MB, Java SE may still be too large to swallow for a lot of applications.

It is duly noted that the criticism leveled at Sun for exacting too much control over the Java platform has been widespread.  Like it or not though, one benefit of Sun's initial stewardship has been that Java SE has remained a standard platform, and threats to splinter it have thus far been reasonably thwarted.  Accordingly, in order to avoid incompatibilities, there are restrictions spelled out in the Java SE Licensing Agreement which prohibit modification of the Java SE binary distribution.

That  being said, there are a list of optional files, specified by the Java Runtime Environment's README file which can be removed, without ramification, to lessen the footprint.  They include:

• Deployment tools (e.g. Java Web Start, Java plugin)
• IDL and RMI tools (e.g. rmid, tnameserv)
  
• Security tools (e.g. policytool, keytool)
• orbd
• Localization (charsets.jar)

In addition, further space optimization can be achieved by compressing the class library files contained in the rt.jar file.  By default, Java SE ships this jar file uncompressed.  The tradeoff here is space vs. performance, i.e. the classloader must expend cycles to uncompress the Java classes as they are being loaded.

An Example

So let's download a sample JRE from java.sun.com and see how it's footprint can be minimized.   For this example, we'll use Java SE 1.5.0 Update 13 for Linux x86.

After installation, the JRE is approximately 88MB

   jimc@jalopy:/tmp> du -sk ./jre1.5.0_13/
   88185   ./jre1.5.0_13/

Here's what it looks like after removal of the optional files

   jimc@jalopy:/tmp> cd jre1.5.0_13/

   jimc@jalopy:/tmp/jre1.5.0_13> /bin/rm -rf lib/charsets.jar lib/ext/sunjce_provider.jar \
   lib/ext/localedata.jar lib/ext/ldapsec.jar lib/ext/dnsns.jar bin/rmid \
   bin/rmiregistry bin/tnameserv bin/keytool bin/kinit bin/klist bin/ktab \
   bin/policytool bin/orbd bin/servertool bin/javaws, lib/javaws/ and lib/javaws.jar

   jimc@jalopy:/tmp/jre1.5.0_13> cd ..
   jimc@jalopy:/tmp> du -sk ./jre1.5.0_13/
   77227   ./jre1.5.0_13/

And after rt.jar has been compressed

   jimc@jalopy:/tmp> mkdir rtjar
   jimc@jalopy:/tmp> cd rtjar/

   jimc@jalopy:/tmp/rtjar> jar -xf /tmp/jre1.5.0_13/lib/rt.jar
   jimc@jalopy:/tmp/rtjar> zip -q -r /tmp/rt .
   jimc@jalopy:/tmp/rtjar> mv /tmp/rt.zip /tmp/jre1.5.0_13/lib/rt.jar

   jimc@jalopy:/tmp/rtjar> du -sk /tmp/jre1.5.0_13/
   59358   /tmp/jre1.5.0_13/  

Conclusion

In many cases, you can lop off about a third of the Java Runtime Environment footprint with no ill effects.  In a future post, we'll discuss how Sun has further reduced Java SE significantly, not only from the point of view of static footprint, but also from a RAM usage perspective too.  For a preview you can check out Sun's Java SE Embedded technology.

Posted at 11:29AM Feb 14, 2008 by jtc in Sun  |  Comments[3]

Good Things Come To Those Who Wait

Java RTS 2.1 EA (Early Access) marks the arrival of a commitment made some time back, namely that Sun would provide a version of the Java Real-Time System for Linux.  Perhaps to some, it was long time in the making, but in fact there are at least 2 good reasons why a Linux version wasn't available till now:

1. Until recently, there was no standard Linux release/kernel which had true real-time support.  Typically the versions available were non-standard and did not constitute any considerable volume.  Mainstream Linux distributions are only now incorporating the necessary real-time underpinnings.
2. Porting the Java Real-Time System VM to additional platforms is non-trivial.

Support and testing for Java RTS 2.1 EA at this time is limited to the currently shipping SUSE Linux Enterprise Real Time10 platform and the upcoming Red Hat Enterprise MRG 1.0 release.  It is however possible that other versions of Linux could run Java RTS 2.1 EA as it utilizes the real-time POSIX programming interface.  At minimum they would require a 2.6.21 kernel or greater and a glibc of 2.5 or greater.  In addition, the latest RT patch would also be needed.

This announcement pertains only to Linux, but of course a 2.1 EA version for both Solaris x86/x64 and Sparc will be shortly forthcoming. In the interim, a version of Java RTS 2.0 update 1 is readily available.  Documentation for both Java RTS 2.0 and 2.1 EA can be found here.

Regardless of platform, an evaluation version of the software is available for download at the Java RTS download page.

 

Posted at 09:52AM Jan 31, 2008 by jtc in Sun  |  Comments[0]

Java RTS 2.0 Update 1 Released

After blogging about the relationship between high resolution timers and Java RTS, a comment was posted by a reader stating that they couldn't use Java RTS 2.0 with the latest Solaris 10 update.  Unfortunately that reader was correct.

At the time of the post, Sun had released Solaris 10 Update 4, while Java RTS 2.0 would only support Solaris 10 update 3.  To further exacerbate the situation, once Sun releases a new version of Solaris, they do their best to encourage users to use the latest version by making it difficult (if not impossible) for them to locate previous update versions.

There is a dependency between the aforementioned high resolution timer interface and Solaris which is update specific.  We're working to rectify this situation, but as the process behind Solaris is deliberately bureaucratic, this might take a while.  In the interim we'll need to keep in better lock step with Solaris releases.

On November 27, Java RTS 2.0 update 1 was released: 

• It supports the latest Solaris 10 update (called 08/07 or update 4)
• Support for Java SE has been updated to Java SE 5 update 13
  
• A new Java RTS DTrace Provider is available
• Other enhancements are listed here
  

 

Posted at 03:37PM Dec 19, 2007 by jtc in Sun  |  Comments[0]

General Purpose Always Wins ... Why Not for the Real-Time Market Too?

The brief but brilliant era of computing has seen companies come and go, many relegated to the ash heap of history because they failed to heed this simple rule:  
 
  In the long run general purpose solutions will always win out over specialized proprietary ones. 
 
As a long time employee of Sun Microsystems, I've witnessed firsthand the effects, both positive and negative, that this law has had on my company.  Sun can attribute it's initial meteoric rise to the fact that it offered a viable alternative to the popular minicomputer platform of the day.  The first Sun workstations were built from commercial off-the-shelf components, and although nearly equal in performance to the minicomputer, they were so much more affordable that they became good enough.  And of course over time, as Moore's law might dictate, those initial deficiencies quickly dissipated, and in fact surpassed traditional minicomputer capabilities.
 
Somewhere along the line Sun lost sight of this ideal and began incorporating more proprietary technology into their products.  At first the strategy appeared to be successful as Sun was well positioned to reap huge benefits during the Internet bubble.  Meanwhile, low-cost general purpose servers were continuously improving.  When the bubble burst they in turn became good enough alternatives to the powerful but costly Sun servers.  The resulting decline of Sun was rapid, and it's taken the better part of a decade for us to recover.   This story has been told -- and will continue to be again and again -- for those refusing to learn this lesson.  A professor of mine once told me, "If there's anything we've learned from history, it's that we haven't learned anything from history".

When markets mature, even those where technology is paramount, economic considerations dominate. General purpose systems scale on every dimension (unit, management, training and integration costs) whereas proprietary systems do not.  A switch to more standard components should in no way be construed as stifling innovation.  Rather, general purpose systems help create new innovation by building from common elements economically appealing in their own right, and presumably even more economically beneficial once combined.¹

[1] The above paragraph was taken in bits and pieces from an email exchange with Dave Hofert, Java Embedded and Real-Time Marketing Manager.  His points were so compelling I couldn't help myself.

Real-time industrial controllers could be the next market ripe for disruption.  Admittedly this is an entrenched and conservative lot.  But the economics of general purpose computing cannot be denied.  As organizations strive to further eliminate cost out of their system, revisiting usage and deployment of industrial controllers, typically via custom proprietary PLCs, is falling under review.  Consequently, at the behest of one of the world's largest industrial corporations, Sun has partnered with iGoLogic, a systems integrator, and Axiomtek, an industrial PC board manufacturer, to create the real-time controller platform pictured below.
 
Among others, here are some of the compelling benefits of this platform: 

• It's based on standard x86 hardware.  The motherboards aren't much larger than a postcard, are energy efficient, and yet have PC class processing power.  The number of competing manufacturers in this space eliminates vendor lock-in and insures price sensitivity and further rapid innovation.

• It runs on standard Solaris.  No custom proprietary real-time operating system is required.  One of the best kept secrets of Solaris is its real-time capabilities.

• Real-time applications are developed using Sun's Java Real-Time System, enabling you to leverage the largest development community on the planet.  Obscure development languages and highly specialized environments are longer needed.
  

• Industrial Networking Protocols (e.g PROFIBUS, EtherCAT) are easily migrated to this platform, partly because of the wealth of development tools available.

• The system utilizes an IDE flash drive from Apacer.  In addition to eliminating the moving parts associated with a traditional disk drive, it consumes less power and makes the system more resistant to shock and vibration.  Overcoming the longevity limitations of flash memory, Apacer has done some interesting work on wear leveling algorithms effectively extending the lifetime of the flash device well past the expected lifetime of the industrial controller.
  

 
Let this be our first salvo fired over the proprietary industrial encampment.  We believe the opportunity is immense, but also understand that to achieve any measure of success, partnering with organizations who are truly experts in this arena is critical.  If you think you can add further value, we'd love to talk.

Posted at 09:19AM Oct 15, 2007 by jtc in Sun  |  Comments[2]

Real-Time Java and High Resolution Timers

Any modern x86/x64 processor worth its salt comes equipped with an Advanced Programmable Interrupt Controller, or APIC.  Among the features that an APIC provides is access to high resolution timers.  Without such capability, the default interrupt source for timer and cyclic operations has a precision on the order of 10 milliseconds -- hardly fine-grained enough for any serious real-time work.

The cyclic subsystem, introduced in Solaris 8, gives Solaris the capability to support high resolution timers.  The Sun Java Real-Time System version 2.0, available for Solaris on both x86 and Sparc platforms, includes an additional package called SUNWrtjc:  Java Real-Time System cyclic driver.  This package exposes an interface to the cyclic subsystem, normally only available to the kernel.  For those already familiar with Sun's Java RTS, you've no doubt noticed that you either need to run as superuser or assign a set of fine-grained privileges to an ordinary user account. (sys_res_config, proc_priocntl, proc_lock_memory and proc_clock_highres).  The proc_clock_highres privilege gives access to those timers.

Originally developed on an AMD Athlon-based PC, I recently moved a Real-Time Java project over to my Toshiba Tecra A1 laptop running the same version of Solaris and Java RTS.  With the goal of getting in a little development time during a long flight, that migration suddenly casued timings to be all wrong.  Why, might you ask, would moving an application from one seemingly identical Solaris environment to another cause this unusual behavior?  Turns out that even though the laptop, a Pentium 4M based system, has an APIC, it was never exposed by the laptop BIOS.  In this scenario, regardless what you do from a Solaris perspective, you'll never get access to the high-res timers. 

This phenomenon appears to be most prevelant in laptops.  As they account for about 50% (or more?) of PCs sold today, developers have a realistic chance of running into this situation.  You can issue this magic Solaris incantation to determine if your system has high-res timer support:

   # echo "psm_timer_reprogram::print"  | mdb -k

If anything but 'apic_timer_reprogram' is then displayed, your machine has no exposed APIC, and is probably unsuitable for Java RTS.  In some cases the BIOS may be configurable with regards to APIC support; in many others it is simply not available.

In the absence of an APIC, there is the potential to improve the high-resolution timing by setting the following tunable in /etc/system:

   set hires_tick=1

Following a reboot, this would change the clock tick frequency from 100 to 1000 ticks per second. This frequency can then be further tuned by setting the hires_hz tunable to the desired value:

   set hires_hz=10000

The default value is 1000 ticks per second; higher values are not officially supported.

Note that tuning your machine in this way does not come without cost.  It is likely to degrade overall performance, as the system will need to spend a larger proportion of time handling the larger frequency of clock interrupts.¹

   [1] Thank you  Christophe Lizzi for your explanation of the problem and potential workaround.
 

 

Posted at 09:57AM Jun 19, 2007 by jtc in Sun  |  Comments[2]

Audio Transcript and Slides Available for "Real-Time Java Meets Wall Street"

Better late than never ...

As Promised, you can find the audio transcript for the Java ONE 2007 technical session TS-1205: The Sun  Java Real-Time System Meets Wall Street at the aforementioned link.  The slide deck should also be available there, or can be downloaded here.
 

Posted at 01:44PM Jun 11, 2007 by jtc in Sun  |  Comments[0]

Real-time Java Meets Wall Street

Jim Clarke, Eric Bruno and I have been selected to present this year at Java ONE.  The title of our session is called "TS-1205 The Sun Java Real-Time System Meets Wall Street".  We're scheduled to present on Wednsday, May 9 at 10:55AM - 11:55AM Pacific in the Moscone Convention Center in San Francisco.  If you plan on attending, please stop by and see us.  For those that can't make the event, we'll be sure to post the slides once we're permitted.

To get an idea of what we'll discuss, take a look at the abstract below; 

 ABSTRACT

Today's financial institutions use program trading systems to automatically execute split-second trades based on sophisticated algorithms that assess current market conditions. Trade decisions and execution must occur in a timely fashion to capitalize on the market. Missing a trade opportunity, even by a few seconds, can lead to significant losses.

With improvements in Java technology performance, firms are starting to use Java technology to implement these algorithms and thereby realizing productivity gains over more-traditional C/C++ development. However, in these time-critical systems, there is still no guarantee that at any instant, the process will not be interrupted by the Java platform garbage collector. Although trade execution occurs most of the time within an acceptable response time, the trade execution is delayed every once in a while, due to garbage collection or some other system event.

Sun's implementation of the Real-Time Specification for Java (JSR 001), the Sun Java Real-Time System, enables real-time processing through techniques that protect important threads from garbage collection and other system interrupts. This means that trading systems can confidently monitor the market and take action consistently within a calculated window of opportunity.

To demonstrate the impact of these techniques, Sun's OEM Software Systems Engineering technical team has written a demonstration of a trading system that uses real trade data. The demonstration compares a regular Java virtual machine against the Sun Java Real-Time System. For each run, a graph shows the difference in the actual trade price and the price when the trade should have executed. Running this trading system with the Sun Java Real-Time System shows that no money is lost due to garbage collection latencies. The results will be contrasted with the same application run with the standard (non-real-time) Java virtual machine.

This presentation demonstrates a working system and proves real-time Java technology's ability to satisfy hard real-time trading requirements.

Posted at 10:46AM Mar 30, 2007 by jtc in Sun  |  Comments[3]

Crosstool Environment in a Solaris Zone

Background
The task of building Java ME CDC-HI binaries and their associated cross development environments tends to be very linux-centric.  Utilities like Crosstool, which make this process much more tolerable, also make various linux and GNU assumptions that differ from standard Solaris.  By introducing new paths and GNU versions of applications, it is possible to mimic this environment in Solaris.   Furthermore, by creating a new zone with this environment, we can isolate these changes without affecting other Solaris system settings.

1. Create a Solaris zone called 'toolzone'

Note: This step is system specific.  For example, the hostname and IP address for 'toolzone' was predefined.  In addition, the network interface (i.e. 'set physical=iprb0') is likely to be different also.

Here's the command-line session for creating the zone:

phoenix://# mkdir /zone
phoenix://# zonecfg -z toolzone
toolzone: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:toolzone> create
zonecfg:toolzone> set zonepath=/zone/toolzone
zonecfg:toolzone> set autoboot=true
zonecfg:toolzone> add net
zonecfg:toolzone:net> set address=toolzone
zonecfg:toolzone:net> set physical=iprb0
zonecfg:toolzone:net> end
zonecfg:toolzone> info
zonepath: /zone/toolzone
autoboot: true
pool:
inherit-pkg-dir:
        dir: /lib
inherit-pkg-dir:
        dir: /platform
inherit-pkg-dir:
        dir: /sbin
inherit-pkg-dir:
        dir: /usr
net:
        address: toolzone
        physical: iprb0
zonecfg:toolzone> verify
zonecfg:toolzone> commit
zonecfg:toolzone> ^D

phoenix://# zoneadm list -vc
  ID NAME             STATUS         PATH
   0 global           running        /
   - toolzone      configured        /zone/toolzone

phoenix://# zoneadm -z toolzone install
Preparing to install zone <toolzone>.
Creating list of files to copy from the global zone.
Copying <6666> files to the zone.
Initializing zone product registry.
Determining zone package initialization order.
Preparing to initialize <945> packages on the zone.
Initialized <945> packages on zone.
Zone <toolzone> is initialized.
Installation of <2> packages was skipped.
Installation of these packages generated warnings: <SUNWcsu SUNWsogm>
The file </zone/toolzone/root/var/sadm/system/logs/install_log> contains a log of the zone installation.

2.  Configure the zone.

2a. Create a zone-specific /usr/bin directory and copy the contents of the global zone's /usr/bin to this new directory.

phoenix://# mkdir -p /zone/toolzone/usr/bin
phoenix://# cd /usr/bin
phoenix://# tar cf - . | (cd /zone/toolzone/usr/bin; tar xfp -)

2b. Create a loopback mount for toolzone to this new version of /usr/bin:

phoenix://# zonecfg -z toolzone
zonecfg:toolzone> addfs
zonecfg:toolzone:fs> set dir=/usr/bin
zonecfg:toolzone:fs> set special=/zone/toolzone/usr/bin
zonecfg:toolzone:fs> set type=lofs
zonecfg:toolzone:fs> end
zonecfg:toolzone> ^D

2c. Boot and configure the newly created zone:

phoenix://# zoneadm -z toolzone boot
phoenix://# zlogin -C toolzone

2d. Change the path of the default shell to /usr/bin/bash.  This is required for crosstool to operate correctly.

toolzone://# cd /usr/bin
toolzone:bin/# mv sh sh.ORIG
toolzone:bin/# ln -s /usr/bin/bash sh

2e. Create a new user in toolzone.  For this example, we'll use 'cdc'.  This exercise is left to the user.

toolzone://# grep cdc /etc/passwd
cdc:x:600:10:CDC-HI build user:/export/home/cdc:/usr/bin/bash

2f. Create a directoy called /opt/gnulinks and make sure the 'cdc' user owns it.  This directory will house versions and links  to GNU utilities which differ from their Solaris counterparts.

toolzone://# mkdir /opt/gnulinks
toolzone://# chown cdc:staff /opt/gnulinks
toolzone://# ls -ld /opt/gnulinks
drwxr-xr-x   2 cdc      staff        512 Jun 19 13:14 /opt/gnulinks

3. As the newly created 'cdc' user, configure the zone to build both the crosstool environment and CDC-HI

3a. Login to the zone as the 'cdc' user.

phoenix://# zlogin -l cdc toolzone
[Connected to zone 'toolzone' pts/6]
Last login: Mon Jun 19 12:57:05 on pts/6
Sun Microsystems Inc.   SunOS 5.10      Generic January 2005
toolzone:~/$

3b. To get a zone-specific prompt, you might want to have an entry in ~cdc/.bash_profile like:

export PS1='\h:\W/\$ '

3c.  Create links in /opt/gnulinks/bin which point to GNU versions of unix utilities.  This can be accomplised by executing the gnulinks.sh script which looks like:

#!/bin/sh
GNULINKS_DIR=/opt/gnulinks/bin
GNU_PROGS="ar as egrep grep m4 make nm objcopy objdump strings strip tar thumb"

mkdir -p ${GNULINKS_DIR}
cd ${GNULINKS_DIR}
for PROG in ${GNU_PROGS}
do
    ln -s /usr/sfw/bin/g${PROG} ${PROG}
done

toolzone:~/$ sh gnulinks.sh

3d. Modify PATH of 'cdc' user by putting the following line in ~cdc/.bash_profile:

export PATH=/opt/gnulinks/bin:/usr/sfw/bin:$PATH

toolzone:~/$ which tar
/opt/gnulinks/bin/tar

3e.  Build and install GNU specific utilities required for crosstool which differ from those provided by Solaris.  They include binutils, fileutils, gawk(1), patch(1) and sed(1).  A build-gnu-bin.sh is furnished to automate this, and looks like:

#!/bin/sh
GNULINKS_DIR=/opt/gnulinks

mkdir -p ${HOME}/GNU

for PROG in $*
do
    cd ${HOME}/GNU
    PREFIX=`echo ${PROG} | sed 's/-.*//'`
    wget ftp://ftp.gnu.org/gnu/${PREFIX}/${PROG}.tar.gz
    tar xzf ${PROG}.tar.gz
    cd ${PROG}
    ./configure --prefix=${GNULINKS_DIR}
    make
    make install
done

toolzone:~/$ sh build-gnu-bin.sh fileutils-4.1 gawk-3.1.5 patch-2.5.4 sed-4.1.4

3f. Create a native version of gcc(1).  The Solaris 10 version of gcc found in /usr/sfw/bin uses the stock Solaris linker, ld(1), found in /usr/ccs/bin.  Later versions of glibc will only accept the GNU version of ld(1).

toolzone:~/$ cd ~/GNU
toolzone:GNU/$ wget ftp://ftp.gnu.org/gnu/gcc/gcc-3.4.3/gcc-3.4.3.tar.bz2
toolzone:GNU/$ bunzip2 gcc-3.4.3.tar.bz2
toolzone:GNU/$ tar -xf gcc-3.4.3.tar
toolzone:GNU/$ cd gcc-3.4.3
toolzone:gcc-3.4.3/$ ./configure --prefix=/opt/gnulinks
toolzone:gcc-3.4.3/$ make 
toolzone:gcc-3.4.3/$ make install

3g. Build and install a GNU version of binutils using the aforementioned build-gnu-bin.sh script.

 

toolzone:~/$ sh build-gnu-bin.sh binutils-2.16

 4a.  As the 'cdc' user, build a cross development environment using crosstool.  These instructions are taken from http://www.kegel.com/crosstool/crosstool-0.43/doc/crosstool-howto.html#quick

toolzone:~/$ cd
toolzone:~/$ wget http://kegel.com/crosstool/crosstool-0.43.tar.gz
toolzone:~/$ tar -xzvf crosstool-0.43.tar.gz
toolzone:~/$ su -
toolzone://# mkdir /opt/crosstool
toolzone://# chown cdc:staff /opt/crosstool
toolzone://# exit
toolzone:~/$

4b. As an example, use these configuration files to build a crosstool for a Sharp Zaurus SL-5000D running OpenZaurus 3.5.1:

toolzone:~/$ cd ~/crosstool-0.43
toolzone:crosstool-0.43/$ cat oz-3.5.1.sh
#!/bin/sh
set -ex
TARBALLS_DIR=$HOME/downloads
RESULT_TOP=/opt/crosstool
export TARBALLS_DIR RESULT_TOP
GCC_LANGUAGES="c,c++"
export GCC_LANGUAGES

# Really, you should do the mkdir before running this,
# and chown /opt/crosstool to yourself so you don't need to run as root.
mkdir -p $RESULT_TOP

# Build the toolchain.  Takes a couple hours and a couple gigabytes.

eval `cat arm-softfloat.dat oz-3.5.1.dat` sh all.sh --notest

echo Done.

toolzone:crosstool-0.43/$ cat oz-3.5.1.dat
BINUTILS_DIR=binutils-2.15
GCC_DIR=gcc-3.4.2
GLIBC_DIR=glibc-2.3.2
LINUX_DIR=linux-2.4.18
GLIBCTHREADS_FILENAME=glibc-linuxthreads-2.3.2
GDB_DIR=gdb-6.4

4c. Build the toolchain

toolzone:crosstool-0.43/$ sh oz-3.5.1.sh > OUT.oz-3.5.1 2>&1 &

Posted at 04:34PM Mar 08, 2007 by jtc in Sun  |  Comments[0]

Solaris Was Real-time Before Real-time Was Cool

In the financial services market, there is a general trend to move key systems close to, or even right inside the exchanges themselves -- the idea being that the nearer you are to the source, the less network infrastructure and latency you'll experience.  With this advantage firms can potentially take on additional transaction loads at higher transaction rates.  These systems typically use the latest Intel or AMD processors and run a commercial distribution of Linux.¹

[1] Thank you Eric Bruno for your brief description, and for unknowingly letting me (slightly) plagiarize your comments.

Indeed these co-located systems perform as expected almost all the time.  But there are periodic intervals where the latency increases by several orders of magnitude, the ramifications of which could be financially disastrous.  After eliminating other components, the street seems to be focusing its wrath on commercial Linux distributions and their lack of real-time capabilities.

The linux community is actively working to include underpinnings to support real-time, but as of yet these capabilities are not part of a standard major (i.e. Red Hat, SuSE) distribution.  Instead, an alternate version of linux with real-time extensions is offered.  These extensions are in effect separate non-standard OS releases, and have not had the soak time required by many institutions.

Back in the early 90's, I volunteered to move over to Sun's newly formed SunSoft business unit.  One of it's main charters was to push the concept of running Solaris on alternate, i.e. Intel, platforms.  (Don't get me started here, can you imagine where Solaris would be right now if Sun had actually taken this initiative seriously back then?)  As part of that transition, I had the opportunity to take a Solaris Internals course, and couldn't help but notice the effort put in architecturally to address short latencies.  I still have the course notebook; it is dated September 1993.

The point is Solaris already has the real-time capabilities claimed by these add-on linux extensions.  It is built into the operating system, has been for quite some time, is rock solid and doesn't require any additional components.  A partial list of features include:

• Real-time Scheduling Class - In order to allow equal opportunity to system resources, traditional Unix schedulers transparently change process priorities to give competing processes a fair chance.  Although well suited for timesharing systems, this is unacceptable real-time behavior.  Since its outset, Solaris through its SVR4 roots, introduced the concept of alternate scheduling classes.  It includes a real-time scheduling class, which furnishes fixed-priority process scheduling at the highest priority levels in the system.
  

• Fine-Grained Processor Control / Processor Sets - Solaris allows threads and applications to be bound to specific individual processors. In addition, processors within a system can be grouped together as a processor set and dedicated to real-time tasks.²  Here's a nice article describing processor sets. Dated June 2001,  processor sets have been a part of Solaris since release 2.6.

• Interrupt Sheltering - Available since Solaris 7, this feature enables CPUs to be sheltered from unbound interrupts.  It can be used in conjunction with processor sets to shelter all CPUs in a processor set.  Note: At least one processor in the system must be kept unsheltered.
  

• Priority Inheritance - Priority inversion occurs when a high-priority thread blocks on a resource that is held by a lower-priority thread. A runnable thread with a priority between the high and low-priority threads creates a priority inversion because it can receive processor resources ahead of the high-priority thread. 

To avoid priority inversions with kernel synchronization primitives, Solaris employs a transient priority inheritance protocol. The protocol enables the low-priority thread holding the resource to “inherit” the higher priority of the blocked high-priority thread. This approach gives the blocking low-priority thread the CPU resources it needs to complete its task as soon as possible so that it can release the synchronization primitive. Upon completion, all threads are returned to their respective priorities by the kernel.³

• High Resolution Timers - Solaris 8 introduces the cyclic subsystem; this allows for timers of much better granularity -- in the microsecond and nanosecond range -- without burdening the system with a high interrupt rate.

• Memory Locking - The paging in and out of data from disk to memory may be considered normal behavior for virtual memory systems, but it is unacceptable for real-time applications.  Solaris addresses this problem by allowing the locking down of a process' pages into memory, using mlock(3C) or mlockall(3C) system calls.
  

• Early Binding - By default, linking of dynamic libraries in the Solaris is done on an as-needed basis. The runtime binding for a dynamically linked function isn't determined until its first invocation. Though flexible, this behavior can induce indeterminism and unpredictable jitter in the timing of a real-time application. To avoid jitter, the Solaris provides for early binding of dynamic libraries. By setting the LD_BIND_NOW environment variable to "1", libraries are bound at application startup time. Using early binding together with memory locking is a very effective approach to avoiding jitter.⁴

[2,3,4] Shameful plagiarism from Scalable Real-Time Computing in the Solaris ™ Operating Environment. A Technical White Paper. To further prove the maturity of Solaris' real-time features, this document was written in the Solaris 8 time frame.  It was copyrighted in 2000.

So why not give Solaris more consideration?  It's way more mature.  And in the unlikely event (chcukle, chuckle) that a lower-latency OS might not solve all your performance problems, I'd put my money on Solaris and DTrace over anything Linux could offer in finding the real problem.

Posted at 10:37AM Feb 23, 2007 by jtc in Sun  |  Comments[1]

The Verbal Regret Coefficient

I am convinced that scientists will some day find an explanation for my continual verbal ineptitude.  It will probably be identified as a sequence in our genome and they'll call it something like The Verbal Regret Coefficient.  We are all born with it, and we can't escape how it influences us everyday.  In advance of its discovery, I propose this definition: The Verbal Regret Coefficient is that innate ensemble of dispositions which guarantees that after uttering a number of words, you will regret having said a certain percentage of them. 

My first choice for a name was Regret Coefficient, but incredibly a quick search seems to indicate that the insurance industry already uses that term.   If my theory catches on -- and you'll know that when someone dedicates a wikipedia entry -- I don't want to have to deal with copyright and trademark infringement.  So Verbal Regret Coefficient it is.

The Verbal Regret Coefficient or VRC will ultimately be quantified, and like a cholesterol count, we'll all be assigned a VRC value.  Most will fall into an average range, but there will be outliers.  A high VRC manifests itself behaviorally in many ways: some high VRC'ers are purposely controversial, outspoken, brash or arrogant.  While others simply fail to think before they speak.  In case you're wondering, I am one of those outliers with a dangerously high VRC.  Furthermore, I'm quite sure I don't fit into the arrogant and controversial group.  But it's not all good news for those with a low VRC.   Although low VRC'ers tend not to say anything regrettable, they tend also not to say anything of substance either.  We call these people politicians.

If the current social trend continues, we'll get away with blaming any verbal faux pas to our VRC.  They'll be VRC support groups and celebrity VRC rehabilitation clinics.   Or maybe we could avoid all this silliness and just heed the advice of Mark Twain who once said:

It is better to keep your mouth closed and let people think you are a fool than to open it and remove all doubt.

But that is sooooo hard. 

Posted at 11:40AM Feb 16, 2007 by jtc in Personal  |  Comments[1]

Framework to Help Create Small Footprint RAM Resident Solaris Configurations

As Sun continues to avail more of its intellectual property to the community, the advantages Sun employees have regarding access to internal resources almost disappear.  In fact now when attempting to post questions to internal Sun mail aliases, I am often times redirected to the community.  The ramifications of this change hit me square in the gut this summer.

Having stumbled upon an internal project investigating how Solaris might be minimized for embedded use, I thought an interesting offshoot of this effort might be to create a ZFS appliance.  This device would boot from flash entirely into RAM, and all state would be maintained by the ZFS volumes.  Turns out this may be a little more tricky than anticipated, and future ZFS enhancements to Solaris (ZFS boot) may make this idea moot.

Based on an OpenSolaris ZFS discussion I initiated, observers went off and wrote about this topic elsewhere, some predicting that Sun would be releasing embedded ZFS appliances.  Whoa, hold on there, not so fast.  We have no plans (at least that I know of) to do any such thing.  This was nothing more than a pet project of mine.  Serves me right for announcing that I was a Sun employee.

But there was some good that came out of this dialog.  In addition to learning the valuable lesson of being careful what you write, interest in the notion of using Solaris as an "embedded" OS was quite apparent.   As a consequence,  I thought it might make sense to publish the basic framework used to create a custom Solaris miniroot.  Included below is the introduction section of the README file: 

1.0 Introduction

With the advent of Solaris 10 Update 1 and its migration to the grub(5) bootloader, it becomes quite feasible and straightforward to consider creating small footprint "embedded" versions of Solaris which boot directly into RAM. This project is based upon work done by Shudong Zhou to create a minimized Solaris for embedded use.  The doc/ directory contains some of the original documentation and scripts used to build such an environment.

It is expected that entities may want to provide further functionality and customizations to this environment. In order to assist in this endeavor, the original work has been enhanced to utilize the Java-based ant(1) build tool. For a further description on how a miniroot image is created, see the section on "Understanding the ant(1) build process".

You can download the framework here

Miscellaneous notes:

• This framework assumes availability of a standard Solaris distribution.  Although not confirmed, I suspect it may not be real hard to augment for OpenSolaris.
• The current framework produces a RAM resident version of Solaris that is about 60MB in size.  Note: there is no windowing included.
  
• These configs are system specific.  The reference implementation for the included framework is an iGoLogic i84810 motherboard.  Some key specs are:
• 1 GHz Celeron Processor
• 512 MB RAM on board
• Compact Flash slot on bottom of motherboard
• 146mm x 104mm
  
• Runs Solaris 10 U3 out of the box
  
• For more info contact iGoLogic an http://www.igologic.com
• Once an image is created, you'll need to set up grub(5) on your media.  Here's a pointer to a URL explaining Grub on a stick.  A copy of this URL is also included in the framework under the doc/ directory.
  
• Here's a picture of the motherboard with ruler included to give you a feel for its size. 
  

Posted at 04:33PM Feb 07, 2007 by jtc in Sun  |  Comments[6]