The Random Bit Bucket: Moinak Ghosh's Weblog

Recently I have been doing a bunch of performance work on the BeleniX LiveCD. BeleniX version 0.3 used to have bad boottime performance and it has improved significantly on 0.4 and I am working on further improvements.

While doing all this I have also been grokking the search engines looking for writeups and resources on this topic. Unfortunately such information is difficult to find. I couldn't find a good guide to tweaking livecds for performance. So I will be listing out my efforts here and in future blog posts. Hopefully this will come in handy for others as well.

A) The performance bottlenecks

1) As I had mentioned in earlier posts a CDROM is not a good random access device since it has high access time. high seek time + high rotational latency. It is however good at continuous sequential access.

2) SMF starts services in parallel. This may be good for a harddisk boot but causes some pain during CDROM boot.

3) Starting heavyweight apps linked to lots of libraries has significant delays because the library contents are accessed piecemeal. That is different sections of the libraries are loaded at different times. This causes a lot of random access that moves the CDROM head all over the place.

B) Approaches to tackle the above issues

1) Firstly the critical piece of any LiveCD is compression. The CD contents must be compressed to pack in enough apps to make the CD useful. 700MB uncompressed space is just not enough. In addition compression reduces the amount of I/O increasing performance. Apart from reducing I/O more data is packed in a physically smaller area so the CDROM head movement is reduced.

2) However compression alone is not enough. Physical arrangement of file data on the CDROM matters. If the file data is scattered at various places on the CDROM surface then performance suffers. So the most frequently accessed files must be kept in contiguous storage on the CD. This is possible using the "-sort" option of mkisofs. The sorted list of files is obtained via a trial boot of the CDROM with dtrace enabled and running the iosnoop.d script from the DTrace Toolkit. This script lists all the files accessed and in what order. I order the files on the CD based on processing the iosnoop output. This is not exactly accurate since SMF implements parallel service startup so the order will vary. But in general the frequently requested files during boottime will be clustered closely together and will reduce boottime. This has greatly improved startup of GUI desktops like KDE.

3) Disable unneeded services. About 15 default OpenSolaris services are disabled in the BeleniX LiveCD as they are not required in a LiveCD envronment.

4) Use crle. This utility called Configure Runtime Linking Environment can store a cache of all libraries and their dependencies. Thus lookup in the search path and LD_LIBRARY path for shared objects is avoided improving application load time.

5) Use fc-cache. This utility from fontconfig creates a cache of font information in the various font directories. This reduces font lookup and load time significantly. The caches are generated once and then included while building the LiveCD image.

6) Some services do a lot of processing like setting up quotas in their startup scripts. Some of these are not relevant and are skipped in a LiveCD environment.

7) Use full path while executing commands in startup scripts. This avoids having the shell to do a PATH lookup.

8) When starting GUI desktops, preload most of the required libraries into the page cache. This is done via mmap. This reduces the piecemeal loading of shared libraries since the library contents are now fetched from the cache. Preloading has had a significant effect in KDE startup performance.

9) Use a prebuilt ksycoca cache for KDE instead of building it each time. To do this KDE is started once and the ksycoca file if copied. Then the "System Configuration Startup Check" in KDE is disabled and the copied cache is included while building the LiveCD.

10) Disable the KDE startup sound and a couple of KDE serivces having less relevance.

11) Simplify KDE and Xfce startup scripts. A lot of the functionality are not required in a LiveCD so can be skipped.

12) In addition the KDE and Xfce startup scripts execute which to find out the locations of some executables like ssh-agent. In Solaris which is a csh script that takes a while to execute and hammers the CD quite a bit - found it from iosnoop.d output. So I changed the scripts to first look for a specific environment variable which will contain the full path and then call which if the variable is not set.

13) Reduced the wait time while probing for DHCP on an interface to 5 secs.

C) Optimizations to be looked at next:

1) Reduce space usage in the ramdisk by moving /lib, init etc onto the CD. init will be replaced by a shell script that runs in a custom shell having minimal built-in versions of mount, ls etc. It will mount the CD and pass control to the real init.

2) Do block level profiling in lofi and rearrange compressed blocks to achieve best-effort optimization.

3) Currently preloading is done via mmap and then a loop to touch every page. This however causes the compressed page to be faulted in from lofi and then decompressed and passed back to the filesystem above lofi. This causes unnecessary decompression overhead double caching of the compressed and uncompressed data reducing the effectiveness of preloading if memory is less. Instead it is enough to just preload and cache the compressed pages without decompressing them. An enhancement needs to be done in lofi to achieve this.

4) After booting the CDROM can spin down after about 30 secs leading to subsequent delays in starting apps/commands till the CDROM spins up again. A small daemon is needed that can issue spin-up requests to the CDROM to keep it active. However it should also be able to detect activity and keep increasing the spin-up interval if no user activity is detected.

5) Currently "Xorg configure" is used to generate the initial xorg.conf which is then refined. This contends a lot with the other services vying for CDROM access. The need here is to have a complete Xorg auto configurator having a videocard database that avoids having to run "Xorg configure".