Tuesday July 12, 2005

Performance Improvements for the File System Cache in Solaris 10

One of the major improvements in Solaris 10 was the improvement of the file system cache. Here is a quick summary of the perf changes that I grabbed recently using Filebench.

The file system caching mechanism was overhauled to eliminate the majority of the MMU overhead associated with the page cache. The new segkpm facility providesfast mappings for physical pages within the kernel's address space. It is used by the file systems to provide a virtual addres that can be used to copy data to and from the user's address space for file system I/O.

Since the available virtual address range in a 64 bit kernel is always larger than physical memory size, the entire physical memory can be mapped into the kernel. This eliminates the need to have to map/unmap pages everytime they are accessed via segmap, significantly reducing code path and the need for TLB shoot downs. In addition, segkpm can use large TLB mappings to minimize TLB miss overhead.

Some quick measurements on Solaris 8 and 10 show the code path reduction. There are three important cases:

• Small In-Memory: When the file fits in memory, and is smaller than the size of the segmap cache
• Large In-Memory: When the file fits in memory, and is larger than the size of the segmap cache
• Large Non-Cached: When the file does not fit in memory

Measurements show that the most significant gain is a reduction of CPU per system call when a file is in memory (fits in the page cache), and is larger than segmap (which is 12% of physmem on SPARC, and 64MB on x64). Importantly, this is the most common case I've seen with real applications, too.

Random Read of 8k	Solaris 8	Solaris 10
Small In-Memory (Less than segmap)	89us	83us
Large In-Memory (Greater than segmap but less then physmem)	181us	98us
Large Non-Cached	236us	153us

The throughput gains for a CPU constrained system are shown here:

With Solaris 10 you should expect significant improvements for applications which are I/O intensive into the file system cache. The actual improvement will vary, and will be greater on systems with higher CPU counts. You should also expect to see the cross call count drop (see xcal in mpstat), and a significantly reduced amount of system time.

Technorati Tag: OpenSolaris

Technorati Tag: Solaris

( Jul 12 2005, 12:42:51 PM PDT ) Permalink Comments [3]