Weblog

x86 32 bit virtual address space - the "small" frontier...
Users running applications on Solaris x86 32-bit kernel should be aware of the limitations of the User virtual address space especially on larger physical memory configurations.¹

For performance reasons, the 32 bit virtual address space has to be shared between User and Kernel. The size of the Solaris Kernel address space has had to be increased due to various Kernel address space exhaustion problems which included bug 4979763.²  In addition, larger physical memory configurations require more Kernel virtual memory to manage the physical memory. This, of course, has the side effect of decreasing the amount of available User address space. The following comment from the Solaris x86 32 bit startup code shows the approximate division of the Kernel and User address spaces for different memory configurations.

 * kernelbase: On a 32-bit kernel the default value of 0xd4000000 will be
 * decreased by 2X the size required for page_t.  This allows the kernel
 * heap to grow in size with physical memory.  With sizeof(page_t) == 80
 * bytes, the following shows the values of kernelbase and kernel heap
 * sizes for different memory configurations (assuming default segmap and
 * segkp sizes).
 *
 *      mem     size for        kernelbase      kernel heap
 *      size    page_t's                        size
 *      ----    ---------       ----------      -----------
 *      1gb     0x01400000      0xd1800000      684MB
 *      2gb     0x02800000      0xcf000000      704MB
 *      4gb     0x05000000      0xca000000      744MB
 *      6gb     0x07800000      0xc5000000      784MB
 *      8gb     0x0a000000      0xc0000000      824MB
 *      16gb    0x14000000      0xac000000      984MB
 *      32gb    0x28000000      0x84000000      1304MB
 *      64gb    0x50000000      0x34000000      1944MB (*)

The 32 bit virtual address space is split at the 'kernelbase' variable with the User address space occupying the virtual space below kernelbase and the Kernel virtual address space occupying the space above. For example, the default size of the User virtual address space is less than 1 Gigabyte (0x34000000) for configurations with 64 Gigabytes of physical memory.

Applications that require large amounts of memory may not run or may run in a degraded mode because of the size of the User virtual address space is not sufficiently large. For example, trying to run java with a 3 Gigabyte large heap (e.g. java -Xms3600m -Xmx3600m) will likely fail to start on all but the smaller physical memory configuration. Users finding themselves in such a situation with a large memory application can try the following work-arounds:

Manually configure the 'kernelbase' kernel variable via the eeprom(1M) command to a higher value. This effectively increases the size of the User address space while decreasing the Kernel address space. Please note that this may increase the risk for Kernel virtual address space exhaustion.

Deconfigure parts of physical memory to reduce the amount of Kernel address space to manage the physical memory. This can also be accomplished by setting the 'physmem' kernel variable to a smaller value.

The best way to overcome limitations in the 32 bit Virtual address space is to run on the Solaris x64 (64 bit) kernel. 32 bit applications that run on the Solaris x86 32 bit kernel are guaranteed to run on the Solaris x86 64 bit kernel with the added bonus that the application can have close to the full 32 bit (4 Gigabytes) of User virtual address space available.³ Better yet, applications that can be converted to 64 bits will effectively have no User virtual address space limitations.

---

¹ PAE (Physical Address Extension), in 32 bit processors, provides for an additional 4 physical address bits to address up to 64 Gigabytes (36 address bits) of physical memory.
² A lookup of this bug will show a bunch of related Kernel virtual address space exhaustion bugs.
³ The User virtual address space for 32 bit processes will be limited to 3 Gigabytes on "older" AMD processors with Erratum 95. See AMD Revision Guide.

---

Technorati Tag: OpenSolaris
Technorati Tag: Solaris ( Jun 13 2005, 05:12:20 PM PDT ) Permalink Comments [14]