Bits and pieces Seongbae's random thoughts on code, compiler and processor.

Be careful with unlimited stack (or why my 32bit app can not use more than 2GB of memory)

Once in a while, this same question pops up on some of the mailing lists. Usually it's about some 32bit application trying to use more than 2GB of memory, or address space, to be exact. And the application mysteriously can not get past 2GB marker for no apparent reason - the system has plenty of memory and enough free swap space, and 64bit programs work just fine.

I've seen various reasons for this, like a bug in malloc() in libc for example. But by far the most common reason was due to unlimited stack size.

On Solaris, if you set the stack size to unlimited, your heap can not grow past 2GB. That's because the kernel reserves upper 2GB for stack usage. IMHO, this "unlimited" is a little misleading, since it actually means the maximum stack size is 2GB.

The kernel has to put a guard page (or pages?) at the end of the ulimit-specified maximum stack size, so that the overflow of the stack can be detected as well as interim pages are mapped on demand. This means that essentially ulimit-specified stack sized address space is always reserved for stack use.

Following code shows what's going on:

# cat t.c
#include 
#include 
#include 

int
main(void)
{
    pid_t pid;
    int *p = (int*)(0xff000000);

    *p = 0;
    pid = getpid();
    char buf[100];
    sprintf(buf, "pmap %d\n", pid);
    system(buf);
    return 0;
}
# cc -g t.c
# ulimit -a
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) unlimited
cpu time             (seconds, -t) unlimited
max user processes            (-u) 29995
virtual memory        (kbytes, -v) unlimited
# ./a.out
11267:  ./a.out
00010000       8K r-x--  /home/seongbae/tmp/a.out
00020000       8K rwx--  /home/seongbae/tmp/a.out
7FA80000     848K r-x--  /lib/libc.so.1
7FB64000      32K rwx--  /lib/libc.so.1
7FB6C000       8K rwx--  /lib/libc.so.1
7FB8A000       8K rwxs-    [ anon ]
7FB90000       8K r-x--  /platform/sun4u-us3/lib/libc_psr.so.1
7FBA0000      24K rwx--    [ anon ]
7FBB0000     176K r-x--  /lib/ld.so.1
7FBEC000       8K rwx--  /lib/ld.so.1
7FBEE000       8K rwx--  /lib/ld.so.1
FF000000   12288K rwx--    [ stack ]
 total     13424K
# ulimit -s 10240
# ulimit -a
core file size        (blocks, -c) unlimited
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 10
stack size            (kbytes, -s) 10240
cpu time             (seconds, -t) unlimited
max user processes            (-u) 29995
virtual memory        (kbytes, -v) unlimited
# ./a.out
Segmentation Fault (core dumped)
# dbx ./a.out
Reading a.out
Reading ld.so.1
Reading libc.so.1
(dbx 1) r
Running: a.out 
(process id 11275)
Reading libc_psr.so.1
signal SEGV (no mapping at the fault address) in main at line 11 in file "t.c"
   11       *p = 0;
(dbx 2) p p
p = 0xff000000
(dbx 3) quit
# 

( Apr 25 2005, 04:11:27 PM PDT ) Permalink Comments [2]