Friday June 01, 2007

Thread Local Storage support in Sun Studio compilers for x86/64

Typically, in a multithreaded process, all the threads belonging to a particular
process share the same address space, therefore a global or static variable would
reside in the same memory location. While such sharing of global
variables has its own advantages, sometimes it is desirable to have
a facility that allows a thread to have its private copy of a global
variable and not share it with other threads. Such a mechanism is called
Thread Local Storage (TLS). Several vendors of compiler and runtime systems
have implemented support for TLS. Sun Microsystems was an early adoptor
of TLS by impmenting its support in the Sparc-solaris platform.

Sun Studio compilers introduced support for TLS in the x86/x64 platform
in Sun Studio 10. TLS support for linux is being introduced in
Sun Studio 12.

A variable is declared as thread local by using the keyword __thread e.g.

__ thread int x;

A thread local variable can be statically initialized e.g.

__thread int x = 1;

However, dynamic initialization is not allowed. The & operator of a
thread local variable is evaluated at run time and returns the address
of the variable in the current thread. For more language specific
issues please refer to the C or C++ user's guide.

Besides language specific and compiler code generator support, TLS
requires extensive support from the runtime system including the dynamic and
static linkers, libc and threads library.

There are four models of thread local storage allocation -
general dynamic, local dynamic, initial exec and local exec.
Dynamic models are needed for supporting TLS in shared libraries
and the static models are recommended for use in applications which
are statically linked. Static TLS is significantly faster than
dynamic TLS. The compiler is required to generate different
code sequences for accessing thread local data in static or
dynamic TLS so that the linker can take the appropriate actions
to resolve the address pf thread local data. The details of
run time handling of thread local storage are described in the
solaris linker guide.

The solaris specifications for TLS are a little different from the gnu
variant for IA-32 and x86-64. The  implemention of TLS support in
Sun Studio 12 compilers for linux in x86/x64 was done following the gnu
variant of the TLS specifications.  Details about the gnu are  also available at
ELF Handling for Thread-Local Storage which also happens to be an
excellent resource for TLS description.

The user is expected to provide the compiler about choice of
static or dynamic TLS by using the compile time flag -xthreadvar.
While compiling to build a module with dynamic TLS access, it is
recommended to use -xthreadvar=dynamic. Just using -xthreadvar will
also result in dynamic TLS. Static TLS is indicated by
-xthreadvar=no%dynamic.

In absence of the -xthreadvar flag, the compiler tries to guess the
anticipated use by determining if position independent code is being
used (-KPIC/-Kpic). Otherwise, the code generator chooses static TLS
model by default (-xthreadvar=no%dynamic).

The choice of local dynamic or local exec models are by the compiler
or the linker under appropriate conditions.

The following example shows usage of TLS and also its comparison with
an alternative approach using pthreads_setspecific and pthread_getspecific:

% cat tlsbench.c

/* Copyright 2003 Sun Microsystems, Inc. All Rights Reserved
 */

#include <pthread.h>
#include <thread.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/time.h>

#define NUM_THREADS 5
#define NO_KEYS 8
#define LOOP_COUNT 1000000

/* compile as cc -o tlsbench -mt -xO3 tlsbench.c */

static void *thread_func_tls(void *arg);
static void *thread_func(void *arg);

/* global data */
static pthread_t tid[NUM_THREADS];  /* array of thread IDs */
static __thread int keys_tls[NO_KEYS];
static pthread_key_t keys[NO_KEYS]; /* list of keys */

static void *thread_func_tls(void *arg)
{
  int i, j, val;
  for(i = 0; i < LOOP_COUNT; i++){
    for(j = 0; j < NO_KEYS; j++)
    /* now get and set the local values */
    keys_tls[j] = j;
  }
  for(i = 0; i < LOOP_COUNT; i++){
    for(j = 0; j < NO_KEYS; j++){
    val = keys_tls[j];
    if ( val != j ){
    (void) fprintf(stderr, "Error getting val: %d\n", val);
    exit(-1);
    }
    }
  }
  return 0;
}
static void *thread_func(void *arg)
{
  int i, j, val;
  for(i = 0; i < LOOP_COUNT; i++){
    for(j = 0; j < NO_KEYS; j++)
      /* now get and set the local values */
      pthread_setspecific(keys[j], (const void *)&j);
  }
  for(i = 0; i < LOOP_COUNT; i++){
    for(j = 0; j < NO_KEYS; j++){
      val = *(int *)(pthread_getspecific(keys[j]));
      if ( val != j ){
        (void) fprintf(stderr, "Error getting val: %d\n", val);
     exit(-1);
      }
    }
  }
  return 0;
}

int main(int argc, char *argv[])
{
  int i;
  hrtime_t start, end;
  for ( i = 0; i < NO_KEYS; i++)
    pthread_key_create(&keys[i], NULL);
  /* first time get/set */
  start = gethrtime();
  for ( i = 0; i < NUM_THREADS; i++)
    pthread_create(&tid[i], NULL, thread_func, NULL);
  for ( i = 0; i < NUM_THREADS; i++)
    pthread_join(tid[i], NULL);
  end = gethrtime();
  (void) printf("Avg time using pthread_[get|set] = %.2f sec\n", (end - start)/1000000000.0 );
  /* now time tls */
  start = gethrtime();
  for ( i = 0; i < NUM_THREADS; i++)
    pthread_create(&tid[i], NULL, thread_func_tls, NULL);

  for ( i = 0; i < NUM_THREADS; i++)
    pthread_join(tid[i], NULL);
  end = gethrtime();
  (void) printf("Avg time using TLS          = %.2f sec\n", (end - start)/1000000000.0 );
  return 0;
}


This example and a few others are available in Sun Studio installations
e.g. in /opt/SUNWspro/examples/general/tls