manu's blog

Get rid of fopen file descriptor limits.

A usual problem for 32bits application using fopen call is the limitation

on the file descriptor stored in the FILE* structure. Even increasing

the quota on fd won't help. As soon as 255 is reach, fopen calls

fail. There is a lot blogs entries and discussion on the web regarding that

problem. This issue become even more complicated when you do not

own the code you are running (you are loading plugins, you are using third part code etc...). you cannot rewrite the executed code and you cannot prevent usage of fopen. Another example is when your aplication is a Java application and you

of course does not handle things like File object creation.
 

basically the trick about all this is to redirect all the call to fopen  and open

to newly created calls. Like a filter. The filter will "catch" the calls to open and make sure that returned fd is greater than 255.

In another hand , fd returned for fopen (used by fopen)cannot be greater than 255, so the trick is to make sure that fd less than 255 are always available.

As I said, you can find already lot's of page on the web about this. The idea 

is not mine at all but I did not find a complete implementation of it and

I was curious to try. Here is the code I used , I hope it can help somebody else. 

The idea is to have a new implementation of the open call. The real open is

called to actually open the file but the file descriptor returned is then

duplicate on a new file descriptor greater then 255. We have to know when we

(new open) are called from an fopen call because is that case , the fd returned must not be duplicated (hoping that the fd returned is less than 255)

To make that trick usable for non provileged user (or user with low quota)

we are using reasonable file descriptor range:

0 to 128 are dedicated to fopen

128 to current file descriptor limit are dedicated to open.

Note : calls to dlsym should also be protected by mutex.

This code as to be used as a shared library

cc -G -o libnewopen.so 

and used using LD_PRELOAD 

 LD_PRELOAD=./libnewopen.so ./<your progam>

#include <stdio.h>
#include <dlfcn.h>
#include <stdarg.h>
#include <thread.h>
#include <synch.h>
#include <sys/resource.h>
#include <stdlib.h>
#include <sys/types.h>
#include <unistd.h>
#include <fcntl.h>
#include <errno.h>

#ifdef _DEBUG
#define TRACE(A) printf A;
#else
#define TRACE(A)
#endif

#pragma init (stdio_filter_init)
#pragma fini (stdio_filter_fini)

static thread_key_t stdio_filter_key = NULL;
static int fd_max = 0;
static short safe = 1;
static mutex_t fd_lock;


void stdio_filter_init(void *arg);
void stdio_filter_fini(void *arg);
static void init_thread_data();
static void mark_fopen_as_called(short called);
static short is_fopen_called();
FILE *fopen(const char *filename, const char *mode);
static int get_free_fd();
int open(const char *path, int oflag, ...);

/*
 * - the filter must be thread safe.
 * - When we enter an fopen call we set a flag to warn open to not duplicate.
 * - we also need a mutex to be thrad safe when we choose a new file descriptor to return
 */

void stdio_filter_init(void *arg) {
  struct rlimit lim;

  if(thr_keycreate(&stdio_filter_key,NULL)) {
    /*we cannot protect thread*/
    /*disable all mechanism*/
    safe = 0;
    TRACE(("warning not safe mode\n"));
  }

  
  getrlimit(RLIMIT_NOFILE,&lim);
  fd_max = lim.rlim_cur;
  
  mutex_init(&fd_lock,USYNC_THREAD,NULL);

  TRACE(("init done\n"));

}

void stdio_filter_fini(void *arg){
  
  
  mutex_destroy(&fd_lock);

   TRACE(("fini done\n"));

}



static void init_thread_data() {
  short *thr_data = NULL;

  if (!safe) return;

  thr_getspecific(stdio_filter_key, (void**)&thr_data);
  
  if (thr_data == NULL) {
    thr_data = (short*)calloc(1,sizeof(short));

    thr_setspecific(stdio_filter_key, (void*)thr_data);
  }


}

static void mark_fopen_as_called(short called) {
  short *thr_data = NULL;
  
  if (!safe) return;

  thr_getspecific(stdio_filter_key, (void**)&thr_data);
  *thr_data = called;
  
  thr_setspecific(stdio_filter_key, (void*)thr_data);

  TRACE(("fopen marked as [%d]\n",called));

}

static short is_fopen_called() {
   short *thr_data = NULL;
  
  if (!safe) return 0;

  thr_getspecific(stdio_filter_key, (void**)&thr_data);
  
  return (*thr_data == 1);

}

FILE *fopen(const char *filename, const char *mode) {
        FILE *file = NULL;
	static void *(*actualfunction)();
	
	
	init_thread_data();

       

        if (!actualfunction) {
                actualfunction = (void *(*)()) dlsym(RTLD_NEXT, "fopen");
        }
	
	if (safe) {
	  /*if not safe : bypass our filter*/
          /*warn underlying open function that we are called from fopen*/
	  mark_fopen_as_called(1);
	}
	
	file = actualfunction(filename, mode);

	if (safe) {
	  /*if not safe : bypass our filter*/
	  mark_fopen_as_called(0);
	}

        return(file);
}

static int get_free_fd() {
  int idx;

 
  
  for (idx = 128 ; idx < fd_max; idx++) {
    /*check if fd is free*/
    if (fcntl(idx,F_GETFD,0) == -1 && errno == EBADF) {
      TRACE(("get_free_fd : fcntl on %5d returned [%d]\n",idx,errno))
      return idx;
    }
  }

  
  return(-1);

}

int open(const char *path, int oflag, ...) {
  static void *(*actualfunction)();
  int fd;
  int new_fd;

  va_list ap;

  mode_t open_mode = NULL;

  

  if (!actualfunction) {
    actualfunction = (void *(*)()) dlsym(RTLD_NEXT, "open");
  }

  init_thread_data();
  
  if ((oflag & O_CREAT)) {
    TRACE(("+++++++++++++ O_CREAT is used\n")); 
    va_start(ap,oflag);
    open_mode = va_arg(ap,mode_t);
    va_end(ap);
  } else {
    TRACE(("+++++++++++++ O_CREAT is NOT used\n")); 
    open_mode = (mode_t)-1;
  }

  if (open_mode != (mode_t)-1) {
    TRACE(("calling real open(%s,%d,%d)\n",path,oflag,open_mode));
    fd = (int(*)(char*,int))actualfunction (path,oflag,open_mode);
  } else {
    TRACE(("calling real open(%s,%d)\n",path,oflag));
    fd = (int(*)(char*,int,int))actualfunction (path,oflag);
  }  

  if (!is_fopen_called()) {
    /*not called from fopen, it is safe to push the fd further*/
    (void)mutex_lock(&fd_lock);
    
    new_fd = get_free_fd();

    if (new_fd == -1)  {
      TRACE(("warning failed to get a free fd returning [%d]\n",fd));
      new_fd = fd;
    } else {
      dup2(fd,new_fd);
      close(fd);
      TRACE(("open -> returning [%d] instead of [%d]\n",new_fd,fd));
    }

    (void)mutex_unlock(&fd_lock);

  } else {
    new_fd = fd;
  }

  return(new_fd);
  

}

Posted at 07:41PM août 08, 2007 by ejannett in Common  |  Comments[0]

Who is stealing my port ?

I often had to deal with problems about port already in use. The main question
was why my server cannot bind on port xxx ? On Linux the --program
option of netstat show you the id of the process currently owning a specific socket connection.

Solaris's netstat command does not have this option. Tired to scan the
/proc  with pfiles I've made a small script. I hope it can be useful to somebody else.

#!/bin/sh
if [ $# -ne 1 ]
then
    echo "usage $0 <port number>"
    exit 1
fi

PORT=$1
echo "looking for process consuming port $PORT"

old=`pwd`
cd /proc
for pid in *
do
  /usr/bin/pfiles $pid 2>/dev/null | /usr/bin/nawk -v port_num=$PORT '
            $0 ~ /^[0-9]*:/ {
                             pid=$1
                             program=$2
            }      
            $1 ~ /^[\t ]*sockname/ && $5 == port_num {
                  printf ("program [%d:%s] is bounded on %s:%d\n",pid,program,$3,port_num)
            }'
done
cd $old

Posted at 07:40PM août 08, 2007 by ejannett in Unix  |  Comments[3]

get stack trace of a detached jvm

To get the stack trace of a running java process the usual way is to hit Ctrl-\ on the terminal.

The jvm then print out its current state and threads'stacks.

This become a problem when the Java program is detached from the terminal as all the outputs are then discarded.

Starting to JDK5, the jstack tool is here to help or when you own the code running into the jvm and/or you can change

the way the jvm is launched an easy way is to redirect the standard streams to log file and then to send the

QUIT signal to the jvm pid.

I had to deal with another situation in which jvm was running on jdk 4 and It was not possible to change anything about it.

The trick I then used was to write a small script which catch all the write calls on  stderr on the jvm .

The script use strace(1) on Linux and truss(1) on Solaris to catch what is passed to write calls on file descriptor 1.

This script is available here : catch_vm_dump.tcl

Posted at 07:40PM août 08, 2007 by ejannett in Unix  |  Comments[0]