Friday May 27, 2005

Playing with STREAMS module from the User-Land

Playing with STREAMS module from the User-Land

In my previous STREAMS blog entry, we discussed, how to construct a do-nothing STREAMS module. Now we will come back to the user-land and see how we can play with STREAMS modules. We will learn, how to

• List all modules in the stream;
• Push a new module;
• Pop a module;

Show me the STREAM

Suppose that you have an open file descriptor fd and would like to know what STREAMS modules and drivers live behind the scene in the kernel in the STREAM representing the file.

The article by Rajesh Ramchandani on the Sun Developer Network provides an excellent example with the full source code of the printmod() function. It uses the I_LIST> ioctl which returns a list of modules in the struct str_mlist structure.

Please add my module

The next thing we are going to try is pushing our new module onto the STREAM. The following simple function should do the trick:

int pushmod(int fd, char *modname)
{
	int rc;

	if ((rc = ioctl(fd, I_PUSH, modname)) < 0) {
		perror("I_PUSH");
		fprintf(stderr, "pushmod(%d, %s) failed\n", fd, modname);
	}
	return rc;
}

The function takes a file descriptor and the module name and pushes the module on top of the stream. It returns 0 on success and -1 on failure. We can extend it a bit to put a whole list of modules. Suppose that the module list is a string with commas separating module names:

/* Push list of modules separated by commas */
int pushlist(int fd, char *s)
{
	char *comma = strchr(s, ',');
	int rc = 0;

	if (comma == NULL)
		return (pushmod(fd, s));

	*comma = '\0';

	if (((rc = pushmod(fd, s)) >= 0) && *(comma+1) != '\0') {
		*comma = ',';
		rc = pushlist(fd, comma + 1);
	}

	return (rc);
}

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <stropts.h>
#include <strings.h>

void main(int argc, char *argv[])
{
	if (argc == 1)
		return;

	if (pushlist(0, argv[1]) < 0)
		exit(1);
	exit(0);
}

We can name this program as pushmod.c and try it (assuming that you have installed the nullmodmodule from the previous example:

 $ cc pushmod.c -o pushmod
 $ strconf
 ttcompat
 ldterm
 ptem
 pts
 $ ./pushmod nullmod,nullmod,nullmod
 nullmod
 nullmod
 nullmod
 ttcompat
 ldterm
 ptem
 pts

I don't want to see you any more!

Finally, you may remove the module from the top of the stream using a simple call

  if ((rc = ioctl(fd1, I_POP, 0)) < 0) {
  	perror("I_POP");

Conclusions.

Now you know how to manipulate the content of the STREAM. You may want to play with it a bit and see what happens if you insert and remove some interesting modules.

NOTE: It is quite likely that your terminal window will become unusable as a result of your experiments. Many modules assume certain context and the are designed to play in concert with others, so your terminal may misbehave if it is incorrectly configured.

Saturday May 21, 2005

Open Solaris is Good for Linux!

Open Solaris is Good for Linux!

There is quite a lot of discussion of various reasons and motivations for Sun to open source its crown jewel - the Solaris operating system. Quite a few people, both internally at Sun and externally believe that the move will help Sun as a company, but here I'd like to explore why opening Solaris is good for everyone else - for Linux for FreeBSD and for the computer community, in general.

In my opinion, the real value of the OpenSolaris is the opening of a vast amount of knowledge about the design of very complex computer systems. To really appreciate the value of this knowledge it helps to think about the way humans learn and understand the meaning of things.

The following quote is from the paper by Marvin Minsky:

Castles In The Air.

The secret of what something means lies in the ways that it connects to all the other things we know. The more such links, the more a thing will mean to us. The joke comes when someone looks for the "real" meaning of anything. For, if something had just one meaning, that is, if it were only connected to just one other thing, then it wold scarcely "mean" at all!

That's why I think we shouldn't program our machines that way, with clear and simple logic definitions. A machine programmed that way might never "really" understand anything -- any more than a person would. Rich, multiply-connected networks provide enough different ways to use knowledge that when one way doesn't work, you can try to figure out why. When there are many meanings in a network, you can turn things around in your mind and look at them from different perspectives; when you get stuck, you can try another view. That's what we mean by thinking!

That's why I dislike logic, and prefer to work with webs of circular definitions. Each gives meaning to the rest. There's nothing wrong with liking several different tunes, each one the more because it contrasts with the others. There's nothing wrong with ropes - or knots, or woven cloth - in which each strand helps hold the other strands together - or apart! There's nothing very wrong, in this strange sense, with having all one's mind a castle in the air!

To summarize: of course no computer could understand anything real -- or even what a number is - if forced to single ways to deal with them. But neither could a child or philosopher. So such concerns are not about computers at all, but about our foolish quest for meanings that stand by themselves, outside any context. Our questions about thinking machines should really be questions about our own minds.

In Minsky terminology, Solaris source is an extremely rich body of interwoven knowledge about the design of the state of the art computer systems. This body of knowledge was produced (and packed in the form of C code) in the course of many years of Solaris development by many extremely competent engineers. For a long time this knowledge was only available only to the small community of engineers and soon it will be available to everyone curious enough to tap it.

I am not suggesting that the knowledge embedded in other operating systems source code is any "better" or "worse" than the one embedded in the Solaris code. It was created by different people having different background, different objectives, different environments and different customer bases. It is just different. And, together, all of these provide even richer web of knowledge, that is much more useful then each individual part because they represent quite different dimensions.

So why is opening up a bunch of source code is really important to Linux (or FreeBSD, or any other software project)? Because, someone who takes the time and effort to read and understand even small parts of this embedded knowledge will almost certainly get new insights in whatever projects he or she is currently working on or thinking about. Even if the developer will not reuse any single line of the code, he will, definitely, gain in understanding his own area of expertise. Not to mention the trivial fact that the CDDL license allows developers to directly build their software based on the Solaris source. Consider, for example, an "open-sourcing" of a small part of Solaris design - the Slab Allocator, made by Jeff Bonwick in the form of the USENIX Paper and the followup paper. Was it useful to Linux and other software projects? If we ignore that fact that the slab allocator based on these papers is now the standard Linux kernel memory allocator, I am pretty sure that just reading these two papers was a very useful journey for a reader. And, be assured that the person who invented the Slab Allocator has more to say - and, indeed, says a lot - in C.

Another, more recent, example is DTrace, which is already available. It is not immediately obvious why DTrace is good for Linux, consider how much more effort is now put in creating the adequate Linux tracing facility that could compete with DTrace! Even if not a single line of DTrace source will find its way into the Linux distribution, it would definitely serve as a "prove of existence" and an inspiration. And, as Linux tracing facility will improve under the influence of DTrace, DTrace itself will improve to stay a relevant tool.

As a result of such cross-influence of ideas, the whole body of available software improves in its quality and the coverage and everyone wins!

And for this reason it makes sense for many computer engineers, students and just curious minds around to set aside some time to read and understand some parts of the OpenSolaris source code. And those who think that something embeds the whole meaning should, probably, reread the Minsky paper.

Technorati Tags: OpenSolaris, Solaris.

Friday May 13, 2005

STREAMS Hello World

Ever wondered, what would it take to write a minimum STREAMS module that can be correctly installed onto Solaris system? Here is an example of such module (which we will call nullmod) with some explanations.

Every STREAMS module should define the following entry points:

o Module open routine;

o Module close routine;

o Module read side put procedure;

o Module write side put procedure;

o Module read side service procedure (optional);

o Module write side service procedure (optional);

Since our module will only pass messages back and forth it will use putnext(9f) as both write and read side put procedure and will not have any service procedures. So we need to define only open/close functions which we will call nullmodopen() and nullmodclose().

  /*
   * Module open routine.
   * Mark the module as "opened" and link it to
   * the STRREAM.
   */
  static int
  nullmodopen(queue_t *rq, dev_t *dev,
        int oflag, int sflag, cred_t *crp)
  {
        if (sflag != MODOPEN)
                return (EINVAL);

        /*
         * Prevent duplicate opens.
         * The q_ptr is reserved for module private use
         */
        if (rq->q_ptr != NULL)
                return (0);

        /* Mark the module as "opened" */
        rq->q_ptr = WR(rq)->q_ptr = (void *)1;

        /* Link the module into the STREAM */
        qprocson(rq);
        /*
         * At this point module is linked in the STREAM
         * and can send/receive messages.
         * Its put/service procedures may execute at any time.
         */
        return (0);
  }

  /*
   * Module close routine.
   * Disconnect the module from the STREAM.
   */
  static int
  nullmodclose(queue_t *rq)
  {
        /* Disconnect the module from the STREAM */
        qprocsoff(rq);
        /*
         * At this point module is disconnected from the STREAM and can
         * no longer receive messages. Its put or service procedures are not
         * running.
         */
        rq->q_ptr = WR(rq)->q_ptr = NULL;
        return (0);
  }

This is pretty much the only code we need to write. The rest is the glue code that should be present in any module. Here is the description of this glue code.

Every source file should start with comments, so we shall start the nullmod.c with the appropriate comment:

  /*
   * Nullmod: the minimal functioning STREAMS module.
   *
   * Copyright ..... (place your favorite one here).
   *
   */

We will need certain system include files:

  /*
   * Required include files.
   */
  #include      <sys/types.h>
  #include      <sys/conf.h>
  #include      <sys/cred.h>
  #include      <sys/ddi.h>
  #include      <sys/modctl.h>

As we discussed before, our module will define only two functions - nullmodopen() and nullmodclose():

  /*
   * Function prototypes.
   */
  static int    nullmodopen(queue_t *, dev_t *, int, int, cred_t *);
  static int    nullmodclose(queue_t *);

Every STREAMS kernel module should have a corresponding module_info(9S) structure:

  static struct module_info     nullmod_minfo = {
        1,              /* mi_idnum */
        "nullmod",      /* mi_idname */
        0,              /* mi_minpsz */
        INFPSZ,         /* mi_maxpsz */
        0,              /* mi_hiwat */
        0               /* mi_lowat */
  };

Also it should have qinit(9S) structures for both the read and write sides:

  static struct qinit nullmod_rinit = {
        (int (*)())putnext,     /* qi_putp */
        NULL,           /* qi_srvp  */
        nullmodopen,    /* qi_qopen */
        nullmodclose,   /* qi_qclose */
        NULL,           /* qi_qadmin */
        &nullmod_minfo, /* qi_minfo */
  };

  static struct qinit nullmod_winit = {
        (int (*)())putnext,     /* qi_putp */
        NULL,           /* qi_srvp */
        NULL,           /* qi_qopen */
        NULL,           /* qi_qclose */
        NULL,           /* qi_qadmin */
        &nullmod_minfo, /* qi_minfo */
  };

The streamtab(9S) structure links both together:

  static struct streamtab nullmod_info = {
        &nullmod_rinit, /* st_rdinit */
        &nullmod_winit, /* st_wrinit */
  };

The fmodsw(9S) structure describes tqhe module to the operating system, providing the pointer to the streamtab structure above:

  static struct fmodsw fsw = {
        "nullmod",      /* module name */
        &nullmod_info,  /* streams information */
        D_MP            /* module flags - multithreaded module */
  };

Now we need to provide the linkage information for the module:

  /*
   * Module linkage information for the kernel.
   */
  struct mod_ops mod_strmodops;

  static struct modlstrmod modlstrmod = {
        &mod_strmodops, "Example pass-through module 1.0", &fsw
  };

  static struct modlinkage modlinkage = {
        MODREV_1, (void *)&modlstrmod, NULL
  };

Every loadable kernel module should also provide _init, _fini and _info entry points.

_init(9E) initializes a loadable module. It is called before any other routine in a loadable module. Most modules do not require any specific initialization and can just call mod_install(9F). _fini(9E) prepares a loadable module for unloading. It is called when the system wants to unload a module. In most cases it can just call mod_remove(9F). _info(9E) returns information about a loadable module. In most cases _info(9E) just returns the value returned by mod_info(9F).

  /*
   * Standard module entry points.
   */
  int _init(void)
  {
        return (mod_install(&modlinkage));
  }

  int _fini(void)
  {
        return (mod_remove(&modlinkage));
  }

  int_info(struct modinfo *modinfop)
  {
        return (mod_info(&modlinkage, modinfop));
  }

Now we need to put our definitions for nullmodopen() and nullmodclose() above and we are done! Our ``do-nothing'' module is ready.

Let us save the module in the file nullmod.c. To compile it we need a C compiler which can generate the code for the native kernel mode - 32-it for x86 and 64-bit for Sparc or AMD64 platforms. If we are using Sun compilers we can use the following commands to produce the module binary on sparc:

  cc -c  -D_KERNEL -D_SYSCALL32 -D_SYSCALL32_IMPL -xarch=v9 nullmod.c

Or with gcc

  gcc -c -D_KERNEL -D_SYSCALL32 -D_SYSCALL32_IMPL -m64 nullmod.c

After that we need to use ld with -r option to produce the final binary:

  ld -r -o nullmod nullmod.o

Now we need to copy our module to /usr/kernel/strmod/sparcv9 (you need to be a super-user for this):

  # cp nullmod /usr/kernel/strmod/sparcv9

!!WARNING!!: you are going to install the kernel module. It is possible that the system will panic if there is some problem in the module. Please make sure that no one else is using the system and you will not upset anyone (including yourself) if the system panics as a result of your experiments. End of WARNING.

Now we can load the module:

  # modload nullmod

And verify that it is loaded:

  # modinfo | grep nullmod
  147 7bbbde30 320 - 1 nullmod (Example pass-through module 1.0)

We can also insert our module onto STDIN for our shell:

  # strchg -h nullmod

and verify that it is present:

  # strconf 
  nullmod
  ttcompat
  ldterm
  ptem
  pts

When we are done, we can remove the module from our shell STDIN:

  # strchg -p
  # strconf 
  ttcompat
  ldterm
  ptem
  pts

You can also unload the module (it is usually not rquired as Solaris unloads unused modules itself):

  # modunload -i 147
  # modinfo|grep nullmod
  #

Now you know how to

- Write a trivial Solaris STREAMS kernel module

- Compile the module

- Install the module on your system

- Load and unload the module

Why this module may be of any use? I can think of several reasons:

It provides a good template for more complicated modules

You can use it to estimate performance impact of inserting a module in a STREAM. This is the fastest module possible, so any additional code will only slow things down.

You can demonstrate your Solaris kernel skills duuring the job interview.

Not bad for a simple do-nothing program!