I've been away from the spe code for some time doing bug work, gatekeeping, and web design. I'm back to it and trying to get it ready to integrate into our development gate. I've decided to move the spe decision making into the kernel and as such I'll need to pass the policies from user space to the kernel.

Traditionally, you have to define 64 and 32 bit versions of your data structures (user land is typically 32 bit address space). Your application packs in the data and sends it to the kernel via syscall. And then you have to copy your data from user space to the kernel.

The STRUCT_* macros in the following code chunk show the steps you take:

1. You declare a local variable, u_sped in this case, which allows two views of the memory (see STRUCT_DECL(9F))
2. You determine whether the user land address space is 32 or 64 bit
3. You configure your local variable to use the current model
4. You copy the memory from user land (sped_in in this case) to the kernel address space
5. You get a local copy out of there.
6. You get a local copy out of there.
7. This one is different, it is a variable length "string"

void
nfs4_sped_svc(struct nfssped_args *sped_in)
{
        spe_policy_t    *sp = NULL;
        nfssped_op_t    opcode;
        char            *buf = NULL;
        size_t          len;
        size_t          tlen = 0;

        XDR             xdrs;

        model_t         model;

        STRUCT_DECL(nfssped_args, u_sped);  /* 1 */

        model = get_udatamodel();  /* 2 */

        /*
         * Initialize the data pointers.
         */
        STRUCT_INIT(u_sped, model); /* 3 */
        if (copyin(sped_in, STRUCT_BUF(u_sped), STRUCT_SIZE(u_sped))) {   /* 4 */
                set_errno(EFAULT);
                return;
        }

        opcode = STRUCT_FGET(u_sped, nsa_opcode);  /* 5 */
        len = STRUCT_FGET(u_sped, nsa_xdr_len);    /* 6 */

        if (len) {
                buf = kmem_zalloc(len, KM_SLEEP);

                if (copyinstr(STRUCT_FGETP(u_sped, nsa_xdr),         /* 7 */
                    buf, len, &tlen)) {
                        goto err_out;
                }
        }

One problem is that you need to know how big the buffer is to extract from it. If you have a fixed number of elements, i.e., 1, this is an easy problem. But I want to send an arbitrary number of elements. I could cycle through a linked list, sending one at a time, but that sounds gross.

But XDR was created to handle differences in network byte order, machine address ranges, variable number of data sets, etc.

So we take a pass in userland to encode the data into XDR, which also tells us how large the resulting data will be. And then we can unpack it in the kernel.

The largest issue I faced was that you typically start out with XDR notation and use rpcgen to produce headers and code. I already had the data structures, so I took my header and turned it into a .x file. And I had a hard time getting a union in there:

typedef union {
        uid_t           uid;
        gid_t           gid;
        int             i;
        char            *sz;
        spe_network_t   net;
} spe_data_t;

I would have twigged onto what was needed if I had started earlier - discriminated unions. So now I need to provide a way to tell the XDR code how to determine which field was to be used. Luckily, I knew that:

typedef enum {
        SPE_DATA_ADDR,
        SPE_DATA_GID,
        SPE_DATA_INT,
        SPE_DATA_NETNAME,
        SPE_DATA_NETWORK,
        SPE_DATA_STRING,
        SPE_DATA_UID
} spe_type_t;

I rearranged the .x file to get:

enum spe_type {
        SPE_DATA_ADDR,
        SPE_DATA_GID,
        SPE_DATA_INT,
        SPE_DATA_NETNAME,
        SPE_DATA_NETWORK,
        SPE_DATA_STRING,
        SPE_DATA_UID
};

union spe_data switch (spe_type data_type) {
        case SPE_DATA_UID:
                uid_t           uid;
        case SPE_DATA_GID:
                gid_t           gid;
        case SPE_DATA_INT:
                int             i;
        case SPE_DATA_NETNAME:
        case SPE_DATA_STRING:
                char            *sz;
        case SPE_DATA_ADDR:
        case SPE_DATA_NETWORK:
                struct spe_network   net;
};

But that produced the following for the header file:

struct spe_data {
        spe_type data_type;
        union {
                uid_t uid;
                gid_t gid;
                int i;
                char *sz;
                struct spe_network net;
        } spe_data_u;
};
typedef struct spe_data spe_data;

I wanted a nice clean separation, so I fought this for a while. But it is logical. In the end I went with:

typedef struct {
        spe_type_t      sd_type;
        union {
                uid_t           uid;
                gid_t           gid;
                int             i;
                char            *sz;
                spe_network_t   net;
        } sd_u;
} spe_data_t;

It fits my naming style and I also avoided doing the following:

#define uid sd_u.uid
#define gid sd_u.gid

Of course, with that, I should pick more unique field names. And what I hate about this method is that if you are sitting there with one window open to code and another on a debugger, you can't examine foo.uid. You have to dig through the headers to find that you really need to be looking at foo.sd_u.uid. I find it easier to avoid the macros.

Like everyone else at Sun, we've cut over to Mercurial to manage our source code. We have our own gate and we want to be able to view the set of applied changes. hg serve will do that for you. And immediately, I asked the group if someone had a smf service for it.

The answer was no. So Robert Gordon kindly supplied one.

You place hg_serve.xml in /var/svc/manifest/network and hg-serve in /usr/local/smf_method (you can place it elsewhere, just modify the path in hg_serve.xml).

And then comes the fun part of activating the service. I had to read the man page. :)

# svccfg import hg_serve.xml 
# svcadm enable svc:/network/hg_serve

And now the hg serve will survive a reboot!

I'm trying to prove that I've fixed a bug and I happen to know that if a certain function returns an error code, then I might have hit the code in question. How do I quickly determine whether or not I have hit such a case?

[root@pnfs-3-11 ~/dtrace]> more mms.d 
#! /usr/sbin/dtrace -Fs

:nfs:nfs4_trigger_mount:return
/args[1] != 0/
{
        printf("rc1 = %d\n", args[1]);
}

:nfs:nfs4_ephemeral_umount:return
/args[1] != 0/
{
        printf("rc1 = %d\n", args[1]);
}

[root@pnfs-3-11 ~/dtrace]> ./mms.d
dtrace: script './mms.d' matched 2 probes
CPU FUNCTION                                 
  0  <- nfs4_ephemeral_umount                 rc1 = 16

  0  <- nfs4_ephemeral_umount                 rc1 = 16

So if the function name is nfs4_trigger_mount or nfs4_ephemeral_umount, then print the return code if it is not 0. Hmm, I should probably remove the '\n's.

[root@pnfs-3-11 ~/dtrace]> ./mms.d
dtrace: script './mms.d' matched 2 probes
CPU FUNCTION                                 
  1  <- nfs4_ephemeral_umount                 rc1 = 16
  1  <- nfs4_ephemeral_umount                 rc1 = 16

I'm actually looking for an 11 (EAGAIN) coming back from nfs4_trigger_mount, but the abundance of 16 (EBUSY) from nfs4_ephemeral_umount tells me I've probably fixed another bug I was hitting.

Down for routine maintenance - you can at least enjoy the following until then:

• Mirror Mounts
• spe - Simple Policy Engine

Cthon '08 went off without a hitch. It started out uneventfully as Kerberos worked right out of the box. Evidently Sun's Kerberos team have been working on making initial configuration being painless. And they succeeded.

The public talks were well received and we've started posting the slides as they are sent in. You can check them out on Talks 08.

I'll post more as they arrive.

Also, we videoed most of the talks this year. As that content becomes available, we will post it up as well.

Be sure to visit www.connectathon.org and see when the talks are scheduled. These are open to the public.

Sun Microsystems, Inc. is involved with 6 presentations and then NetApp has 5 of them. I'll be giving two of them, but I'm actually more excited about the one on nfsreplay by Shehjar Tikoo and the Linux development git one by Bruce Fields and Benny Halevy.

Normally we can't share images of the event, but here is one from before the other vendors setting up their gear:

Each of the Sun workstations is probably a node in a pNFS community.

I've given up trying to explain to people what it is I do for a living. But I think I'm going to keep on trying to explain what a BakeAThon is to them.

How can I do one without the other? Well, I can abstract the process.

So a BakeAThon (and a ConnectAThon) is an interoperability testing event. And the best way to describe it is that you have 10 different people with a set of rules for a game (if you say AD&D edition 3 rules, you have another set of problems to deal with). Each of them has read the rules and believes that they know all of the intricacies. But none of them have played the game with anyone else.

So they all get together and start to play each other. And they start to argue about each and every move. Sometimes it is pretty obvious whose interpretation is wrong. And sometimes they call someone else over to help decide.

As soon as player A is done with player B, they start with player C. Except sometimes they are also playing with player D at the same time. Or player B comes back to see if they have gotten rule 5.3 correct now.

Sometimes they all vote on how to interpret a rule and even change the rule book. And sometimes player F was at the bathroom when that happened and causes the debate to start back up again.

Then they all go away again for 3 months, promising to play games against each other remotely. They meet back up again at the next BakeAThon - sometimes there is a new player or someone didn't show up. But they are willing to chime in over email.

But they have to start all over from scratch because no one played remotely and they've been busy playing with themselves.

A further complication is that some people only play defense and some only play offense. Sometimes you get a team where they split those duties. So when you talk to one person about how they run their offense, they shrug and say that they only do defense. And the problem that arises here is when the team's offense only plays against their defense - they get pretty good at it and understand some simple shortcuts that make it easy. But when they play another team, those same shortcuts cause problems.

So a BakeAThon is pretty much like that. The major difference is that the competitiveness isn't in winning a game but in getting the game adopted by other people. I.e., Foo Inc. and Bar Inc. may have differences and fight over customers, but while at a BakeAThon, they work together to make NFS a better protocol.

I've slammed the Linux NFSv4 implementation before for not having the same namespace as NFSv3. I.e., it used the 'fsid=0' hack to export the root of the v4 namespace and thus that path may not be the same as '/'.

Well, over on the nfsv4 <at> linux-nfs.org mailing list, Steve just announced a prototype which fixes that problem! And the crowd goes wild!

The following patch series gives rpc.mountd the ability to allocate
a dynamic pseudo root, so the 'fsid=0' export option is no longer 
required. This allows v2, v3 and v4 clients mounts without any 
changes to the server's exports list.

One anomaly of the Linux NFS server is that it requires a pseudo root
to be defined. Currently the only way a pseudo root can be defined is by 
setting the fsid to zero (i.e. fsid=0). So if we wanted to make v4
the default mounting version and have things just work like v2/v3
all of the existing exports configurations would have to change 
(i.e. a 'fsid=0' would have to be added) to support a v4 mounts,
which, imho, is unacceptable. So this patch series address
this problem.

I think this might also mark the first major piece of work on the Linux NFSv4 code to come from some place other than CITI. I might be wrong, but I think this is a sign of the maturity of code.

What a group effort. I love how teammates come out of the woodwork to help you get the final touches on a project. The code was putback today and you should start seeing it in build 77. I'll leave you with a teaser of things to come:

And here is a roll call:

Core team:
Calum Mackay: Cambridge, UK (team lead)
Tom Haynes: Tulsa, OK, senior engineer
Bill Baker: Austin, TX, srstaff engr/architect/advisor
Rob Thurlow: Ft Collins, CO, senior engr/advisor
Spencer Shepler: Austin, TX, srstaff engr/RTI advocate/advisor
Helen Chao: Menlo Park, CA, QE lead
Lily Li: Beijing, QE engineer
Evan Layton: Broomfield, CO, engineer
Alok Aggarwal: Atlanta, GA, engineer
Rich Brown: Chicago, IL, sr engr/PSARC Intern

And there were more who stepped up and made significant contributions to get the work out of the door.

I've been very busy trying to get Mirror Mounts out the door. Our last task was to fix a find bug:

[tdh@mrx ~]> sudo mount kanigix:/ /mnt
[tdh@mrx ~]> cd /mnt
[tdh@mrx /mnt]> cd zoo
[tdh@mrx zoo]> cd mms
[tdh@mrx mms]> ls -la
total 20
drwxr-xr-x   5 root     sys            5 Jul 19 21:41 .
drwxr-xr-x  11 root     sys           11 Oct  8 13:08 ..
drwxr-xr-x   5 root     sys            5 Jul 19 21:41 node1
drwxr-xr-x   5 root     sys            5 Jul 19 21:41 node2
drwxr-xr-x   5 root     sys            5 Jul 19 21:42 node3
[tdh@mrx mms]> df -F nfs -h
Filesystem             size   used  avail capacity  Mounted on
kanigix:/               20G   8.9G    11G    46%    /mnt
kanigix:/zoo           637G    42K   637G     1%    /mnt/zoo
kanigix:/zoo/mms       637G    31K   637G     1%    /mnt/zoo/mms

Now if we try to find, it should mount a subdirectory for us and traverse down it.

[tdh@mrx mms]> find .
.
./node3
find: cannot open .: Resource temporarily unavailable

It fails because of a security check to make sure the stat(2) information from before an opendir(2) matches a fstat(2) from after the opendir(2). The pre-stat gets the vnode which will be mounted on and the post-fstat gets the root of the new filesystem.

But now we can see that since the mount took place, that we pass the test which just failed.

[tdh@mrx mms]> df -F nfs -h
Filesystem             size   used  avail capacity  Mounted on
kanigix:/               20G   8.9G    11G    46%    /mnt
kanigix:/zoo           637G    42K   637G     1%    /mnt/zoo
kanigix:/zoo/mms       637G    31K   637G     1%    /mnt/zoo/mms
kanigix:/zoo/mms/node3
                       637G    31K   637G     1%    /mnt/zoo/mms/node3
[tdh@mrx mms]> find .
.
./node3
./node3/sub2
find: cannot open .: Resource temporarily unavailable
[tdh@mrx mms]> df -F nfs -h
Filesystem             size   used  avail capacity  Mounted on
kanigix:/               20G   8.9G    11G    46%    /mnt
kanigix:/zoo           637G    42K   637G     1%    /mnt/zoo
kanigix:/zoo/mms       637G    31K   637G     1%    /mnt/zoo/mms
kanigix:/zoo/mms/node3
                       637G    31K   637G     1%    /mnt/zoo/mms/node3
kanigix:/zoo/mms/node3/sub2
                       637G    31K   637G     1%    /mnt/zoo/mms/node3/sub2

But we puke on the next mirror mount.

The problem is actually deep down in nftw(2C) and you can look at the PSARC case at Add S_IFTRIGGER to st_mode [PSARC/2007/563 FastTrack timeout 10/04/2007]. This is pretty interesting reading. Most people told me that my proposal would generate considerable controversy.

The PSARC community decided that there was still a hole in the security check inside nftw(2C). While I can argue against this, it is pretty hard to rename/move an export on a live filesystem, in the end I decided that a counter-proposal made more sense that mine.

You've got to realize, I've been working on this project for the past 6 months. We were going to putback on Friday - time to turn over onto the pNFS project.

We decided as a group that the new proposal was the right thing to do for both the Mirror Mount project and Solaris (and indirectly OpenSolaris).

I believe that the majority of OpenSolaris development occurs within Sun Microsystems Engineering. As much as we would like for it to snowball in the wild, that has not happened. I'm saying this from my biased view, I know some projects have been proposed externally from Sun, e.g., the i18n port of the closed library. I also acknowledge the work that Dennis Clark is leading for the PPC port. There are more and I am not trying to take away from them. I am relating my experience with trying to get projects off the ground on OpenSolaris - see for example OpenSolaris Project: NFS Server in non-Global Zones.

So what does happen is that a new project gets started and there is no external indication of forward progress. People might start asking for code drops and the reality is that because of the huge internal pressure towards quality in Sun Engineering, that is not going to happen until the code has baked a bit. It gets to the point that a prime question on new project proposals is will code be released. Again, there isn't some hidden agenda within Sun to withhold the code - we are just new to this model and we want things to be perfect, not just good enough.

Look back at the discussion that went on for Project Proposal -- Honeycomb Information and dev tools and the lack of a code drop. The OpenSolaris Project: HoneyComb Fixed Content Storage already shows a binary drop and plans for a code drop in the Fall of 2007. Some valid reasons for a group to not drop code right away are that they do not understand the process (they need someone to help them) and they need to clear a legal hurdle to make sure that they are not violating the rights of either an individual or a company. I've seen both occur internally. The good news is that we have internal people ready and willing to help development groups.

What I find really exciting are projects that have a significant external presence. And sometimes that external pressure doesn't contribute directly to the code work. In NFSv4 and NFSv4.1, the external collaboration takes place through the IETF and Connectathon. Both companies and open source developers come together to design and implement future NFS protocol extensions. Interoperability across multiple OS platforms is ensured via the yearly meetings at Connectathon. And with the UMICH CITI developers working on Projects: NFS Version 4 Open Source Reference Implementation, which is mainly distributed to Linux, but forms a reference for both BSD directly and OSX indirectly, and Sun working on OpenSolaris, it is possible for vendors to do compatibility testing all year long.

Take for example NetApp, which provides only a NFS server. They are able to test new NFSv4.1 features against Linux and OpenSolaris clients. Admittedly this isn't new, NetApp was able to use the Solaris 10 beta code to test NFSv4. And the companies in question all sign NDAs and exchange hardware and engineering drops of binaries for testing.

So there is almost no work being driven from OpenSolaris into this open design project. There is a OpenSolaris Project: NFS version 4.1 pNFS, but it is mainly a portal to the Sun NFS team's work. A question that they asked themselves was whether they were going to do binary drops, code drops, or any drop at all. It wasn't a legal issue, the design is done in the open and all of the coding is new development. It wasn't a fear of the unknown, they had already shared binaries in the past. No, rather it was a concern on the impact of providing a drop on the development schedule. Would the overhead of publishing code and/or binaries kill the final deliverable?

Another OpenSolaris reality is that Sun expects to make money. I know that is an evil concept to some open source developers, but we bet the company on being able to deliver quality and sell service along with the source. So making the deadline for the pNFS deliverable is a major concern for the group.

I'm happy that the group decided that they could both deliver on time and make code and binary drops. Lisa just announced for the group the latest drop in FYI: pNFS Code and BFU Archives posted. You can check out the b66 implementation by downloading it. The code is rough in the sense that you wouldn't want to put it in production, but it gives other developers a chance to see what is going on and allows them to test their own implementations. Remember, this code has not been putback into Nevada - it lives in a group workspace. Before OpenSolaris, it would have only been shared under NDA and the expectation that the person installing the code assumed responsibility for any problems.

Project development in OpenSolaris is different than that occurring in other open source communities. There are different hurdles to jump, but there are different expectations as well. Internal developers are proud of the quality that they demand of the code and want to keep that bar high. That in turn makes early code drops hard for them to deliver. It is something they are learning to do. And the pNFS team is leading the way.

We occasionally get a bug assigned to us about how a Solaris client can not mount a Linux export over NFSv4, but can over NFSv3. In some cases we get it as an automounter bug. We will close the bug because it is an intrinsic problem with the NFSv4 implementation on Linux - well, it is actually false advertising.

Consider the following export on a Linux box:

[tdh@adept ~]> more /etc/exports
/home *(rw,fsid=0,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534)
[tdh@adept ~]> exportfs
/home           <world>
[tdh@sandman ~]> showmount -e adept
export list for adept:
/home/mrx    *
/home/tdh    *
/home/spud   *
/home/coach  *
/home/loghyr *

Hmm, that looks wrong - I know I had them exported like that earlier, but clearly the server is only exporting one export.

Anyway, it would seem that the one export is '/home' and that we should be able to go to '/home' and then access the remote filesystem. And we can't:

[tdh@sandman ~]> cd /net/adept/home
[tdh@sandman home]> ls -la
total 7
dr-xr-xr-x   6 root     root           6 May  2 16:31 .
dr-xr-xr-x   2 root     root           2 May  2 16:31 ..
dr-xr-xr-x   1 root     root           1 May  2 16:31 coach
dr-xr-xr-x   1 root     root           1 May  2 16:31 loghyr
dr-xr-xr-x   1 root     root           1 May  2 16:31 mrx
dr-xr-xr-x   1 root     root           1 May  2 16:31 spud
dr-xr-xr-x   1 root     root           1 May  2 16:31 tdh
[tdh@sandman home]> cd tdh
tdh: Permission denied.

What happened?

A snoop trace shows right off the bat that things are not going well:

     sandman -> adept.internal.excfb.com NFS C NULL4
adept.internal.excfb.com -> sandman      NFS R NULL4
     sandman -> adept.internal.excfb.com NFS C 4 (secinfo     ) PUTROOTFH LOOKUP home SECINFO tdh
adept.internal.excfb.com -> sandman      NFS R 4 (secinfo     ) NFS4ERR_NOENT PUTROOTFH NFS4_OK LOOKUP NFS4ERR_NOENT

We should remove the automounter from our testing:

[tdh@sandman home]> sudo mount -o vers=4 adept:/home /mnt
nfs mount: adept:/home: No such file or directory

Again, we see:

     sandman -> adept.internal.excfb.com NFS C 4 (secinfo     ) PUTROOTFH SECINFO home
adept.internal.excfb.com -> sandman      NFS R 4 (secinfo     ) NFS4ERR_OP_ILLEGAL PUTROOTFH NFS4_OK ILLEGAL NFS4ERR_OP_ILLEGAL

Okay, but it works with NFSv3:

[tdh@sandman home]> ls -la /mnt
total 162
drwxr-xr-x  20 root     root        4096 Feb 18 17:49 .
drwxr-xr-x  27 root     root        1024 Apr 23 09:31 ..
drwxr-xr-x   2 1094     100         4096 Feb 18 13:12 nfsv2
drwxr-xr-x   2 1813     100         4096 Feb 18 13:12 nfsv3
drwxr-xr-x   4 3530     100         4096 Feb 18 17:46 nfsv4

Note that we now have uids and earlier we had root ownerships.

What is going on here? Well another simple example shows what the Linux server is doing to us:

[tdh@sandman home]> sudo umount /mnt
[tdh@sandman home]> sudo mount -o vers=4 adept:/ /mnt
[tdh@sandman home]> ls -la /mnt
total 162
drwxr-xr-x   2 nobody   nobody      4096 Feb 18 13:12 nfsv2
drwxr-xr-x   2 nobody   nobody      4096 Feb 18 13:12 nfsv3
drwxr-xr-x   4 nobody   nobody      4096 Feb 18 17:46 nfsv4

And note that I have chopped some of the output for space.

Okay, it is clear to me that adept:/home is actually adept:/ as far as NFSv4 is concerned. The problem is that the Linux NFSv4 implementation of the pseudo-fs namespace is done such that one of the exports is made the root fs. If we look at the export again, we see:

[tdh@adept ~]> more /etc/exports
/home *(rw,fsid=0,insecure,no_subtree_check,sync,anonuid=65534,anongid=65534)

The 'fsid=0' option is assigning '/home' to be '/' as far as NFSv4 is concerned. The problem is that this breaks automounter impelmentations which use the MOUNTPROC_EXPORTALL RPC call to get a list a mountable exports. It also breaks humans reading such output to get a sense of what to mount.

You can find a discussion of how the NFSv4 namespace is built for Linux at Exporting Directories.

I've started posting the slides for Connectathon 2007: Talks 2007. As I get the remaining slides, I'll add them there.

Went to the local Starbucks here at Connectathon 2007. The guy looked up and said "Awake, right?" The guy I was with was floored. I told him, what is so hard - I'm 6'5", currently sporting a handlebar, and always wearing a Green Lantern hoodie.

The event is going along fine. The main problem is that NFSv3 is too solid and the NFSv4 implementations are also getting that way. The NFSv4.1 stuff is really still in the design phase. But developers are getting small victories when they either get code to compile or even run against other vendors. I think that Connectathon 2008 will be more frantic and the victories will be larger.

I just posted the presentation schedule for Connectathon 2007 as Talks 2007. The talks are open to the public (go ahead, jam the room and ask probing questions):

Parkside Hall
180 Park Ave.
San Jose, CA 95113

This is the building which is connected to the The Tech Museum of Innovation in downtown San Jose. In most other cities, I'd tell you to look for the introverted and pale computer geeks. That wouldn't work for that area.