Kool Aid Served Daily

I've started a new NFSv4 FAQ over at NFSv4 in Action. I'll add new entries as I scour the various mailing lists.

I gave my presentation on Scaling NFS Services at Connectathon 2006. The main thesis is that you should be aware of presentations you find at Connectathon.org :-> .

Brent Callaghan presented some work that touched on the Authentication Cache that the NFS group put into Solaris back in 1995 - Multiple Flavors per Export and 1996 - Public NFS. Brent presented a concept and included several graphs. Those graphs were not of real data and were pretty idealized.

When I implemented the Access Cache at NetApp, I did so based on those graphs and some converstations with Mike Eisler, who had been involved in the project at Sun. We ended up not seeing the characteristics presented in the papers.

We found a lot of interesting things, but since I'm no longer associated with Network Appliance, Inc., I had to talk in general terms about the assumptions which must have been implicit in Brent's model and why they did not work when scaled to the large client farms which have become very common in this day and age. I also presented several obvious approaches to making Brent's model scale.

As a disclaimer, I had asked Network Appliance's NFS group to make this presentation. It was only after I found some Sun code which made the same assumptions about scalability that I decided to make the presentation. I'm also aware of some issues with the Linux code and I'm sure given enough time, I could find problems with any implementation. I was also asked by a Network Appliance technical director if they could review the slides before I presented at the event. I concurred and I was not asked to remove anything from the slides.

I'm more than willing to discuss in general terms the presentation. By the way, I'm no longer an academic, so the presentation is in no way a slam of Brent or what he presented. Just as I would ask you to be careful in analyzing his graphs, I would also warn you to not take the ones I presented at face value. I based some of the numbers (for the slides on Rise of the Machines) on client farms I am aware of - by no means is that data scientific.

I was walking across the Santa Clara campus Thursday morning and it was still wet - either from rain, dew, or watering. As I walked down one pavement, I noticed a snail in front of me. I immediately thought I should help it get off to the grass. After all, it could easily be stepped on. But sometimes a snail is fragile, and definitly yucky, so I passed on. I then counted about 10 snails, all trying to get to the grass. I realized that snails are just like customers. And I decided to take some pictures to prove my point.

First of all, I went back to get that first snail, if I had helped him, I would have been trapped helping them all. That happened to me at NetApp when I started helping NGS on the customer mailing lists. They came to expect my help and if I didn't respond fast enough, they sought me out. They would bypass policy and procedure to call me up at home, at the pool, after I left NetApp, etc.

That picture didn't come out very well. Perhaps I was blocking the pain.

I also wanted to take a picture of two snails close together. One had to be larger than the other. The point was if two customers come to you at the same time, which do you help? At NetApp, the answer might be both. I got 4 escalations in a 12 hour period one weekend. But do you help the bigger one? Perhaps the smaller one is a reference account and no one told you how important they are to the company. You can't trust the sales guys to tell you how important the customer is to your bottom line. After 2 months of doing escalations, I had to yawn when I was told an account was worth $2 million. Yawn, I was supposed to drop a $20 million account to help a $2 million one?

Anyway, I almost threw out that picture as well. But it proves a point:

Sometimes you can't help a customer because you can't figure out what is going on. It could be that they've supplied you with bad information, your customer support organization has held onto the case for too long, or there is too much going on.

Another way to look at this is in this image, where at just the right magnification, the problem looks clear:

But as you get closer, as you peel the onion (a term I hate thanks to 4 months of startup hell), you find that things aren't really that clear:

Sometimes you might find a customer like this snail. They've been working with you so long to find a problem that they are going in circles: (If you look at the snail trail, you can see it has been circling the wagons.)

Sometimes you really need to avoid being pulled into this mess. No matter what you do, the customer will have no faith that you have helped them solve their problem. Also, you might fix the current problem and expose the next problem for some other department to fix. And then the customer comes back to you. The upside is if you really do fix their problem, your customer support and sales teams will remember that and come straight to you with the next tough one.

I want to close with a fascinating one:

This one is crystal clear and the customer is safe at home. If you help out, you might make the problem worse. (By the way, in reality, this snail was moving fast. I tried to get a movie, but my batteries were too low.)

My final thought was I should step on one and show you what happens when a customer gets really bad support. But I don't do that to snails, even to prove a good point. Perhaps that also says I wouldn't do that to a customer.

By the way, I understand NetApp now has a Level 3 customer support organization in place and is making head roads into reducing the strain on development engineers.

Here are some pictures of me and the Monster out enjoying the snow. It was the 3rd day and the top of the hill had melted. We had started to leave after playing for a bit, but Stacy showed up and made us have more fun.

Here is the Monster clearing a ramp:

Here is what happens when someone a bit heavier tries it:

Here is a full frontal of the Monster, I also like the black and white effect my wife applied later:

Finally, here are some of myself, standing tall and proud:

I saw this statue at DIA:

As normal, someone hadn't respected it and left their trash on the base. I moved it away and took a picture with the last juice in the camera. I had to edit it, it came out too dark.

I don't know how they find images hidden in reflective surfaces in the movies. I should be in the helmet and I can't see didley.

Okay, we got a new player while I was out of town. USC did not change our team name. And finally, the new player was from a team that didn't have a coach.

Guess which team we played today?

USC was on the ball again - they told the refs not to show up, but not us. Well, we had a 20 minute practice.