Here's my question for you, Bryan. And I know you're going to have an enthusiastic answer! I've been a Solaris sysadmin for 7 years. In the large group of people who I work with, I'm among the best in tracking down performance problems. I've got a decent enough handle on C, which it seems so many of my coworkers don't have. I absolutely see the benefits in Dtrace. Let me explain why, with a truss example. Recently, my customer had a production problem with a very slow in-house application which accessed a Sybase database. They had no idea why. As a last result, they ask the SA to take a look at it in production and see what they can see. By just running truss, I was able to spot something unusual. And usually it takes me a few passes at truss to spot something interesting. I start off sounding like a cold read on a psychic hotline and steadily grow more and more pinpointed. What I found is that most of the application activity consists of a loop. It polled a file descriptor to see if anything was there. Then, 99.9% of the time, it immediately followed that by a read operation. What did that tell me? Either each time it is doing a read operation, it is not grabbing enough to make a good dent in the buffer, or that the database was providing way more data than expected. (Okay. I used lsof to figure out it was talking to the database. Sun should provide lsof, but I'm hoping you'll tell me I can do the same with Dtrace.) Now, this was news to the application team. Why? Because, as it turns out, they did API calls to a vendor's black box to work a particular business function. The problem wasn't in their code, it was the vendors. So they went to the vendor, armed with what I had found, and got a very quick fix. The point here, though, is that I had to have some good general knowledge of C, system calls, and the general situation in order to help debug the issue. My coworkers don't code in C. Not in the slightest. They're systems administrators. They wouldn't have had a chance to figure out what a constant cycle of polling and reading meant. They might say, "I see it is doing a lot of reading, but I don't see anything wrong." But once I point out to them what was happening, they'd get it. Now I go back to your Register article. After reading it, I understand that mmap, msync, and munmap would be costly, but I wonder if I could have came to that conclusion myself. Perhaps with times in the tracings, I could see that it was taking a long time. But I wouldn't necessarily know that opening with O_DSYNC would have been a better replacement. You could say, well, really this is more of a software development/debugging tool, and not a systems administrator's tool. But then the story would be reversed in that the application team can't be expected to understand much from an OS perspective. I think one of my coworkers hit the nail on the head when he asked, "Do you have to know how to program in C in order to use dtrace?" I think that question is two-fold. First, in creating some good traces in D. Second, in interpreting what dtrace tells you, because applications are generally written in C, and the OS is in C. The second item seems harder than the first item. So I kind of wonder what it takes to train someone up in dtrace. (BTW, I'd love to take a course from SunSolve on Dtrace!) They'd have to be a systems administrator. They'd have to know C. How much C is an interesting point for debate. You might require them to have taken the Solaris Internals class? Maybe their are other requirements. So, while I think Dtrace is totally awesome, I'm just wondering. Is it too awesome for my coworkers to get some real use out of? My follow-up question is that while I have a particular problem that I know I can get some use out of today (an application that performs slow in production, but just fine in test/dev), there is no way that I can get them to migrate to Solaris 10. The vendor won't have it certified. Monitoring, backups, internal tools. All not Solaris 10 ready. (Yes, I know... the binary compatibility guarantee. It just doesn't work that way in the business world.) I'm just a bit unhappy that is going to be some time until I can sink my teeth into some real problems with Dtrace because a new version of the OS limits my opportunities of where I can implement this. And in an environment where we're finding our performance problems in production and not so much in test/dev. Sorry for so many words.

Posted by Josh McCormick on November 24, 2004 at 06:39 PM PST #