Weblog

All | General | Java | Music
Main | Next month (Jul 2005) »
20050615 Wednesday June 15, 2005

Timing is Everything Timing is Everything

http://opensolaris.org

Timing is Everything

At a testing seminar I attended a couple of years ago I found myself surrounded by a large number of engineers responsible for testing applications whereas I've typically worked on operating system components. As we discussed situations we'd encountered on the job, I realized that there's often quite a gulf between the two types of testing. One of the areas of difference was in dealing with timing-related issues. The application test engineers were typically concerned with the timing of user-initiated operations, whereas I had to consider both user-initiated actions and the timing of various operations happening down below the surface. Most of the other engineers attending the seminar were surprised at the level of detail and precision that OS testing went into.

For this blog I'll use the Solaris Volume Manager (SVM) as an example as that's what I've spent the most time on. SVM allows you to combine multiple physical devices into a single logical device (known as a metadevice in SVM parlance) or split a single physical device into multiple metadevices.

One of the more complicated metadevice types in SVM is RAID5. RAID (Redundant Array of Inexpensive Disks) level 5 stripes data across multiple disks and provides data redundancy by adding one extra drive and generating and storing parity based on the data. The parity is spread across all disks, not concentrated on one. The following crude ASCII diagram gives an idea of how data and parity are laid out on a RAID5 metadevice:

The data in a RAID5 is split up into chunks based on the 'interlace size' for the device. When a write occurs to the device, the SVM code breaks the data up into chunks of this size and calculates parity. The chunks of data and parity are written to the components (disk slices) that make up the RAID5 device. The advantages of a RAID5 are that you get read performance gains from the striping of the data across multiple disks, and that you can lose any one component and still be able to access all data by looking at the remaining data and the parity info. The disadvantage is that calculating and writing the parity information slows down write operations. Also, the RAID5 must be 'zeroed out' (write zeroes at all addresses) before it can be used to insure that the data and parity match.

The following example shows data being written to a RAID5 device. Lower-case characters, 'd' and 'p', are used to show the existing data and parity information on the RAID5. Upper-case letters, 'D' and 'P', show the new data and parity information being written to the RAID5.

When no writes are being made to the RAID5, the data and parity on the metadevice are static so the timing of a failure doesn't really matter much. Things start to get interesting when the device is being written to. Consider what would happen if the system encountered a problem (power failure, panic, etc.) at the middle stage where the write was only partially complete. The data and parity no longer match, so if a component failed the data that would be reconstructed by using the parity would be corrupt. If it were possible to do an 'atomic' operation in which we could be guaranteed that either all three components had been updated or none of them had then we wouldn't have a problem. Unfortunately we have no method of doing so.

Since we can't guarantee simultaneous updates to all components, we need some other mechanism to keep the data and parity consistant. In the case of an SVM RAID5 metadevice, this is handled by having a prewrite area on each RAID component:

The prewrite area takes up a small amount of space at the start of each RAID5 component. When writing to a RAID5, data is first written to the prewrite area. Once the prewrite area for all components have been updated, the data can start to be written into the desired blocks within each component:

Note that the use of a prewrite area adds still more overhead to RAID5 writes, as all of the data gets written twice (first to the prewite area, then to the final destination).

Now let's look at what happens if we encounter a failure at any given point in the above example:

This explanation is a bit oversimplified (e.g. we must obviously store the target address for the write as well as the data) but it should give you an idea of how a RAID5 metadevice insures that the data and parity match.

To validate the functionality of RAID5 we need to be able simulate failures at or in between each of these points. This requires 'white box/glass box' testing where the test engineers have visibility to the internal workings of the product code in order to test it adequately. Understanding the order in which these steps occur, and having a syncronization method such that failures can be simulated at critical points, is necessary to make sure we've covered all the bases. Black box testing, where we're aware of the interfaces and the general operation but not the internal details, could easily miss a bug that will only manifest itself if a system or disk failure happens at one specific point or transition that's not visible to the user. If you look at the SVM code you'll find references to 'notify' -- these are 'hooks' built into the product code to allow an outside entity (like automated tests) to be informed of internal SVM events and state changes so that they can synchronize their actions with the SVM activity.

This isn't to say that black box testing isn't valuable. White box testing tends to be very 'development centric', as you're looking at how the code works and probing areas where you suspect there could be issues. However, while you may do a good job of proving that the specific implementation works or doesn't, it doesn't necessarily tell you if the code meets the larger goal of meeting a customer need. By contrast black box testing approaches the product more from a user/customer point of view, helping you determine if it fulfills it's intended purpose. Testing from both of these angles is better than using only one or the other.

The requirement to keep the system and data in a coherent and usable state after failures is one of the things that differentiates operating system validation and application testing. In most cases applications don't have stringient demands placed on their ability to recover after an error; as long as the application will start back up then the user will be satisfied; simply keeping a backup copy of any document modified by application is usually sufficient to avoid any significant impact on the user. For operating systems, however, it's critical to be able to recover without having corrupted any data and without requiring major intervention on the part of the user who probably doesn't have a lot of OS expertise. Identifying windows of vulnerability and being able to probe them with some degree of precision is a difficult but necessary part of operating system validation.

OpenSolaris

Solaris ( Jun 15 2005, 06:09:13 AM PDT ) Permalink Comments [0]

20050614 Tuesday June 14, 2005

RAID 0+1 vs. RAID 1+0 and SVM RAID 0+1 vs. RAID 1+0 and SVM

http://opensolaris.org

RAID 0+1 vs. RAID 1+0 and SVM

Six years ago when I first started working on Solaris Volume Manager's earlier incarnation (known as SDS) I was confused about whether it implemented RAID 0+1 or RAID 1+0. The answer ended up being more complicated than simply one or the other. The same implementation has carried forward into the current version of SVM. Since this question still comes up with some regularity I thought it was worth spending some time describing how this particular part of SVM works.

Background

RAID stands for 'Redundant Array of Inexpensive Disks', and the different numbers correspond to differing ways of placing data on the disks. There are two basic RAID levels that pertain to this subject in general plus an additional logical device type that's involved when you're dealing with SVM:

Since RAID0 improves performance, and RAID1 provides redundancy, someone came up with the idea to combine them. Fast and reliable. Two great tastes that taste great together!

When combining these two types of 'logical' devices there's a choice to be made -- do you mirror two stripes, or do you stripe across multiple mirrors? There are pros and cons to each approach:

SVM specifics

So, does SVM do RAID 0+1 or RAID 1+0? The answer is, "Yes." So it gives you a choice between the two? The answer is "No."

Obviously further explanation is necessary...

In SVM, mirror devices cannot be created from "bare" disks. You are required to create the mirror on top of another type of SVM metadevice, known as a concat/stripe*. SVM combines concatenations and stripes into a single metadevice type, in which one or more stripes are concatenated together. When used to build a mirror these concat/stripe logical devices are known as submirrors. If you want to expand the size of a mirror device you can do so by concatenating additional stripe(s) onto the concat/stripe devices that are serving as submirrors.

So, in SVM, you are always required to set up a stripe (concat/stripe) in order to create a mirror. On the surface this makes it appear that SVM does RAID 0+1. However, once you understand a bit about the SVM mirror code, you'll find RAID 1+0 lurking under the covers.

SVM mirrors are logically divided up into regions. The state of each mirror region is recorded in state database replicas* stored on disk. By individually recording the state of each region in the mirror, SVM can be smart about how it performs a resync. Following a disk failure or an unusual event (e.g. a power failure occurs after the first side of a mirror has been written to but before the matching write to the second side can be accomplished), SVM can determine which regions are out-of-sync and only synchronize them, not the entire mirror. This is known as an optimized resync.

The optimized resync mechanisms allow SVM to gain the redundancy benefits of RAID 1+0 while keeping the administrative benefits of RAID 0+1. If one of the drives in a concat/stripe device fails, only those mirror regions that correspond to data stored on the failed drive will lose redundancy. The SVM mirror code understands the layout of the concat/stripe submirrors and can therefore determine which resync regions reside on which underlying devices. For all regions of the mirror not affected by the failure, SVM will continue to provide redundancy, so a second disk failure won't necessarily prove fatal.

So, in a nutshell, SVM provides a RAID 0+1 style administrative interface but effectively implements RAID 1+0 functionality. Administrators get the best of each type, the relatively simple administration of RAID 0+1 plus the greater resilience of RAID 1+0 in the case of multiple device failures.

* concat/stripe logical devices (metadevices)

The following example shows a concat/stripe metadevice that's serving as a submirror to a mirror metadevice. Note that the metadevice is a concatenation of three separate stripes:

** State database replicas

SVM stores configuration and state information in a 'state database' in memory. Copies of this state database are stored on disk, where they are referred to as state database replicas. The primary purpose of the state database replicas is to provide non-volatile copies of the state database so that the SVM configuration is persistant across reboots. A secondary purpose of the replicas is to provide a 'scratch pad' to keep track of mirror region states.

OpenSolaris

Solaris ( Jun 14 2005, 08:10:53 AM PDT ) Permalink Comments [8]

20050608 Wednesday June 08, 2005

The beatings will continue until the blogging improves

Well, it's come to this. Blogging is all the rage at Sun right now and there's a lot of pressure to blog. I've held out up until now, but have decided to give in to the peer and management pressure. The fact that I'm choosing to start blogging right before APR (annual performance review) time is of course purely coincidental. :-) I do get to count this as an accomplishment, right?

Presumably an introduction is in order, on the off chance that somebody actually ends up reading this...

I've been with Sun as a test/QA engineer for nearly a decade now. I started out in the Computer Systems division qualifying SCSI HBAs. With the creation of Network Storage my group got sucked into the new division, but I soon returned to Computer Systems as a lead in the software QA group for desktops. After a while I moved to the Solaris division to work on Solaris Volume Manager and I've been there for the past six years.

I expect most of my ramblings will be with regards to test engineering at Sun, how we try to help ship quality products in the extremely short time frames that this industry forces upon you, and general observations on my experiences at Sun compared to other places I've worked. Hopefully I'll spare you tales of my family life, my poor taste in movies and music, etc. However, as long as the pressure to blog continues, I make no guarantees as to the type of content I'll post. Gotta be ready with everything and anything in case they impose some sort of word count quota.

Getting involved with blogging at Sun is kind of strange. You hear tons about it, there's a lot of interest in having employees do it, and there's both overt and subtle suggestions to get a blog going. However, when you do sign up for a blogging account, you're presented with a bunch of warnings. Basically, you're given all sorts of reasons why screwing up could cost Sun money and possibly cost you your job and you're asked, "Are you sure you want to do this?" The corporate culture seems to be heading towards "Blog or die!" while the legalese is suggesting "Blog and die!"

Seems like a good time to go read Catch-22... ( Jun 08 2005, 06:44:55 AM PDT ) Permalink Comments [1]

Calendar

June 2005 »
SunMonTueWedThuFriSat
   
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
  
       
Today

RSS Feeds

XML
All
/General
/Java
/Music

Search

Links

Navigation

Referers

Today's Page Hits: 11