Jim Walker: Sun Software Engineer

This email (ZFS Committed to the FreeBSD Base) says it all.

Alex Brooks talks about ZFS on Mac OS X Leopard.

Mike Byrne did a nice write-up on ztest at: http://www.opensolaris.org/os/community/zfs/ztest.

Bill Moore also discusses ztest in his blog.

This is a "whitebox" test suite that is maintained by the ZFS developers, and is in the ON source code base on opensolaris.org.

In addition, we plan on releasing the ZFS test suite that is developed and maintained by the ZFS test team in January. Stay tuned...

So, I was going over the types of Sun storage arrays we have tested with ZFS and came up with this list as a starting point:

D2, MultiPack, T3, T3+, A1000, A5000, A5100, A5200, D1000, SE3120 JBOD, SE3310, SE3320, SE3510, SE3510 JBOD, SE3511, SE6120, SE6130, SE6140, SE6910, SE6920, SE9910, SE9960, SE9970, SE9980, SE9990

Of course the X4500 (Thumper) should be listed here as well. Is it a storage device, or is it a server? It's both!! It used to take two full sized racks to get 16TBs. Now with the X4500, I get 24TBs in one U2 slot, and 48TBs will be available before you know it.

This list is far from complete. It doesn't include all the NAS and other vendor storage we have tested with ZFS.

If you have other storage products you have used with ZFS and would like me to add, let me know. I know ZFS will work with just about any storage you give it, but it's still fun to make a list, and watch it grow.

MTBUFS (multi-terabyte UFS) was first released in Solaris 9 update 4. Lori Alt (a member of the ZFS project team) developed MTBUFS after some very innovative thinking and careful review of the UFS data structures and employing EFI disk labels which allowed greater than a terabyte of storage per disk. This allowed her to extend UFS to 16 terabytes. Genius!!

But, how do you test this capability when you only have a few gigabytes of storage? Luckily the first prototype of ZFS was available which allowed us to create sparse volumes up to 16 exabytes in size. We still did real 16 terabyte MTBUFS testing using SVM of course, but ZFS greatly expanded the number of test cases we could perform. Here's how you create a 16 terabyte MTBUFS file system on a much smaller disk:

1) Create a sparse (256 terabyte) zvolume using a 33 gigabyte disk 

$ zpool create testpool c0t6d0 
$ zfs list 
NAME                   USED  AVAIL  REFER  MOUNTPOINT 
testpool              31.5K  33.5G     8K  /testpool 
$ zfs create -s -V 256T testpool/testvol 
$ zfs list 
NAME                   USED  AVAIL  REFER  MOUNTPOINT 
testpool              43.0K  33.5G     8K  /testpool
testpool/testvol      7.00K  33.5G  7.00K  - 
$ zfs list -o volsize testpool/testvol 
VOLSIZE 
   256T

2) Create a 16 terabyte MTBUFS file system 

$ newfs /dev/zvol/dsk/testpool/testvol 
newfs: construct a new file system /dev/zvol/rdsk/testpool/testvol: (y/n)? y
Warning: 4112 sector(s) in last cylinder unallocated 
/dev/zvol/rdsk/testpool/testvol: 34359738352 sectors in 5592406 cylinders
of 48 tracks, 128 sectors
        16777216.0MB in 39108 cyl groups (143 c/g, 429.00MB/g, 448 i/g) 
super-block backups (for fsck -F ufs -o b=#) at:
 32, 878752, 1757472, 2636192, 3514912, 4393632, 5272352, 6151072, 7029792
Initializing cylinder groups: 
...............................................................................
...............................................................................
...............................................................................
...............................................................................
......................................................................
super-block backups for last 10 cylinder groups at:
 34351197472, 34352076192, 34352954912, 34353833632, 34354712352, 
 34355591072, 34356461600, 34357340320, 34358219040, 34359097760 

Note the space used for the MTBUFS metadata. 

$ zfs list 
NAME                   USED  AVAIL  REFER  MOUNTPOINT
testpool              2.47G  31.0G     8K  /testpool 
testpool/testvol      2.47G  31.0G  2.47G  - 

Test out the MTBUFS file system 

$ mkdir /mtbufs 
$ mount /dev/zvol/dsk/testpool/testvol /mtbufs 
$ df -lk | grep mtbufs 
/dev/zvol/dsk/testpool/testvol 17177052768 65560 17005216688 1% /mtbufs
$ cd /mtbufs 
$ mkfile 2G testfile 
$ zfs list 
NAME                   USED  AVAIL  REFER  MOUNTPOINT 
testpool              4.47G  29.0G     8K  /testpool 
testpool/testvol      4.47G  29.0G  4.47G  - 
$ ls -l 
total 4196402 
drwxr-xr-x   3 root     root         512 Nov 21 14:44 ./ 
drwxr-xr-x  45 root     root        1024 Nov 21 14:30 ../ 
drwx------   2 root     root        8192 Nov 21 14:15 lost+found/
-rw------T   1 root     root  2147483648 Nov 21 14:49 testfile

Now as long as the file system data doesn't exceed the available zpool space, we can do as much testing as we like. Enjoy!

BTW. Note the MTBUFS magic number in usr/src/uts/common/sys/fs/ufs_fs.h

#define FS_MAGIC        0x011954
#define MTB_UFS_MAGIC   0xdecade

Lori decided not to use her birthday.

Advanced technology like ZFS is a blog rich environment, so get ready for more blogs from me...

One thing that got released in the same build as ZFS is EFI label support for IDE (ATA,SATA) hard disks. The GEOM team did a nice job on this. This support allows us to do the following command, where c2d0 is an IDE disk:

$ zpool create testpool c2d0

Before this support was added you could only have slices of a SMI labeled disk in a zpool. Now you can add the whole IDE disk. EFI labels are important for many reasons, mainly that they provide support for one terabyte disks and larger. I am very familiar with this, since I was the test lead for the MTB (Multi-terabyte) UFS project which extended UFS to support 16 terabyte file systems. Early versions of ZFS were used for testing MTB UFS since with ZFS we could create sparse volumes of enormous size on very small disks. But, more on this another day... For now, all you home users with extra IDE drives and anyone else enjoy. You can run ZFS the way it is intended.

Testing ZFS reminds me of when I was working for IBM at NASA on the Space Shuttle Flight Software Project. There we had 10,000 people around the world testing code being produced by 100 developers. The code had to get the crew safely into orbit, keep them in orbit and get them safely back to earth, plus perform a myriad of supporting system management tasks, like monitoring oxygen levels or deploying a satellite. On top of this, the interface had to be easy for the astronauts to use. Most of the time they needed just a high level "GO" button that hid all the complexity and performed the big jobs for them, like getting them into orbit. But, when adjustments were needed, they needed clear status information, and straight forward ways to change the system to meet their specific needs. Our job was to verify it all worked, and make sure the astronauts liked the look and feel. We were dedicated to this effort, and because the technology was so interesting and advanced, it was never hard to find people willing to help us.

20 years later, enter ZFS. Instead of protecting and managing systems for a crew of astronauts, ZFS protects and manages data for everyone. The scale of the development and test groups is a little different, but the task of verifying that it all works is just as critical and the team is just as dedicated. I joined the ZFS project team and became the test lead a few months after the first prototype became available. Immediately after this a core test team was formed, and we began developing tests and updating others to work with ZFS. The test team, which is part of the OPG Engineering Operations Quality Engineering organization is tasked with organizing all aspects of ZFS testing including developing automated functional and stress tests and performing daily test runs. In addition, other areas include regression, conformance, interoperability, memory leak, documentation and internationalization testing. The core ZFS test team alone has put over 20 people years of time into verifying ZFS so far. And, because our team is spread across the world from Dublin to Beijing and points in between, tests are running all the time.

But that's only part of the story. Like the Space Shuttle Flight Software, it was easy to find people interested in testing and supporting testing ZFS everywhere. Which is good, because file system and volume management software is complex and important to everyone, and needs verification from every angle. Some of the groups within Sun involved in this effort include, Central Storage Systems Test, Solaris Pre-Integration Test, Solaris Performance Test, Performance and Availability Engineering, Solaris Standards Conformance Test, Sun Cluster Test, the Enterprise Technology Center, OPG Engineering Operations Labs, the Data Management Group, Sun-On-Sun and the Diagnostics Engineering Group. These groups combined contributed another 30 people years of effort into improving the quality of ZFS. It's amazing! ZFS is not even released officially yet and we have over 50 years of testing behind us, and this doesn't even include all the contributions made by customers in the beta test program. Thanks to everyone who has made ZFS what it is today.

There's still work to do (we engineers always want more), but ZFS is definitely ready for initial blast off!