Note: the latest Data Validation change required a change to the C code. At this time I have not had the chance to do a compile on AIX, OSX and HP. OSX is pending at this time. If you have a system with a compiler and you have a few minutes, contact me to do a small compile and link for me.
Data Validation was written to make sure that whatever data was written to a disk device would be correct when read back.
For that I developed a methodology that allows Vdbench to remember, in system memory or optionally also in a journal file, what data was written where so that when a data block was referenced again I could compare the contents.
The first 32 bytes on each 512-byte sector of a data block contains a header with unique value and Vdbench after a read operation makes sure that the contents have not changed. The other 480 bytes contain by default the values 8,9,10, etc. up to the total data transfer size divided by four, the default Vdbench data pattern. See ‘Data Validation and Journaling’ in the documentation.
Data Validation has been running successfully now for more than six years, and has been very helpful for all those people that love to fiddle with device drivers, HBAs and other hardware and do all kind of things like pulling cables, drives, and other fun error injections. Vdbench has identified numerous data integrity problems over the years, which then could be solved before the product arrived at the customer. Wonderful!
About a month ago I started seeing some very strange data corruptions. It almost looked as if the data buffers were filled with the correct data, but then once the data buffers were passed to the pwrite64() function it appears that some of the data did not make it from processor memory cache to real memory. I heard terms like ‘cache line’ and indeed we saw corruptions of exactly 64 bytes, which I understand is a ‘cache line’.
And then, in a discussion with Peter Dunlap here in Broomfield (thank you Peter) he asked me “it would be nice to see if we have similar cache line problems outside of the 32-byte header”. And then it dawned on me that, since by default the remaining 480 bytes always have the same data pattern copied to the same buffer over and over again, any data that is lost will not be recognized.
Every time when I explain to people how Data Validation works I tell them “if you write the value ‘A’ 100 times to the same block and you then successfully read an ‘A’ back, you could have lost 99 writes without noticing it, so you always must write unique values”.
Data Validation always made sure that the storage devices lost no data, not realizing that a data buffer already can be (partially) corrupted even before the data makes it to the device. And if part of the corruption includes losing a cache line but with the contents of that cache line always being the same, I made the same mistake that I was trying to solve: “when you store the value ‘A’ 100 times in the same memory address and you then successfully get an ‘A’, you could have lost 99 store operations”.
Vdbench501 has changed the Data Validation data pattern. Instead of storing the values 8,9, etc it now stores random values generated using Linear Feedback Shift Register (LFSR) code that I received from Tim O’Brien (Thank you Tim!).
This now makes sure that not only the 32-byte header, but also the whole 512-byte sector always has different contents avoiding things like the cache line loss mentioned above.
One side effect of always using random data patterns with Data Validation is that when it is done against tape devices the ‘compression=’ value will be ignored.
Vdbench 5.00 under certain circumstances will run with more threads than requested.
Version Vdbench501 will fix this.
The best way to check thread counts is in the kstat.html file. Add columns ‘avg I/O waiting’ and ‘avg I/O active’. When running a max I/O rate these values combined may never be higher than the requested thread count. (They may be slightly lower if the OS cannot keep up queuing new I/O).
For more detail, please read on:
Vdbench uses Java threads. Since Java runs within a single process, all Java threads therefore also run under that same single process.
Many moons ago when the second version of the good old T3 came out, IOPS started to climb to an astonishing 7,200. With the then available processor speed and versions of Solaris, at about 5,000 IOPS a certain process lock became 100% busy, limiting the amount of I/O that could be done under a single process.
I then introduced in Vdbench the concept of running high IOPS test using multi-JVM (Java Virtual Machine). Vdbench spawns up to eight copies of itself, dividing the requested workload over these eight JVMs (Vdbench slaves).
One Storage Definition (SD) would use only one single JVM.
Processor speeds have gone up and Solaris has seen some dramatic changes in its performance, so nowadays I can easily see 50,000 IOPS and even 100,000 IOPS under a single process.
The world keeps changing, storage devices are getting faster and faster, and Solid State Devices (SSDs) now are allowing the amount of IOPS to grow to huge values.
About a year and a half ago I realized that Vdbench therefore needed to change, preferable before the arrival of these new solid-state devices.
One thing that needed to change was the limitation that a Storage Definition (SD) could be used by only one single JVM. Future higher IOPS for a single SD could possibly cause the same ‘single process lock’ problems to come back.
And here comes the problem: All operating systems have the ability to run multiple threads. None of those operating systems however allow you to run only half a thread, or a quarter thread or an eight of a thread. So, when spreading out the requested thread count over multiple JVMs I could not use fractions. Piece of cake. Just round up the number to the next whole number.
And this is where I went wrong: If the user requests only one single thread, but I need to spread out the workload over eight JVMs, the end result is that I am running with
8 times 1/8 (rounded) threads, or 8 * 1, for a total of 8 threads instead of the requested single thread. The same problem of course happens with other thread and/or JVM counts.
Oops, I missed that one, and even though Vdbench 5.00 was in beta for six months as of April last year, and GA since October last year nobody else did until Jim Britton gave me a call last month. (Thanks Jim).
It took me about
7 years, but I finally made some time to create a simple program that takes the
flatfile, picks out the columns and rows that the user wants, and then writes
it to a comma separated (CSV) file.
Note: the
flatfile at this time only contains data for the original RAW functionality.
Vdbench 5.02 will also have a flatfile for the file system functions.
Usage:
./vdbench
parseflat -i flatfile.html -o output.csv [-c col1 col2 ..]
[-a] [-f col1 value1 col2 value2 .. ..]
-i
input flatfile, e.g. output/flatfile.html
-o
output CSV file name (default stdout)
-c
which column to write to CSV. Columns are
written in the order specified.
-f
filters: 'if (colX == valueX) ... ...'
(Alphabetic compare)
-a
include only the 'avg' data. Default: include
only non-avg data.
Example:
./vdbench parse
-i output\flatfile.html -c run interval rate resp -f run rd1 -o out.csv
This will give
you file out.csv which can be directly read into Excel or StarOffice:
run,interval,rate,resp
rd1,1,104.0000,3.3225
rd1,2,119.0000,2.2137
rd1,3,98.0000,3.6711
rd1,4,104.0000,3.0038
rd1,5,99.0000,2.5214