Useful stuff for your blog-reading pleasure.

All | General

Thursday December 06, 2007

X4500 + Solaris ZFS + iSCSI = Perfect Video Editing Storage

Digital video editing is one of those applications that tend to be very data hungry. At SD PAL resolution, we're talking about 720 pixels x 576 lines x 3 bytes of color x 25 full frames per second = about 30 MB/s of data. That's about 224 GB for a 2 hour feature film. Not counting audio (that would only be around 3-4 GB). And we (in Germany) haven't looked at HD or Digital Cinema a lot yet...

During the last couple of weeks I worked with a customer who bought a Sun Fire X4500 server (you know, Thumper). The plan is to run Solaris ZFS on it, then provide big iSCSI volumes to the video editing systems, which tend to be specialized Windows or Mac OS X machines. Wonderful idea: Just use zpool create to combine a number of disks with some RAID level into a pool, then zfs create -V to create a ZVOL. Thanks to zfs shareiscsi=on, sharing the volume over iSCSI is dead easy.

But it didn't work.

First, Windows wouldn't mount the iSCSI volume. After some trying, we discovered that there must be an upper limit of 2TB to the size of iSCSI volumes that Windows can mount (we initially tried something like 5 ot 10TB). So be it: zfs create -V 2047G videopool/videovolume.

Now it mounted ok, we formatted the disk with NTFS (yuck!) and started the editing system's speed test. Then came the real issue: The test reported a write performance of 8-10 MB/s, but the editing system needs something like 30 MB/s sustained to be able to record reliably!

After some trying, we started the systematic approach:

A simple dd from one disk to another yielded >39 MB/s.
dd'ing from one small ZFS pool to another exceeded 120 MB/s (I later learned that cp is a better benchmark because it works asynchronously with large chunks of data vs. dd's synchronous block approach), so that was again more than we needed.
We tried re-attaching our ZVOL through iscsiadm to test the iSCSI stack's performance and ran into a TCP fusion issue. Ok, I've always wanted to play with mdb, so we followed the workaround instructions and we were able to attach our own ZVOL over the loopback interface. Slightly less performance (due to up the stack, down the stack effects, I presume) but still way more than we needed. So, it wasn't the X4500's nor ZFS' fault.

Finally, Danilo pointed me into the right direction: Nagle's algorithm. What usually helps maximize network bandwidth turns out to be a killer for iSCSI performance. For Solaris iSCSI clients, we know this already, but how do we turn off Nagle on Windows?

The answer is deeply buried inside the Microsoft's iSCSI Initiator user guide: The "Addressing Slow Performance with iSCSI Clusters" chapter mentions a similar issue (although they talk about read not write performance) and they do mention RFC 1122's delayed ACK feature, which is related to Nagle's algorithm. The Microsoft document suggests a workaround which involves setting a variable in the registry, so it was worth a try (and my vengeance for having to use mdb before).

And low and behold, the speed test now yielded 90-100 MB/s (Close to a GBE's raw performance)! Yipee that was it! One little registry entry on the client side gave us a 10x improvement in iSCSI performance!

Now, can someone explain to me, why on Windows 2000 you need to set "TcpAckDelTicks=0" while on Windows 2003 the same thing is accomplished by saying "TcpAckFrequency=1" (which is the same thing, only seen from the other side of the division sign)?

So, to all you storage hungry video editors out there: The Sun Fire X4500 with Solaris ZFS and iSCSI is a great solution for reliable, fast, easy to use and inexpensive video storage. You just need to know how to tell your TCP/IP stack to not delay ACKs...

"X4500 + Solaris ZFS + iSCSI = Perfect Video Editing Storage" has been brought to you by Constantin's Blooog.
This entry was created on 2007-12-06 13:31:53.0 PST and is associated with the following tags:
editing file iscsi nagle opensolaris performance registry solaris system tcp/ip thumper tuning video windows x4500 zfs
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.
Comments [10]

Wednesday August 08, 2007

A True Web 2.0 Chip

Yesterday was the big day in which we launched the UltraSPARC T2 chip, code-named Niagara 2.

Few people realize how significant this announcement really is. The UltraSPARC T1 chip already changed the game of providing a powerful web infrastructure: By providing 32 threads in parallel, the UltraSPARC T1 chip and the associated T2000 server can provide more than double the performance of today's regular chips, at half the power cost. Even now, 18 months after its introduction, this chip still remains ahead of the pack both in absolute web performance and in price/performance and in performance/watt.

UltraSPARC T2 is not just a better version of the T1 chip, it provides three significant improvements:

More parallelism: Instead of 32 concurrent threads, UltraSPARC T2 delivers 64 threads running in parallel. Moore's law gives us twice as many transistors to play with every 18 weeks and the best way to leverage that is to turn them into parallelism. UltraSPARC T1 and T2 are all about maximizing the return on Moore's Law. Check out the specs.
More networking: The UltraSPARC T2 features two 10 Gigabit Ethernet ports directly on the chip. Two. Ten GigaBit. On the chip. The NIC is included, there is no bus system between the NIC and the CPU, the CPU is the NIC is the CPU. Total embedded networking. For applications that live in the network, what more can they ask for in a server?
Built-in, free and fast encryption. In a world where the web becomes social, private data becomes more and more common, but also more and more important to secure. Making security a default feature of your web service is now available for free and it does not impact performance.

Of course, there are many more other improvements, such as 8 FP units, more memory etc., but the three points above alone make the UltraSPARC T2 the perfect chip for web 2.0 applications.

This guy needs UltraSPARC T2! For instance, check out this analysis of the Facebook platform by Marc Andreesen. If you don't want to read it all, here's a summary: Web 2.0 means explosive growth in server capacity, for any reasonably successful application. In the case of iLike, they are growing their user base at the rate of 300k a day! This kind of growth can be fatal for your company if you don't have the infrastructure to sustain it. Well, UltraSPARC T2 is just the kind of technology that was designed to do just that: Handle many, many, many concurrent users at once as efficiently and securely as possible.

So, all you Web 2.0 startups out there, get in touch with your nearest Sun rep or Sun SE and ask them about UltraSPARC T2, or better yet, get a free 60-day trial of UltraSPARC T1, do your favourite benchmark, double that number and forget about that crypto-card to see what UltraSPARC T2 can do for you real soon now. Then, sit back, relax and keep those 300k a day users coming!

"A True Web 2.0 Chip" has been brought to you by Constantin's Blooog.
This entry was created on 2007-08-08 09:07:37.0 PST and is associated with the following tags:
adoption facebook grow lokalisten niagara performance scale solaris sparc sun t1 t1000 t2 t2000 ultrasparc web web2.0
You're welcome to use this Permalink , add a comment below or send your feedback to constantin at sun dot com.