As part of my role in Fishworks, I push systems to their limits to investigate and solve bottlenecks. Limits can be useful to consider as a possible upper bound of performance - as it shows what the target can do. I thought these results would make for some interesting blog posts if I could explain the setup, describe what the tests were, and include screenshots of these results in action. (update: I've included results from colleagues who have tested in the same manner.)
To summarize the performance limits that I found for a single Sun Storage 7410 head node:
| Workload | 7410 (Barcelona) | 7410 (Istanbul) |
|---|---|---|
| NFS streaming read from DRAM | ~1.90 Gbytes/sec | ~3.06 Gbytes/sec |
| NFS streaming read from disk | 1.04 Gbytes/sec | 2.03 Gbytes/sec |
| NFS streaming write to disk | 563 Mbytes/sec | 932 Mbytes/sec |
| NFS 1b read IOPS from DRAM | 281,000 IOPS | 506,000 IOPS |
| NFS 4k read IOPS from DRAM | 191,000 IOPS | 404,000 IOPS |
| NFS 512b reads from L2ARC | 128 - 260 us | |
| NFS 512b sync writes to SLOG | 137 - 181 us | |
| NFS IB/RDMA streaming read from DRAM | N/A | 2.93 Gbytes/sec |
| NFS IB/RDMA streaming read from disk | N/A | 2.11 Gbytes/sec |
| NFS IB/RDMA streaming write to disk | N/A | 984 Mbytes/sec |
| CIFS streaming read from DRAM | ~1.03 Gbytes/sec | |
| CIFS streaming read from disk | 849 Mbytes/sec | |
| CIFS streaming write to disk | 620 Mbytes/sec | |
| CIFS 1b read IOPS from DRAM | 203,000 IOPS | |
All tests are performed on Ethernet (usually 10 GbE) unless otherwise specified ("IB" == InfiniBand).
Like many products, the 7410 will undergo software and hardware updates over time. This page currently has results for:
- 7410 Barcelona: The initial release, with four quad-core AMD Opteron CPUs (Barcelona), and the 2008.Q4 software.
- 7410 Istanbul: The latest release, with four six-core AMD Opteron CPUs (Istanbul), and the 2009.Q3 software.
I should make clear that these are provided as possible upper bounds - these aren't what to expect for any given workload, unless your workload was similar to what I used for these tests. Click on the results to see details of the workloads used.
These are also the limits that were found with a given farm of clients and JBODs - it's possible the 7410 could go faster with more clients and more JBODs.
Updated 3-Mar-2009: added CIFS results.
Updated 22-Sep-2009: added column for 7410 Istanbul. Results will be added as they are collected.
Updated 12-Nov-2009: added Cindi's InfiniBand results.
Would it be possible to have an IOPS benchmark for small reads/writes? I'm thinking mostly about IO in the size of e-mails (-> mbox format).
Posted by Mika on January 10, 2009 at 01:18 AM PST #
Hey Brendan - love the information. Thanks!
I've noticed it seems you use NFS for most of your tests. Could we expect similar performance over iSCSI? I'm doing some pre-purchase research on the 7410, and in my case, most of my clients would be connecting via iSCSI (over 10GigE).
Thanks!
Posted by Erik on January 10, 2009 at 08:28 AM PST #
could you also please post some performance numbers on the 7210.
Thanks.
Posted by eli101 on January 10, 2009 at 08:46 PM PST #
We made a complete review of the 7110, performance numbers are impressive.
To Mika: we tested small writes too.
It's available at http://www.ilm-informatique.fr/review/fishworks_sun_fr/
(in french), but we are translating it to english.
Regards,
Guillaume
Posted by Guillaume Maillard on January 12, 2009 at 08:55 AM PST #
If need information about the 7110, the english version of our review is available at http://www.ilm-informatique.fr/review/fishworks_sun_com/
Posted by Guillaume Maillard on January 27, 2009 at 03:00 AM PST #
Could you explain the hardware setup behind these numbers. Is this a standard 7140, and can I look this up at a webpage? What is the front end pipe, when is the bak-end pipe (SAS controllers/JBODs), a summary of the bottlenecks (front-end versus back-end..)
Posted by Ceri Davies on March 18, 2009 at 08:17 AM PDT #
Ceri,
I did - click on each number for a page describing the setup and workload. It is a standard 7410.
Posted by Brendan Gregg on March 18, 2009 at 10:47 AM PDT #
What kind of "benchmark" is that? What did you use to benchmark, what were the parameters, how many clients, etc?
Posted by John on April 17, 2009 at 02:50 AM PDT #
@John, if you click on the links, for each result I explain the workload type, I/O size, number of clients, and numbers of threads per client. The actual benchmark software is very simple, and documented here:
http://blogs.sun.com/brendan/entry/performance_testing_the_7000_series3
The previous posts in that series will answer many other questions: did I pay attention to various things, etc:
http://blogs.sun.com/brendan/entry/performance_testing_the_7000_series1
http://blogs.sun.com/brendan/entry/performance_testing_the_7000_series2
Posted by Brendan Gregg on May 31, 2009 at 03:19 PM PDT #
Hi Brendan,
What was the configuration used during your performance limits (7410 with SSD or without SSD)?
Is there a big difference between the 7410 (16GB RAM + 2xJ4400 without SSD) and the 7410 (64GB RAM with Readzilla Flash + 2x J4400 1x 18GB logzilla Flash)?
My customer transfers mainly video files in NFS/CIFS.
Thanks in advance,
JNV
Posted by Joaquim Neves on July 16, 2009 at 01:19 AM PDT #
Brendan,
Will you be running these again with the new Istanbul processors?
thanks for any pointers.
Sean
Posted by Sean Cochrane on September 11, 2009 at 11:24 AM PDT #
@Sean: yes!
Posted by Brendan Gregg on September 23, 2009 at 03:14 PM PDT #