BestPerf: Performance & Best Practices

TPC-C Sun SPARC Enterprise T5440 with Oracle RAC World Record Database Result

Sun and Oracle demonstrate the World's fastest database performance. Sun Microsystems using 12 Sun SPARC Enterprise T5440 servers, 60 Sun Storage F5100 Flash arrays and Oracle 11g Enterprise Edition with Real Application Clusters and Partitioning delivered a world-record TPC-C benchmark result.

• The 12-node Sun SPARC Enterprise T5440 server cluster result delivered a world record TPC-C benchmark result of 7,646,486.7 tpmC and $2.36 $/tpmC (USD) using Oracle 11g R1 on a configuration available 12/14/09.

• The 12-node Sun SPARC Enterprise T5440 server cluster beats the performance of the IBM Power 595 (5GHz) with IBM DB2 9.5 database by 26% and has 16% better price/performance on the TPC-C benchmark.
  

• The complete Oracle/Sun solution used 10.7x better computational density than the IBM configuration (computational density = performance/rack).

• The complete Oracle/Sun solution used 8 times fewer racks than the IBM configuration.

• The complete Oracle/Sun solution has 5.9x better power/performance than the IBM configuration.

• The 12-node Sun SPARC Enterprise T5440 server cluster beats the performance of the HP Superdome (1.6GHz Itanium2) by 87% and has 19% better price/performance on the TPC-C benchmark.

• The Oracle/Sun solution utilized Sun FlashFire technology to deliver this result. The Sun Storage F5100 flash array was used for database storage.

• Oracle 11g Enterprise Edition with Real Application Clusters and Partitioning scales and effectively uses all of the nodes in this configuration to produce the world record performance.

• This result showed Sun and Oracle's integrated hardware and software stacks provide industry-leading performance.

More information on this benchmark will be posted in the next several days.

Performance Landscape

TPC-C results (sorted by tpmC, bigger is better)

System	tpmC	Price/tpmC	Avail	Database	Cluster	Racks	w/KtpmC
12 x Sun SPARC Enterprise T5440	7,646,487	2.36 USD	12/14/09	Oracle 11g RAC	Y	9	9.6
IBM Power 595	6,085,166	2.81 USD	12/10/08	IBM DB2 9.5	N	76	56.4
HP Integrity Superdome	4,092,799	2.93 USD	08/06/07	Oracle 10g R2	N	46	to be added

Avail - Availability date
w/KtmpC - Watts per 1000 tpmC
Racks - clients, servers, storage, infrastructure

Sun and IBM TPC-C Response times

System	tpmC	Response Time New Order 90th%	Response Time New Order Average
12 x Sun SPARC Enterprise T5440	7,646,487	0.170	0.168
IBM Power 595	6,085,166	1.69	1.22
Response Time Ratio - Sun Better		9.9x	7.3x

Sun uses 7x comparison to highlight the differences in response times between Sun's solution and IBM.  Although notice that Sun is 10x faster on New Order transactions that finish in the 90% percentile.

It is also interesting to note that none of Sun's response times, avg or 90th percentile, for any transaction is over 0.25 seconds. While IBM does not have even one interactive transaction, not even the menu, below 0.50 seconds. Graphs of Sun's and IBM's response times for New-Order can be found in the full disclosure reports on TPC's website TPC-C Official Result Page.

Results and Configuration Summary

Hardware Configuration:

9 racks used to hold

Servers:

12 x Sun SPARC Enterprise T5440
4 x 1.6 GHz UltraSPARC T2 Plus
512 GB memory
10 GbE network for cluster

Storage:

60 x Sun Storage F5100 Flash Array
61 x Sun Fire X4275, Comstar SAS target emulation
24 x Sun StorageTek 6140 (16 x 300 GB SAS 15K RPM)
6 x Sun Storage J4400
3 x 80-port Brocade FC switches

Clients:

24 x Sun Fire X4170, each with
2 x 2.53 GHz X5540
48 GB memory

Software Configuration:

Solaris 10 10/09
OpenSolaris 6/09 (COMSTAR) for Sun Fire X4275
Oracle 11g Enterprise Edition with Real Application Clusters and Partitioning
Tuxedo CFS-R Tier 1
Sun Web Server 7.0 Update 5

Benchmark Description

TPC-C is an OLTP system benchmark. It simulates a complete environment where a population of terminal operators executes transactions against a database. The benchmark is centered around the principal activities (transactions) of an order-entry environment. These transactions include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses.

See Also

• TPC-C Official Result Page
  

• www.tpc.org Website

Disclosure Statement

TPC Benchmark C, tpmC, and TPC-C are trademarks of the Transaction Performance Processing Council (TPC). 12-node Sun SPARC Enterprise T5440 Cluster (1.6GHz UltraSPARC T2 Plus, 4 processor) with Oracle 11g Enterprise Edition with Real Application Clusters and Partitioning, 7,646,486.7 tpmC, $2.36/tpmC. Available 12/14/09. IBM Power 595 (5GHz Power6, 32 chips, 64 cores, 128 threads) with IBM DB2 9.5, 6,085,166 tpmC, $2.81/tpmC, available 12/10/08. HP Integrity Superdome(1.6GHz Itanium2, 64 processors, 128 cores, 256 threads) with Oracle 10g Enterprise Edition, 4,092,799 tpmC, $2.93/tpmC. Available 8/06/07. Source: www.tpc.org, results as of 11/5/09.

An engineer in our group wrote this blog posting:
"Exadata V2... Oracle grid consolidation in a box"

Link:
http://blogs.sun.com/glennf/entry/exadata_v2_oracle_grid_consolidation

Overview and Significance of Results

Oracle and Sun's Flash Cache technology combines New features in Oracle with the Sun Storage F5100 to improve database performance. In Oracle databases, the System Global Area (SGA) is a group of shared memory areas that are dedicated to an Oracle “instance” (Oracle processes in execution sharing a database) . All Oracle processes use the SGA to hold information. The SGA is used to store incoming data (data and index buffers) and internal control information that is needed by the database. The size of the SGA is limited by the size of the available physical memory.

This benchmark tested and measured the performance of a new Oracle Database 11g (Release2) feature, which allows to extend the SGA size and caching beyond physical memory, to a large flash memory storage device as the Sun Storage F5100 flash array.

One particular benchmark test demonstrated a dramatic performance improvement (almost 5x) using the Oracle Extended SGA feature on flash storage by reaching SGA sizes in the hundreds of GB range, at a more reasonable cost than equivalently sized RAM and with much faster access times than disk I/O.

The workload consisted in a high volume of SQL select transactions accessing a very large table in a typical business oriented OLTP database. To obtain a baseline, throughput and response times were measured applying the workload against a traditional storage configuration and constrained by disk I/O demand (DB working set of about 3x the size of the data cache in the SGA). The workload was then executed with an added Sun Storage F5100 Flash Array configured to contain an Extended SGA of incremental size.

The tests have shown scaling throughput along with increasing Flash Cache size.

Table of Results

F5100 Extended SGA Size (GB)	Query Txns / Min	Avg Response Time (Secs)	Speedup Ratio
No	76338	0.118	N/A
25	169396	0.053	2.2
50	224318	0.037	2.9
75	300568	0.031	3.9
100	357086	0.025	4.6

Configuration Summary

Server Configuration:

Sun SPARC Enterprise M5000 Server
8 x SPARC64 VII 2.4GHz Quad Core
96 GB memory

Storage Configuration:

8 x Sun Storage J4200 Arrays, 12x 146 GB 15K RPM disks each (96 disks total)
1 x Sun Storage F5100 Flash Array

Software Configuration:

Oracle 11gR2
Solaris 10

Benchmark Description

The workload consisted in a high volume of SQL select transactions accessing a very large table in a typical business oriented OLTP database.

The database consisted of various tables: Products, Customers, Orders, Warehouse Inventory (Stock) data, etc. and the Stock table alone was 3x the size of the db cache size.

To obtain a baseline, throughput and response times were measured applying the workload against a traditional storage configuration and constrained by disk I/O demand. The workload was then executed with an added Sun Storage F5100 Flash Array configured to contain an Extended SGA of incremental size.

During all tests, the in memory SGA data cache was limited to 25 GB .

The Extended SGA was allocated on a “raw' Solaris Volume created with the Solaris Volume Manager (SVM) on a set of devices (Flash Modules) residing on the Sun Storage F5100 flash array.

Key Points and Best Practices

In order to verify the performance improvement brought by extended SGA, the feature had to be tested with a large enough database size and with a workload requiring significant disk I/O activity to access the data. For that purpose, the size of the database needed to be a multiple of the physical memory size, avoiding the case in which the accessed data could be entirely or almost entirely cached in physical memory.

The above represents a typical “use case” in which the Flash Cache Extension is able to show remarkable performance advantages.

If the DB dataset is already entirely cached, or the DB I/O demand is not significant or the application is already saturating the CPU for non database related processing, or large data caching is not productive (DSS type Queries), the Extended SGA may not improve performance.

It is also relevant to know that additional memory structures needed to manage the Extended SGA are allocated in the “in memory” SGA, therefore reducing its data caching capacity.

Increasing the Extended Cache beyond a specific threshold, dependent on various factors, may reduce the benefit of widening the Flash SGA and actually reduce the overall throughput.

This new cache is somewhat similar architecturally to the L2ARC on ZFS. Once written, flash cache buffers are read-only, and updates are only done into main memory SGA buffers. This feature is expected to primarily benefit read-only and read-mostly workloads.

A typical sizing of database flash cache is 2x to 10x the size of SGA memory buffers. Note that header information is stored in the SGA for each flash cache buffer (100 bytes per buffer in exclusive mode, 200 bytes per buffer in RAC mode), so the number of available SGA buffers is reduced as the flash cache size increases, and the SGA size should be increased accordingly.

Two new init.ora parameters have been introduced, illustrated below:

db_flash_cache_file = /lfdata/lffile_raw
db_flash_cache_size = 100G

The db_flash_cache_file parameter takes a single file name, which can be a file system file, a raw device, or an ASM volume. The db_flash_cache_size parameter specifies the size of the flash cache. Note that for raw devices, the partition being used should start at cylinder 1 rather than cylinder 0 (to avoid the disk's volume label).

See Also

• Lowering your IT costs with Oracle Database 11g Release 2

• Oracle Database 11g Release 2 new Features Guide

Disclosure Statement

Results as of October 10, 2009 from Sun Microsystems.

The Sun SPARC Enterprise M4000 server combined with Sun FlashFire technology, the Sun Storage F5100 flash array, has produced World Record Performance on PeopleSoft Payroll (North America) 9.0 benchmark.

• A Sun SPARC Enterprise M4000 server with four new 2.53GHz SPARC64 VII processors and a Sun Storage F5100 flash array is 33% faster than the HP rx6600 (4 x 1.6GHz Itanium2 processors) on the PeopleSoft Payroll (NA) 9.0 benchmark. The Sun solution used the Oracle 11g database running on Solaris 10.

• The Sun SPARC Enterprise M4000 server with four 2.53GHz SPARC64 VII processors and the Sun Storage F5100 flash array is 35% faster than the 2027 MIPs IBM Z990 (6 Z990 Gen1 processors) on the PeopleSoft Payroll (NA) 9.0 benchmark with Oracle 11g database running on Solaris 10. The IBM result used IBM DB2 for Z/OS 8.1 for the database.

• The Sun SPARC Enterprise M4000 server with four 2.53GHz SPARC64 VII processors and a Sun Storage F5100 flash array processed 250K employee payroll checks using PeopleSoft Payroll (NA) 9.0 and Oracle 11g running on Solaris 10. Four different execution strategies were run with an average improvement of 25% compared to HP's results run on the rx6600. Sun achieved these results with 8 concurrent jobs using only 25% CPU utilization while HP required 16 concurrent jobs with a 88% CPU utilization.
  

• The Sun SPARC Enterprise M4000 server combined with Sun FlashFire technology processed 8 Sequential Jobs and single run control with a total time of 527.85 mins, an improvement of 20% compared to HPs time of 633.09 mins.

• The Sun SPARC Enterprise M4000 server combined with Sun FlashFire technology demonstrated a speedup of 81% going from 1 to 8 streams on the PeopleSoft Payroll (NA) 9.0 benchmark using the Oracle 11g database.

• The Sun FlashFire technology dramatically improves IO performance for the PeopleSoft Payroll benchmark with significant performance boost over best optimized FC disks (60+).

• The Sun Storage F5100 Flash Array is a high performance high density solid state flash array which provides a read latency of only 0.5 msec which is about 10 times faster than the normal disk latencies 5 msec measured on this benchmark.

• Sun estimates that the MIPS rating for a Sun SPARC Enterprise M4000 server is over 2742 MIPS.

Performance Landscape

250K Employees

System	Processor	OS/Database	Time in Minutes			Version
			Run 1	Run 2	Run 3
Sun M4000	4x 2.53GHz SPARC64 VII	Solaris/Oracle 11g	79.35	288.47	527.85	9.0
HP rx6600	4x 1.6GHz Itanium2	HP-UX/Oracle 11g	81.17	350.16	633.25	9.0
IBM Z990	6x Gen1 2027 MIPS	Z/OS /DB2	107.34	328.66	544.80	9.0
HP rx6600	4x 1.6GHz Itanium2	HP-UX/Oracle 11g	105.70	369.59	633.09	9.0

Note: IBM benchmark documents show that 6 Gen1 procs is 2027 mips. 13 Gen1 processors were in this config but only 6 were available for testing.

500K Employees

System	Processor	OS/Database	Time in Minutes			Version
			Run 1	Run 2	Run 3
HP rx7640	8x 1.6GHz Itanium2	HP-UX/Oracle 11g	133.63	712.72	1665.01	9.0

Results and Configuration Summary

Hardware Configuration:

1 x Sun SPARC Enterprise M4000 (4 x 2.53 GHz/32GB)
1 x Sun Storage F5100 Flash Array (40 x 24GB FMODs)
1 x Sun Storage J4200 (12 x 450GB SAS 15K RPM)

Software Configuration:

Solaris 10 5/09
Oracle PeopleSoft HCM 9.0
Oracle PeopleSoft Enterprise (PeopleTools) 8.49
Micro Focus Server Express 4.0 SP4
Oracle RDBMS 11.1.0.7 64-bit
HP's Mercury Interactive QuickTest Professional 9.0

Benchmark Description

The PeopleSoft 9.0 Payroll (North America) benchmark is a performance benchmark established by PeopleSoft to demonstrate system performance for a range of processing volumes in a specific configuration. This information may be used to determine the software, hardware, and network configurations necessary to support processing volumes. This workload represents large batch runs typical of OLTP workloads during a mass update.

To measure five application business process run times for a database representing large organization. The five processes are:

• Paysheet Creation: generates payroll data worksheet for employees, consisting of std payroll information for each employee for given pay cycle.

• Payroll Calculation: Looks at Paysheets and calculates checks for those employees.

• Payroll Confirmation: Takes information generated by Payroll Calculation and updates the employees' balances with the calculated amounts.

• Print Advice forms: The process takes the information generated by payroll Calculations and Confirmation and produces an Advice for each employee to report Earnings, Taxes, Deduction, etc.

• Create Direct Deposit File: The process takes information generated by above processes and produces an electronic transmittal file use to transfer payroll funds directly into an employee bank a/c.

For the benchmark, we collect at least four data points with different number of job streams (parallel jobs). This batch benchmark allows a maximum of eight job streams to be configured to run in parallel.

Key Points and Best Practices

• Why Sun Storage F5100 is a good option for Peoplesoft NA Payroll Application

• Accelerate your Payroll Performance with F20 PCIe Card

See Also

• PeopleSoft White Papers

Disclosure Statement

Oracle PeopleSoft Payroll (NA) 9.0 benchmark, Sun M4000 (4 2.53GHz SPARC64) 79.35 min, IBM Z990 (6 gen1) 107.34 min, HP rx6600 (4 1.6GHz Itanium2) 105.70 min, www.oracle.com/apps_benchmark/html/white-papers-peoplesoft.html Results 10/13/2009.

TPC-C Sun SPARC Enterprise T5440 with Oracle RAC World Record Database Result

Sun and Oracle demonstrate the World's fastest database performance. Sun Microsystems using 12 Sun SPARC Enterprise T5440 servers, 60 Sun Storage F5100 Flash arrays and Oracle 11g Enterprise Edition with Real Application Clusters and Partitioning delivered a world-record TPC-C benchmark result.

• The 12-node Sun SPARC Enterprise T5440 server cluster result delivered a world record TPC-C benchmark result of 7,646,486.7 tpmC and $2.36 $/tpmC (USD) using Oracle 11g R1 on a configuration available 12/14/09.

• The 12-node Sun SPARC Enterprise T5440 server cluster beats the performance of the IBM Power 595 (5GHz) with IBM DB2 9.5 database by 26% and has 16% better price/performance on the TPC-C benchmark.

• The complete Oracle/Sun solution used 10.7x better computational density than the IBM configuration (computational density = performance/rack).

• The complete Oracle/Sun solution used 8 times fewer racks than the IBM configuration.

• The complete Oracle/Sun solution has 5.9x better power/performance than the IBM configuration.

• The 12-node Sun SPARC Enterprise T5440 server cluster beats the performance of the HP Superdome (1.6GHz Itanium2) by 87% and has 19% better price/performance on the TPC-C benchmark.

• The Oracle/Sun solution utilized Sun FlashFire technology to deliver this result. The Sun Storage F5100 flash array was used for database storage.

• Oracle 11g Enterprise Edition with Real Application Clusters and Partitioning scales and effectively uses all of the nodes in this configuration to produce the world record performance.

• This result showed Sun and Oracle's integrated hardware and software stacks provide industry-leading performance.

More information on this benchmark will be posted in the next several days.

Performance Landscape

TPC-C results (sorted by tpmC, bigger is better)

System	tpmC	Price/tpmC	Avail	Database	Cluster	Racks	w/KtpmC
12 x Sun SPARC Enterprise T5440	7,646,487	2.36 USD	12/14/09	Oracle 11g RAC	Y	9	9.6
IBM Power 595	6,085,166	2.81 USD	12/10/08	IBM DB2 9.5	N	76	56.4
Bull Escala PL6460R	6,085,166	2.81 USD	12/15/08	IBM DB2 9.5	N	71	56.4
HP Integrity Superdome	4,092,799	2.93 USD	08/06/07	Oracle 10g R2	N	46	to be added

Avail - Availability date
w/KtmpC - Watts per 1000 tpmC
Racks - clients, servers, storage, infrastructure

Results and Configuration Summary

Hardware Configuration:

9 racks used to hold

Servers:

12 x Sun SPARC Enterprise T5440
4 x 1.6 GHz UltraSPARC T2 Plus
512 GB memory
10 GbE network for cluster

Storage:

60 x Sun Storage F5100 Flash Array
61 x Sun Fire X4275, Comstar SAS target emulation
24 x Sun StorageTek 6140 (16 x 300 GB SAS 15K RPM)
6 x Sun Storage J4400
3 x 80-port Brocade FC switches

Clients:

24 x Sun Fire X4170, each with
2 x 2.53 GHz X5540
48 GB memory

Software Configuration:

Solaris 10 10/09
OpenSolaris 6/09 (COMSTAR) for Sun Fire X4275
Oracle 11g Enterprise Edition with Real Application Clusters and Partitioning
Tuxedo CFS-R Tier 1
Sun Web Server 7.0 Update 5

Benchmark Description

TPC-C is an OLTP system benchmark. It simulates a complete environment where a population of terminal operators executes transactions against a database. The benchmark is centered around the principal activities (transactions) of an order-entry environment. These transactions include entering and delivering orders, recording payments, checking the status of orders, and monitoring the level of stock at the warehouses.

POSTSCRIPT: Here are some comments on IBM's grasping-at-straws-perf/core attacks on the TPC-C result:
c0t0d0s0 blog: "IBM's Reaction to Sun&Oracle TPC-C

See Also

• TPC-C Official Result Page
  

• www.tpc.org Website

Disclosure Statement

TPC Benchmark C, tpmC, and TPC-C are trademarks of the Transaction Performance Processing Council (TPC). 12-node Sun SPARC Enterprise T5440 Cluster (1.6GHz UltraSPARC T2 Plus, 4 processor) with Oracle 11g Enterprise Edition with Real Application Clusters and Partitioning, 7,646,486.7 tpmC, $2.36/tpmC. Available 12/14/09. IBM Power 595 (5GHz Power6, 32 chips, 64 cores, 128 threads) with IBM DB2 9.5, 6,085,166 tpmC, $2.81/tpmC, available 12/10/08. HP Integrity Superdome(1.6GHz Itanium2, 64 processors, 128 cores, 256 threads) with Oracle 10g Enterprise Edition, 4,092,799 tpmC, $2.93/tpmC. Available 8/06/07. Source: www.tpc.org, results as of 10/11/09.