BM Seer Unofficial thoughts from an anonymous Sun employee

Siebel CRM 8.0 PSPP UltraSPARC T2 beats POWER6 and sets World Record

Thursday Jan 10, 2008

arrgghhh... I've been asked to show only Sun's results. You must now do your own math with the information posted on Oracle's website: http://www.oracle.com/apps_benchmark/doc/Sun_Siebel8_10000_PSPP_On_Solaris.pdf
http://www.oracle.com/apps_benchmark/doc/IBM_Siebel8_7000_PSPP_On_AIX_POWER6%20Final.pdf

IBM now longer holds the world record and really needs to post a correction on:
http://www-03.ibm.com/systems/p/hardware/benchmarks/erp.html

Four Sun SPARC Enterprise T5120 and T5220 servers (UltraSPARC T2 processors) set a new World Record using Siebel's standard Platform Sizing and Performance Program (PSPP) benchmark suite with Siebel CRM 8.0 Industry Applications and Oracle 10g R2 DB running on Solaris 10.

The Sun results using the UltraSPARC T2 supported 30% higher Siebel benchmark concurrent users compared to other results on the Siebel CRM Applications Release 8.0.

Sun again shows the UltraSPARC T2 servers are ideally suited for Oracle database applications. The database server ran Oracle 10g R2 on this Siebel benchmark.

{ Stuff deleted }

Sun's Solaris and Coolthreads based servers proves once again to be the best combination for scalability and resource utilization in the datacenter, giving users a consistent response time on critical applications as shown 10,000 users benchmark on Siebel CRM 8.0.

The 10,000 Siebel benchmark users performance results on 4 Sun SPARC Enterprise T5120/T5220 servers running Solaris 10 delivers a scalable and cost-effective platform for deploying Siebel CRM Application and Oracle 10g R2 deployment.

The result of 10,000 active concurrent Siebel user benchmark was run end to end on the new generation of Sun SPARC Enterprise servers using coolthreads technology with the highest level of space and energy efficiency.

See Also: http://www.oracle.com/apps_benchmark/html/white-papers-siebel.html

Siebel CRM 8.0 PSPP Performance Chart as of 01/04/2008 (bigger is better)

Vendor Users Web Server Application Servers Database Server
Sun 10,000 1 x Sun SPARC Enterprise T5120
4 cores, 1 chip @1.2 GHz US-T2
    8 GB RAM
Siebel CRM 8.0 SIA [20204] ENU
Sun Java System Web
    Server 6.1 SP8
Solaris 10 8/07 		1 x Sun SPARC Enterprise T5220
8 cores, 1 chip @1.4 GHz US-T2
    32 GB RAM
1 x Sun SPARC Enterprise T5220
8 cores, 1 chip @1.2 GHz US-T2
    32 GB RAM
Siebel CRM 8.0 SIA [20204] ENU
Solaris 10 8/07		 1 x Sun SPARC Enterprise T5120
8 cores, 1 chip @1.2 GHz US-T2
    32 GB RAM
Oracle 10gR2 Database
    Server v10.2.0.1.0
Solaris 10 8/07
. . . . .

As noted on the official benchmark report: "Siebel CRM Release 8.0 Industry Application Platform Sizing and Performance benchmarks are based on Siebel CRM Release 8.0 customized industry applications and reflect a heavier scenario mix and more-aggressive think times than earlier version. Results of this benchmark are not comparable with those of prior Siebel CRM Release 7 benchmarks."

Benchmark Description

Siebel CRM 8.0 Platform Sizing and Performance Program (PSPP) is a multi-tier benchmark designed to stress the Siebel CRM Release 8.0 architecture and to demonstrate that large customers can successfully deploy many thousands of concurrent users. Among the Siebel CRM Release 8.0 architecture features exercised are the following:

  • Smart Web Architecture: Takes advantage of the newest Web browser technology to deliver a highly interactive experience. The interaction model, which is similar to Windows-based applications, also improves productivity. Utilization rates on the web server are low, allowing customers to retain existing Web server infrastructure.

  • Smart Network Architecture: Allows Siebel CRM Release 8.0 customers to leverage their existing network infrastructure by compressing and caching user interface components, so that browser/Web server interaction occurs only when the application requests data. This allows customers to avoid expensive network upgrades that can be necessary with competing products.

  • Server Connection Broker: The Siebel Connection Broker (SCBroker) is a server component that provides intraserver loadbalancing. SCBroker distributes server requests across multiple instances of Application Object Managers (AOMs) running on a Siebel server.

  • Smart Database Connection Pooling and Multiplexing: Allows customers to scale their database without intrducing expensive and complex transaction-processing monitors.

  • Server Request Broker: Server Request Broker (SRBroker) processes synchronous server requests - reuqests that must be run immediately, and for which the calling process waits for completion.

  • Enterprise Application Integration: Allows customers to integrate their existing systems with Siebel CRM applications.

  • eScript: eScript is a scripting or programming language that application developers use to write simple scripts to extend Siebel applications. Javascript, a popular scripting language used primarily on Web sites, is its core language.

The test simulated real-world requirements of a large organization, consisting of 10,000 concurrent, active users from multiple departments accessing a call center. Test conditions simulated service representatives running Siebel Financial Services Call Center and partner organizations running Siebel Partner Relationship Management (Web sales and Web service). Siebel Workflow and the Siebel Scripting Engine were used to incorporate business-process-management customizations. The application also simulated integration with Web systems, using the Siebel Enterprise Application Integration component and Siebel Web Services.

Disclosure Statement:

Siebel CRM 8.0 Platform Sizing and Performance Program (PSPP) benchmark as of 01/04/08. Sun Microsystems: 10,000 users, 1 x Sun SPARC Enterprise T5120 web server (4 cores, 1 chip @1.2 GHz US-T2, 8 GB RAM), Siebel CRM 8.0 SIA [20204] ENU, Sun Java System Web Server 6.1 SP8, Solaris 10 8/07, 1 x Sun SPARC Enterprise T5220 application server (8 cores, 1 chip @1.4 GHz US-T2, 32 GB RAM), 1 x Sun SPARC Enterprise T5220 application server (8 cores, 1 chip @1.2 GHz US-T2, 32 GB RAM) Siebel CRM 8.0 SIA [20204] ENU, Solaris 10 8/07, 1 x Sun SPARC Enterprise T5120 database server (8 cores, 1 chip @1.2 GHz US-T2, 32 GB RAM), Oracle 10gR2 Database Server v10.2.0.1.0, Solaris 10 8/07 Oracle, Siebel, registered trademarks of Oracle Corporation and/or its affiliates. More info www.oracle.com/apps_benchmark/html/white-papers-siebel.html

Power Reference:

Sun measured: Database Server (1.2 GHz T5120, 8 core, 32G memory): 291W, Gateway/Application Server #1 (1.4 GHz T5220, 8 core, 32G memory): 323W, Application Server #2 (1.2 GHz T5220, 8 core, 32G memory): 376W, Web Server (1.2 GHz T5120, 4 core, 8G memory): 212W.

IBM power calculation based on the following: The p570 is supplied in building blocks with 2 chips, 4 cores per chassis called a CEC. Up to 4 CECs can be connected together to create a single 16 chip, 32 core SMP system. Each CEC is 4 RU, and each CE is estimatedC to consume 1,040 watts when configured with 2 processors, based on the following: IBM p6 570 power specifications from 80% of maximum report power consumption published here, 06/07/07, posted at ftp://ftp.software.ibm.com/common/ssi/rep_sp/n/PSB01628USEN/PSB01628USEN.PDF

System Configuration

Certified Results 10,000 Users
Reference Date: January 4, 2008
Systems: 1 x Sun SPARC Enterprise T5120, web server (one 1.2GHz UltraSPARC T2)
1 x Sun SPARC Enterprise T5220, gateway/application server (one 1.4GHz UltraSPARC T2)
1 x Sun SPARC Enterprise T5220, application server(one 1.2GHz UltraSPARC T2)
1 x Sun SPARC Enterprise T5120, database server (one 1.2GHz UltraSPARC T2)
Operating System: Solaris 10 8/07
Software: Sun Java System Web Server 6.1 SP8
Siebel CRM 8.0 SIA [20204] ENU
Oracle 10gR2 Database Server v10.2.0.1.0

[21] Comments
Like this post? del.icio.us | furl | slashdot | technorati | digg
Posted at 06:58AM Jan 10, 2008 in Benchmark Issues
Tags:
Comments:

[Trackback] Bookmarked your post over at Blog Bookmarker.com!

Posted by consistent on January 10, 2008 at 09:20 AM PST #

Yeah, this result much better ! Real application, nice result.

Posted by Triffids on January 10, 2008 at 11:41 AM PST #

Hm ... Power6 response time 0.091 vs 0.242

Posted by Triffids on January 10, 2008 at 12:17 PM PST #

don't read too much into response time, the rules state that the response time must be less than a specified amount. IT IS NOT A PERFORMANCE METRIC.
Just like SAP-SD you need to have a specified response time, but it is the throughput metric that is the comparable.

Posted by BM Seer on January 10, 2008 at 12:36 PM PST #

I understand, IMHO response is scalability metric, because we can lock table and make tps by one user while other users idle. SPARC response the same as XEON results. Congratulations - great result.

P.S. Don't you know why on Linux (3500 users) network utilization measured was 19.04 Mbps vs 11.36 Mbps on Windows (3900 users) ?

Posted by Triffids on January 10, 2008 at 01:57 PM PST #

As it turns out with a good benchmark design like Siebel (or SAP),
they make sure you don't idled users to make throughput.

Posted by BM Seer on January 10, 2008 at 03:11 PM PST #

The pSeries platform used for this benchmark is bizarre though. The design is nonsense.

for example, who would use a 2 way p570 as a web server, when a 1U quad core p505q or 2U p510q is mucy more suited to this role.

Also, why 2 seperate P6-p570 systems for App and DB ? Why not a single partitioned p570 ?

If a comparison is to be made, at least make the test platforms realistic.

Posted by AlexR on January 11, 2008 at 06:54 AM PST #

IBM doesn't think Power6 is bizarre, they configured this system because this is what they consider a solution to be.

Posted by BM Seer on January 11, 2008 at 08:39 AM PST #

The Sun results using the UltraSPARC T2 supported 42% (and not just 30%) higher Siebel benchmark concurrent users as 10,000 / 7,000 = 1,42857...

Posted by Danilo Poccia on January 12, 2008 at 12:35 PM PST #

Yeah,

It's find to use the T2 if your application response time is quick already < 1s but if your app take 3-4 seconds on a intel/Power6 box, then it will be 16-20 secs on a T2. More users, but 2-4 times the response time.

Fine for packaged apps that are already written for previous gen CPU's, but new apps, written with developers with high powered pentium 4's / duo's on the desktop, you might find a bit of a issue.

If you are targeting T2 make sure your developers have 1 Ghz desktops to develop on or are over 40 years of age.

Posted by Crook on January 12, 2008 at 01:03 PM PST #

Response time in NOT AN issue in the Siebel benchmark. You guys are reasoning about the wrong data.

The Siebel benchmark was designed to assure response times for all users and is sub-second = .242sec.

Crook, we have measured a wide variety of new/old workloads with wider range then you assert. Your assumptions are completely wrong.

Posted by BM Seer on January 13, 2008 at 11:07 PM PST #

Bottom line is, the IBM benchmark is flawed due to bizarre kit selection and configuration.

Thats not Suns fault, and i can only speculate as to why IBM chose the kit they did for this benchmark.

However, it does mean that announcements highlighting significant differences in performance and power consumption are also misleading, as your comparing against environments which are quite different.

Posted by 217.64.239.41 on January 14, 2008 at 08:50 AM PST #

Actually IBM is pushing this kind of architecture all the time. The comparisons are not misleading. IBM really wants people to drop lots
of $$$ for power6 and continue to think it is the best thing ever.
Power6 is not, it is expensive, slow, power consuming. We see
IBM pushing it at many customers. To prove it Sun does tests on their workloads and then customers buy US T2.

You need to think of it as a blackbox, IBM first put together what they
thought was a great configuration and posted results. Sun's US T2 products are really revolutionary. US T2 servers are better for a wide variety of benchmarks including this one.

Posted by BM Seer on January 14, 2008 at 10:56 AM PST #

Hi,

Re my previous comments points, we have benchmarked a Java J2EE application on Sun M5000 with 4 cpu's vs a T2000, both fully loaded boxes.

Response time of the application:

M5000
Average Response Time: 4s.

T2000
Average Response Time: 20s.

Through put is about equal, now 4s in within customer expectations, but 20s is far too slow. All the developers say it's fine on they development machines, fast x86’s and single thread tests and or AIX boxes.

Conclusion T2000 (CMT type architecture boxes) are much cheap per user (1/4 of the cost), but the application must be already fast for a single thread of execution/parallel to benefit from the reduced costs. Otherwise fast single threaded processors win hands down as they meet the business requirements for response time, regardless of throughput.

Posted by Crook on January 14, 2008 at 11:12 AM PST #

Crook, I would not expect this difference, have you worked with Sun people to figure out the root issue?

I wouldn't over-generalize from one result, as most cases don't show this.

Posted by BM Seer on January 14, 2008 at 01:24 PM PST #

Hi,

We have worked through the it with Sun (Thanks for your help) and have concluded that our app needs faster single threaded performance and memory bandwidth, I agree that we should not generalize from one result, (The response time posted is a simple aggregate of 2 months testing). I just wanted to warn that not all apps will be optimal of the CMT platform, but if your app is, go for it, it's a lot of bang for buck. And if you are designing a new app make sure your developers understand that a CPU bound thread will run slower than their Core Duo desktops in terms of execution time.

Posted by Crook, on January 14, 2008 at 01:36 PM PST #

hi

Just curious, does Sun sell anything beyond its T2 based servers?
Vis-a-vis Power6 - no benchmarks, no data on performance, no power usage comparisons for equivalent performance.

Your comparisons are like IBM comparing a p505 with a E6900 / M8000 and saying it gives better price performance, better power performance. Or HP comparing a rx2600 with a E6900 / M8000.

Posted by Venki on January 15, 2008 at 02:04 AM PST #

Interesting information here and nice to see Sun publishing competitive benchmarks. Having read all of the comments, I agree with what Crook is saying - single threaded performance still matters.

I personally like the price point of the T2's, so:

Question, are there any other "head-to-head" benchmarks with Power6 published. It would be nice to see how T2 competes on different types of applications, especially since you state that UltraSPARC T2 servers are ideally suited for Oracle database applications.

Finally, are you able to offer any insights on Niagara3 and Rock?

Posted by Mr Hunt on January 15, 2008 at 06:34 AM PST #

...for Mr Hunt:
Sun has published all kinds of benchmarks, already blogged about,
click on the CMT, T5120, t5220 tags... or on sun.com...
http://www.sun.com/servers/coolthreads/t5220/benchmarks.jsp
http://www.sun.com/servers/coolthreads/t5120/benchmarks.jsp

Venki:
If on a real application, a small, less costly, low watt, more-revolutionary system based on the US T2 beats the performance
of the current, slower, larger, coal-era design of IBM Power6
p570 4-core, how is that unfair? Sounds like the power6 IBM design
is quickly ready for the history books. Unless you want to spend
lots more on IBM's current system and get less.

Posted by BM Seer on January 15, 2008 at 11:26 AM PST #

I agree it is a real application but what if I want to scale to say 40,000 SAPS in a single OS instance - how do I do that with a T2 server?

Thus the fair comparison for a IBM Power6 p570 would be a equivalently scalable box from Sun (say a M8000/9000) and then compare the price / performance and the power / performance.

Posted by Venki on January 17, 2008 at 04:38 AM PST #

Please don't be sloppy by saying IBM Power6 570, a 4-core IBM p570 4.7GHz can not get to 40000.

A 4-core IBM 4.7GHz p570 (with 32GB fast memory) was soundly beaten in this Siebel benchmark. This is comparable result and IBM loses.

Also the reason why IBM uses two 4-core IBM p570s POWER6 instead a 8-core is that there are cables that add latency and cut bandwidth to make two building blocks into a 8-core system. IBM fails to post LMbench results that would simply prove this.

Posted by BM Seer on January 17, 2008 at 10:06 AM PST #

Post a Comment:
Comments are closed for this entry.