David Dagastine's Weblog

« October 2008
Sun	Mon	Tue	Wed	Thu	Fri	Sat
			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30	31

Today

I received a few comments on my previous blog entry saying the results were bogus since I used an old compiler. I quickly found another test system running Suse SLES 9 U2 with gcc 3.3.3 and repeated the test. If I get around to installing the latest Visual Studio I'll repeat the test there as well. The JVM versions are different as I wanted to quickly post the results. Guess what, the results are a lot better! I ran this several times and its quite repeatable. I appreciate comments so please let me know what your thoughts. Especially if there are issues with the choice of gcc 3.3.3. The system under test was a 2 x 3.0Ghz Intel Xeon MP System (4-core) running Suse SLES 9 U2 and gcc 3.3.3. The C code was compiled with full optimization as shown by the Makefile in the SciMark source package. This time no tuning parameters were used for either 5.0_08 or 6.0 b83. Here's some output from /proc/cpuinfo: vendor_id : GenuineIntel cpu family : 15 model : 4 model name : Intel(R) Xeon(TM) MP CPU 3.00GHz For background here's the skinny on SciMark2. Scimark2 is a set of simple numerical kernels and its performance is directly related to the performance and quality of the generated code. The tests are single threaded and have little to no garbage collection overhead. In short, a great set of applications to compare statically compiled C code and dynamically compiled Java. This time Java is 35% faster than C.

Here's a breakdown of the subtests. C is only ahead on Sparse MatMult by a small margin.

Any one interested to see how the other JVM vendors look? Can JRockit or IBM beat C?

Comments:

I don't get similar results at all. While Java is faster on some subtests, C is overall faster. On an 8-core v490 (four 1.35GHz USIV chips) running Solaris 10 FCS, I get these results:

with S1S11:

$ cc -V   
cc: Sun C 5.8 Patch 121015-02 2006/03/29
usage: cc [ options] files.  Use 'cc -flags' for details
$ cc -o scimark2 -fast *.c     
$ ./scimark2
**                                                              **
** SciMark2 Numeric Benchmark, see http://math.nist.gov/scimark **
** for details. (Results can be submitted to pozo@nist.gov)     **
**                                                              **
Using       2.00 seconds min time per kenel.
Composite Score:          277.96
FFT             Mflops:   110.96    (N=1024)
SOR             Mflops:   558.54    (100 x 100)
MonteCarlo:     Mflops:    58.61
Sparse matmult  Mflops:   200.42    (N=1000, nz=5000)
LU              Mflops:   461.26    (M=100, N=100)

With Java 1.5.0_07

$ java -server -cp scimark2lib.jar jnt.scimark2.commandline

SciMark 2.0a

Composite Score: 171.94382170104376
FFT (1024): 132.9857133778735
SOR (100x100):   330.63743742225955
Monte Carlo : 15.484278331110477
Sparse matmult (N=1000, nz=5000): 139.55706689915522
LU (100x100): 241.0546124748199

java.vendor: Sun Microsystems Inc.
java.version: 1.5.0_07
os.arch: sparc
os.name: SunOS
os.version: 5.10

With Mustang b86:

$ ./jdk1.6.0/bin/java -server -cp scimark2lib.jar jnt.scimark2.commandline

SciMark 2.0a

Composite Score: 178.79431368149795
FFT (1024): 132.09940756695042
SOR (100x100):   330.8645341906918
Monte Carlo : 58.50816619223501
Sparse matmult (N=1000, nz=5000): 128.6028328695025
LU (100x100): 243.89662758811002

java.vendor: Sun Microsystems Inc.
java.version: 1.6.0-beta2
os.arch: sparc
os.name: SunOS
os.version: 5.10

Posted by Derek Morr on June 09, 2006 at 04:39 PM EDT #

Interesting...I would still like to see JRockit and IBM scores as well.

Posted by Surya on June 09, 2006 at 05:40 PM EDT #

$ cc -v Using built-in specs. Target: i386-redhat-linux Configured with: ../configure --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-libgcj-multifile --enable-languages=c,c++,objc,obj-c++,java,fortran,ada --enable-java-awt=gtk --disable-dssi --with-java-home=/usr/lib/jvm/java-1.4.2-gcj-1.4.2.0/jre --with-cpu=generic --host=i386-redhat-linux Thread model: posix gcc version 4.1.1 20060525 (Red Hat 4.1.1-1) compiled with -O3 -funroll-all-loops $ ./scimark2 ** ** ** SciMark2 Numeric Benchmark, see http://math.nist.gov/scimark ** ** for details. (Results can be submitted to pozo@nist.gov) ** ** ** Using 2.00 seconds min time per kenel. Composite Score: 584.10 FFT Mflops: 638.92 (N=1024) SOR Mflops: 558.54 (100 x 100) MonteCarlo: Mflops: 104.45 Sparse matmult Mflops: 461.52 (N=1000, nz=5000) LU Mflops: 1157.06 (M=100, N=100) $ /usr/java/latest/bin/java -version java version "1.6.0-beta2" Java(TM) SE Runtime Environment (build 1.6.0-beta2-b86) Java HotSpot(TM) Client VM (build 1.6.0-beta2-b86, mixed mode, sharing) $ /usr/java/latest/bin/java -client -cp scimark2lib.jar jnt.scimark2.commandline SciMark 2.0a Composite Score: 414.416180412051 FFT (1024): 363.99009455020337 SOR (100x100): 556.2803162318543 Monte Carlo : 79.98672264643358 Sparse matmult (N=1000, nz=5000): 495.73372674922945 LU (100x100): 576.0900418825344 java.vendor: Sun Microsystems Inc. java.version: 1.6.0-beta2 os.arch: i386 os.name: Linux os.version: 2.6.16-1.2096_FC5

Posted by Mikael Gueck on June 09, 2006 at 06:11 PM EDT #

I should have been more specific in the results I was referencing, I was using the scimark -large dataset. Also, figures that the Sun Studio Compiler would set me straight :-)

Posted by dagastine on June 09, 2006 at 07:05 PM EDT #

Mikael, Your running on a client system and need to use the -server flag for optimal performance. Your also using the default dataset rather than -large (I should have been more specific). Could you try -server for your tests? Thanks, -dave

Posted by dagastine on June 09, 2006 at 07:07 PM EDT #

With -large I'm getting results similar to yours.

Posted by Mikael Gueck on June 10, 2006 at 01:13 AM EDT #

Thanks to Derek Morr for posting the Studio 11 compiler results, they just confirm the good work that the compiler team has been doing. This sort of discussion always seems to polarise opinions, but I hope it encourages some healthy competition - then everybody benefits.

Posted by Michael van der Westhuizen on June 10, 2006 at 06:03 PM EDT #

I'm still not seeing that Java is faster than C. I reran the tests with the large dataset on a v20z (dual Opteron 242s) running CentOS 4.3 with the bundled gcc 3.4.5. Java 1.6.0b86:

Composite Score: 206.23725812121938
FFT (1048576): 27.228797028280887
SOR (1000x1000):   483.2690586264598
Monte Carlo : 100.38723018356215
Sparse matmult (N=100000, nz=1000000): 183.05327585601393
LU (1000x1000): 237.24792891178006

gcc 3.4.5:

Composite Score:          204.05
FFT             Mflops:    38.20    (N=1048576)
SOR             Mflops:   358.16    (1000 x 1000)
MonteCarlo:     Mflops:   140.91
Sparse matmult  Mflops:   227.56    (N=100000, nz=1000000)
LU              Mflops:   255.43    (M=1000, N=1000)

While Java (slightly) beats C in composite, the C version is still significantly faster on most of the subtests, and that's using an older compiler with only minimal optimization (-O). Still, the speed boost of Java 1.6 vs 1.5 is impressive.

Posted by Derek Morr on June 10, 2006 at 11:55 PM EDT #