BestPerf: Performance & Best Practices

Significance of Results

The ProMAX family of seismic data processing tools is the most widely used Oil and Gas Industry seismic processing application. ProMAX is used for multiple applications, from field processing and quality control, to interpretive project-oriented reprocessing at oil companies and production processing at service companies. ProMAX is integrated with Halliburton's OpenWorks Geoscience Oracle Database to access prestack seismic data and populate the database with seismic images. This shows the powerful combination of scientific computing merged with commercial database technology.

A cluster of 48 Sun Blade X6275 server modules in a Sun Blade 6048 Modular System was configured with QDR Infiniband and a Lustre File System to demonstrate performance on ProMAX.

• The 3D Prestack Kirchhoff Time Migration showed excellent scalability while utilizing the QDR Infiniband Lustre Filesystem.

  • 70808 Traces : 144x improvement going from 1 to 72 nodes
  • 283232 Traces : 98x improvement going from 1 to 96 nodes
  
  
• The super linear scalability is attributed in part to data caching effects.


• Improved the performance of the current production release of the ProMAX 3D Kirchhoff Time Migration by a factor up to 1.7x through recompilation of the source code using the Intel 11.1 compilers.

High Performance ProMAX allows Halliburton's GeoProbe interpretation application to perform migrations "on the fly" while pulling additional mapping data, well logs, and reservoir data from the OpenWorks Oracle Database.

• Improves velocity modeling throughput for performing iterative Kirchhoff Migrations


• Sun Grid Engine can be used to optimize the throughput of multiple migrations and maximize the return on investment of a Sun Blade 6048 Modular System.

Enabling hyperthreading and running 16 threads per node can benefit current and potential ProMAX users running on a Sun Blade X6275 configuration. In the tests run with the code rebuild, hyperthreading outperformed non-hyperthreading by as much as 27%.

Performance Landscape

Note: Results are all run with 16 Threads per Node with HyperThreading Enabled.

ProMAX 3D Prestack Kirchhoff Time Migration SMP Threads and PVM Mode
Nodes	Procs	70808 Traces		283232 Traces
		Current Release Execution Time (sec)	Code Rebuild Execution Time (sec)	Current Release Execution Time (sec)	Code Rebuild Execution Time (sec)
96	192			18	13
72	144	3	2	22	16
48	96	4	3	38	23
24	48	11	7	76	48
16	32	23	14	117	72
12	24	37	23	165	100
8	16	62	35	258	150
4	8	129	78	514	343
1	2	486	288	2022	1278

ProMAX 3D Prestack Kirchhoff Time Migration Scalability
Nodes	Procs	70808 Traces		283232 Traces
		Current Release Speedup 1-Node	Code Rebuild Speedup 1-Node	Current Release Speedup 1-Node	Code Rebuild Speedup 1-Node
96	192			112	98
72	144	162	144	92	80
48	96	121	96	53	55
24	48	44	41	26	26
16	32	21	20	17	18
12	24	13	12	12	13
8	16	8	8	8	8
4	8	4	4	4	4
1	2	1	1	1	1

ProMAX 3D Prestack Kirchhoff Time Migration 283232 Traces Hyperthreading Performance Comparison
Nodes	Procs	Current Release		Code Rebuild
		8 Threads per Node Perf (sec)	16 Threads per Node Perf (sec)	8 Threads per Node Perf (sec)	16 Threads per Node Perf (sec)
24	48	95	76	59	48
16	32	144	117	91	72
12	24	196	165	122	100
8	16	310	258	190	150
4	8	629	514	390	343
1	2	2518	2022	1554	1278

Results and Configuration Summary

Hardware Configuration:

Sun Blade 6048 Modular System with

48 x Sun Blade X6275 (Vayu) Server Modules, each with

4 x 2.93 GHz Intel Xeon QC X5570 processors
12 x 4 GB memory at 1333 MHz
2 x 24 GB Internal Flash

QDR InfiniBand
Lustre 1.8.0.1 File System

Software Configuration:

OS: 64-bit SUSE Linux Enterprise Server SLES 10 SP 2
PVM: Parallel Virtual Machine
Resource Management: Sun Grid Engine
Compiler: GNU C++ 4.1.2, Intel 11.1 Compilers
Database: OpenWorks Database requires Oracle 10g Enterprise Edition
Additional Libraries: pthreads 2.4, Java 1.6.0_01, BLAS, Stanford Exploration Project Libraries

Benchmark Description

This benchmark compares the current production release of ProMAX built with the GNU C++ and Fortran compilers to builds with the Intel Fortran and C++ compilers. Two different problem sizes were evaluated with the ProMAX 3D Prestack Kirchhoff Time Migration:

• 70808 traces with 8 msec sample interval and trace length of 4992 msec
• 283232 traces with 8 msec sample interval and trace length of 4992 msec

The ProMAX processing parameters used for this benchmark:

• Input data set = Shots
  Minimum output inline = 65
  Maximum output inline = 85
  Inline output sampling interval = 1
  Minimum output xline = 1
  Maximum output xline = 200
  Xline output sampling interval = 1
  Antialias inline spacing = 15
  Antialias xline spacing = 15
  Stretch Mute Aperature Limit with Maximum Stretch = 15
  Image Gather Type = Full Offset Image Traces
  No Block Moveout
  Number of Alias Bands = 10
  3D Amplitude Phase Correction
  No compression
  Maximum Number of Cache Blocks = 500000
  

The compiler flags used for the various builds:

• The Current Production Release Code was built with GNU Fortran and C++ flags.
  -O3 -m64 -march=x86-64 -mieee-fp -mfpmath=sse -msse2 -fforce-addr -fno-inline-functions


• The application was rebuilt with the Intel Fortran and C++ flags.
  -xSSE4.2 -O3 -ipo -no-prec-div -static -m64 -ftz -fast-transcendentals -fp-speculation=fast

Key Points and Best Practices

Super linear scalability of the 70808 trace case for the larger node runs can be attributed to the fact that the dataset decomposition fits in cache which is shared by multiple threads per core.

See Also

• Oracle in Oil and Gas Upstream
• Oracle Committed to Oil and Gas Industry
• ProMAX Family Seismic Data Processing Software
• OpenWorks Geoscience Database Software
• GeoProbe Volume Interpretation Software

Disclosure Statement

The following are trademarks or registered trademarks of Halliburton/Landmark Graphics: ProMAX, GeoProbe, OpenWorks.