A prominent Seismic Processing algorithm, Reverse Time Migration with Optimal Checkpointing, in SMP "THREADS" Mode, was testing using a Sun Fire X4270 server configured with four high performance 15K SAS hard disk drives (HDDs) and a Sun Storage F5100 Flash Array. This benchmark compares I/O devices for checkpointing wave state information while processing a production seismic migration.
- Sun Storage F5100 Flash Array is 2.2x faster than high-performance 15K RPM disks.
- Multithreading the checkpointing using the Sun Studio C++ Compiler OpenMP implementation gives a 12.8x performance improvement over the original single threaded version.
These results show the new trend in seismic processing to run iterative Reverse Time Migrations and migration playback is a reality. This is made possible through the use of Sun FlashFire technology to provide good checkpointing speeds without additional disk cache memory. The application can take advantage of all the memory within a node without regard to checkpoint cache buffers required for performance to HDDs. Similarly, larger problem sizes can be solved without increasing the memory footprint of each computational node.
Performance Landscape
| Reverse Time Migration Optimal Checkpointing - SMP Threads Mode Grid Size -800 x 1151 x 1231 with 800 Samples - 60GB of memory |
|||||||||
|---|---|---|---|---|---|---|---|---|---|
| Number Checkpts |
HDD | F5100 | |||||||
| Put Time (secs) |
Get Time (secs) |
Total Time (secs) |
Put Time (secs) |
Get Time (secs) |
Total Time (secs) |
F5100 Speedup |
|||
| 80 | 660.8 | 25.8 | 686.6 | 277.4 | 40.2 | 317.6 | 2.2x | ||
| 400 | 1615.6 | 382.3 | 1997.9 | 989.5 | 269.7 | 1259.2 | 1.6x | ||
| Reverse Time Migration Optimal Checkpointing - SMP Threads Mode Grid Size -125 x 1151 x 1231 with 800 Samples - 9GB of memory |
|||||||||
|---|---|---|---|---|---|---|---|---|---|
| Number Checkpts |
HDD | F5100 | |||||||
| Put Time (secs) |
Get Time (secs) |
Total Time (secs) |
Put Time (secs) |
Get Time (secs) |
Total Time (secs) |
F5100 Speedup |
|||
| 80 | 10.2 | 0.2 | 10.4 | 8.0 | 0.2 | 8.2 | 1.3x | ||
| 400 | 52.3 | 0.4 | 52.7 | 45.2 | 0.3 | 45.5 | 1.2x | ||
| 800 | 102.6 | 0.7 | 103.3 | 91.8 | 0.6 | 92.4 | 1.1x | ||
| Reverse Time Migration Optimal Checkpointing Single Thread vs Multithreaded I/O Performance Grid Size -125 x 1151 x 1231 with 800 Samples - 9GB of memory |
|||||||||
|---|---|---|---|---|---|---|---|---|---|
| Number Checkpts |
Single Thread F5100 Total Time (secs) |
Multithreaded F5100 Total Time (secs) |
Multithread Speedup |
||||||
| 80 | 105.3 | 8.2 | 12.8x | ||||||
| 400 | 482.9 | 45.5 | 10.6x | ||||||
| 800 | 963.5 | 92.4 | 10.4x | ||||||
Note: Hyperthreading and Turbo Mode enabled while running 16 threads per node.
Results and Configuration Summary
Hardware Configuration:
-
Sun Fire 4270 Server
-
2 x 2.93 GHz Quad-core Intel Xeon X5570 processors
72 GB memory
4 x 73 GB 15K SAS drives
-
File system striped across 4 15K RPM high-performance SAS HD RAID0
-
20 x 24 GB flash modules
Software Configuration:
-
OS: 64-bit SUSE Linux Enterprise Server SLES 10 SP 2
Compiler: Sun Studio 12 C++, Fortran, OpenMP
Benchmark Description
The Reverse Time Migration (RTM) is currently the most popular seismic processing algorithm because of it's ability to produce quality images of complex substructures. It can accurately image steep dips that can not be imaged correctly with traditional Kirchhoff 3D or frequency domain algorithms. The Wave Equation Migration (WEM) can image steep dips but does not produce the image quality that can be achieved by the RTM. However, the increased computational complexity of the RTM over the WEM introduces new areas for performance optimization. The current trend in seismic processing is to perform iterative migrations on wide azimuth marine data surveys using the Reverse Time Migration.
The Reverse Time Migration with Optimal Checkpointing was introduced so large migrations could be performed within minimal memory configurations of x86 cluster nodes. The idea is to only have three wavestate vectors in memory for each of the source and receiver wavefields instead of holding the entire wavefields in memory for the duration of processing. With the Sun Flash F5100, this can be done with little performance penalty to the full migration time. Another advantage of checkpointing is to provide the ability to playback migrations and facilitate iterative migrations.
- The stored snapshot data can be reprocessed with different filtering, image conditioning, or a variety of other parameters.
- Fine grain snapshoting can help the processing of more complex subsurface data.
- A Geoscientist can "playback" a migration from the saved snapshots to visually validate migration accuracy or pick areas of interest for additional processing.
The Reverse Time Migration with Optimal Checkpointing is an algorithm designed by Griewank (Griewank, 1992; Blanch et al., 1998; Griewank, 2000; Griewank and Walther, 2000; Akcelik et al., 2003).
- The application takes snapshots of wavefield state data for some interval of the total number of samples.
- This adjoint state method performs crosscorrelation of the source and receiver wavefields at the each level.
- Forward recursion is used for the source wavefield and backward recursion for the receiver wavefield.
- For relatively small seismic migrations, all of the forward processed state information can be saved and restored with minimal impact on the total processing time.
- Effectively, the computational complexity increases while the memory requirements decrease by a logarithmic factor of the number of snapshots.
- Griewank's algorithm helps define the most optimal tradeoff between computational performance and the number of memory buffers (memory requirements) to support this cross correlation.
For the purposes of this benchmark, this implementation of the Reverse Time Migration with Optimal Checkpointing does not fully implement the optimal memory buffer scheme proposed by Griewank. The intent is to compare various I/O alternatives for saving wave state data for each node in a compute cluster.
This benchmark measures the time to perform the wave state saves and restores while simultaneously processing the wave state data.
Key Points and Best Practices
- Mulithreading the checkpointing using Sun Studio OpenMP and running 16 I/O threads with hyperthreading enabled gives a performance advantage over single threaded I/O to the Sun Storage F5100 flash array. The Sun Storage F5100 flash array can process concurrent I/O requests from multiple threads very efficiently.
- Allocating the majority of a node's available memory to the Reverse Time Migration algorithm and leaving little memory for I/O caching favors the Sun Storage F5100 flash array over direct attached high performance disk drives. This performance advantage decreases as the number of snapshots increase. The reason for this is that increasing the number of snapshots decreases the memory requirement for the application.
See Also
- Reverse Time Migration with Optimal Checkpointing
- IBM Research - Reverse Time Migration on multi-core processors