以实际算法为例评估MapReduce在石油勘探中的应用

doi:10.3969/j.issn.1000-436x.2012.z2.010

摘要/Abstract

摘要：

石油勘探领域需要处理海量的地震数据，以获取地下构造用以发现和定位油藏。为评估云计算编程模型MapReduce对于石油勘探领域应用算法的适用性，设计并实现了基于MapReduce的三维Fresnel层析成像算法，实验发现MapReduce版本的性能比MPI版本慢3倍，而且对MapReduce作业调优的难度相当大。为了拓展MapReduce在石油勘探领域高性能计算领域的应用，需要在支持线程级并行、灵活性和提升I/O可扩展性3个方面进行改进，并提出了研究方法和技术路线。

关键词: 石油勘探, MapReduce, Fresnel层析成像, 高性能计算, 地震数据

Abstract:

Huge amounts of seismic data undergo complex iterative processing in the oil industry to get knowledge of the earth’s subsurface structure to detect where oil can be found and recovered.To evaluate the suitability of MapReduce for seismic processing algorithms,the algorithm design and implementation of Fresnel tomography on Hadoop MapReduce was described.Experiments demonstrate that MapReduce is approximately 3 times slower than MPI,and tuning the performance of MapReduce is really hard.To expand its applicability to high performance computing for oil industry,MapReduce should be improved in the flexibility and provide the opportunity to exploit fine-grained thread-level parallelism.Finally,research ideas to achieve these objectives were presented.

Key words: oil exploration, MapReduce, Fresnel tomography, high performance computing, seismic data

赵长海,晏海华,刘晓朋,熊登,史晓华. 以实际算法为例评估MapReduce在石油勘探中的应用[J]. 通信学报, 2012, 33(Z2): 81-89.

Chang-hai ZHAO,Hai-hua YAN,Xiao-peng LIU,Deng XIONG,Xiao-hua SHI. Evaluating MapReduce for seismic data processing using a practical application[J]. Journal on Communications, 2012, 33(Z2): 81-89.

图/表 8

图1

图2

图3

表1

图4

图5

图6

图7

参考文献 30

[14]	The apache software foundation. MapReduce tutorial[EB/OL]. , 2012.
[15]	王珊, 王会举, 覃雄派 ,等. 架构大数据:挑战、现状与展望[J]. 计算机学报, 2011,34(10): 1741-1752. WANG S , WANG H J , TAN X P ,et al. Architecting big data:challenges,studies and forecasts[J]. Chinese Journal of Computers, 2011,34(10): 1741-1752.
[16]	RAICU I , FOSTER I T , BECKMAN P . Making a case for distributed file systems at Exascale[A]. LSAP '11[C]. San Jose,USA, 2011. 11-18.
[17]	YANG H , DASDAN A , HSIAO R ,et al. Map-reduce-merge:simplified relational data processing on large clusters[A]. SIGMOD '07[C]. Beijing,China, 2007. 1029-1040.
[18]	EKANAYAKE J , LI H , ZHANG B ,et al. Twister:a runtime for iterative MapReduce[A]. HPDC '10[C]. Chicago,USA, 2010. 810-818.
[19]	ZHANG B , RUAN Y , WU T ,et al. Applying twister to scientific applications[A]. CLOUDCOM'10[C]. Indianapdis,USA, 2010. 25-32.
[20]	KAMBATLA K , PATHAK A , PUCHA H . Towards optimizing hadoop provisioning in the cloud[A]. HotCloud'09[C]. San Diego,CA,USA, 2009.
[21]	SANDHOLM T , LAI K . MapReduce optimization using regulated dynamic prioritization[A]. SIGMETRICS'09[C]. 2009. 299-310.
[22]	ZAHARIA M , Konwinski A , Joseph A D ,et al. Improving MapReduce performance in heterogeneous environments[A]. OSDI'08[C]. San Diego,CA,USA, 2008. 29-42.
[23]	EKANAYAKE J , PALLICKARA S , FOX G . Mapreduce for data intensive scientific analyses[A]. IEEE Fourth International Conference on eScience[C]. Indianapdis,USA, 2008. 277-284.
[24]	TALBOT J , YOO R M , KOZYRAKIS C . Phoenix++:modular MapReduce for shared-memory systems[A]. MapReduce '11[C].San Jose,USA. 2011. 9-16.
[25]	RANGER C , RAGHURAMAN R , PENMETSA A ,et al. Evaluating Mapreduce for multi-core and multiprocessor systems[A]. IEEE 13th International Symposium on High Performance Computer Architecture(HPCA)[C]. Phoenix,USA, 2007. 13-24.
[26]	JIANG W , RAVI V T , AGRAWAL G . A Map-Reduce system with an alternate API for multi-core environments[A]. CCGRID '10[C]. Melbourne,Australia, 2010. 84-93.
[27]	HE B , FANG W , LUO Q ,et al. Mars:a MapReduce framework on graphics processors[A]. PACT'08[C]. Toronto,Canada, 2008. 260-269.
[1]	赵改善 . 我们需要多大和多快的计算机[J]. 勘探地球物理进展, 2004,27(1): 22-28. ZHAO G S . How big and fast computers can meet out needs[J]. Progress in Exploration Geophysics, 2004,27(1): 22-28.
[2]	赵改善 . 高性能计算在石油物探中的应用现状与前景[J]. 高性能计算发展与应用, 2009(29): 19-23. ZHAO G S . Current status and outlook of high performance computing for seismic exploration[J]. Development and Application of High Performance Computing, 2009(29): 19-23.
[28]	CHANG H , VANDYKE J P , SOLANO M ,et al. 3-D prestack kirchhoff depth migration:from prototype to production in a massively parallel processor environment[J]. Geophysics, 1998,63(2): 546-556.
[29]	SAULE E , CATALYUREK U V . An early evaluation of the scalability of graph algorithms on the Intel MIC architecture[A]. 26th International Symposium on Parallel and Distributed Processing,Workshops and PhD Forum (IPDPSW),Workshop on Multithreaded Architectures and Applications(MTAAP)[C]. Shanghai,China, 2012.
[3]	BAUTISTA-GOMEZ L , TSUBOI S , KOMATITSCH D ,et al. FTI:high performance fault tolerance interface for hybrid systems[A]. SC '11[C]. Seattle,WA,USA, 2011. 31-32.
[4]	DEAN J , GHEMAWAT S . MapReduce:simplified data processing on large clusters[J]. Communications of the ACM, 2008,51(1): 107-113.
[5]	Oracle Inc . Lustre:a scalable high-performance file system[EB/OL]. , 2012.
[6]	SCHMUCK F , HASKIN R . GPFS:a shared-disk file system for large computing clusters[A]. FAST'02[C]. Monterey,CA,USA, 2002. 231-244.
[7]	SRIRAMA S N , JAKOVITS P , VAINIKKO E . Adapting scientific computing problems to clouds using MapReduce[J]. Future Generation Computer Systems, 2012,28(1): 184-192.
[8]	FADIKA Z , DEDE E , GOVINDARAJU M ,et al. MARIANE:MApReduce implementation adapted for HPC environments[A]. GRID'11[C]. Lyon,France, 2011. 82-89.
[9]	GUNARATHNE T , WU T , QIU J ,et al. MapReduce in the clouds for science[A]. CLOUDCOM'10[C]. USA, 2010. 565-572.
[10]	MACKEY G , SEHRISH S , BENT J ,et al. Introducing Map-reduce to high end computing[A]. Petascale Data Storage Workshop,2008.PDSW'08[C]. Austin,USA, 2008. 1-6.
[11]	Committee of SEG Technical Standards . SEG Y rev1 Data Exchange format[S]. Society of Exploration of Geophysicist, 2002.
[12]	张建中, 杨国辉, 林文 ,等. Fresnel层析成像并行算法研究[J]. 计算机研究与发展. 2007,44(10): 1661-1666. ZHANG J Z , YANG G H , LIN W ,et al. A parallel algorithm for fresnel tomography[J]. Journal of Computer Research and Development, 2007,44(10): 1661-1666.
[13]	刘玉柱, 董良国, 李培明 ,等. 初至波菲涅尔体地震层析成像[J]. 地球物理学报. 2009,52(9): 2310-2320. LIU Y Z , DONG L G , LI P M ,et al. Fresnel volume tomography based on the first arrival of the seismic wave[J]. Chinese Journal of Geophysics, 2009,52(9): 2310-2320.
[30]	Intel many integrated core architecture[EB/OL]. , 2012.

系统	配置
nodes	256
CPUs in each node	2,Intel? Xeon? Processor X5650 2.67GHz
Threads in each CPU	6(Cores)× 2(Threads)
Memory Per Node	24GB
Network	Gigabit
Disks	1,SCSI disk,Read Speed:171MB/sec
I/O servers	8
Software	Hadoop 0.21.0,Sun JDK 6,MPICH2 1.2
OS	Red Hat Enterprise Linux Server release 5.4-64 bit