大数据 ›› 2020, Vol. 6 ›› Issue (4): 30-39.doi: 10.11959/j.issn.2096-0271.2020031

• 专题:大数据异构并行系统 • 上一篇    

面向大数据异构系统的神威并行存储系统

何晓斌1,蒋金虎2   

  1. 1 国家并行计算工程技术研究中心,北京 100080
    2 复旦大学计算机科学技术学院,上海 200433
  • 出版日期:2020-07-15 发布日期:2020-07-18
  • 作者简介:何晓斌(1984- ),男,国家并行计算工程技术研究中心助理研究员,主要研究方向为超大规模存储系统、新型存储软件协议栈等技术|蒋金虎(1974- ),男,复旦大计算机科学技术学院高级工程师,主要研究方向为操作系统、分布式存储

Sunway parallel storage system for big data heterogeneous system

Xiaobin HE1,Jinhu JIANG2   

  1. 1 National Parallel Computing Engineering Technology Research Center,Beijing 100080,China
    2 School of Computer Science and Technology,Fudan University,Shanghai 200433,China
  • Online:2020-07-15 Published:2020-07-18

摘要:

随着大数据应用和传统高性能计算应用的融合以及异构计算的引入,传统面向高性能计算的并行存储系统面临着异构计算I/O支持差、性能干扰和效率低等问题。通过在系统架构引入多层次存储架构、设计缓存映射机制来减轻I/O负载。在转发服务层,调整I/O转发策略,均衡I/O负载。在后端存储层,对系统高可用功能进行调整,解决大数据I/O访问模式与原有高可用措施的冲突。经过优化设计和完善后的并行存储系统更好地适应了异构众核架构,使得某些应用获得了10倍以上的I/O性能提升。

关键词: 大数据, 高性能计算, 神威·太湖之光, 异构, 并行存储

Abstract:

With the integration of big data applications and traditional high-performance computing applications and the introduction of heterogeneous computing,the traditional parallel storage system for high-performance computing faces the problems of poor I/O support,performance interference,and low efficiency.By introducing multi-level storage architecture into the system architecture,the cache mapping mechanism was designed to reduce the I/O load.The I/O forwarding strategy was adjusted in the forwarding service layer to balance the I/O load.In the back-end storage layer,the high availability function of the system was adjusted to solve the conflict between the big data I/O access mode and the original high availability functions.After optimized design and improvement,the parallel storage system can better adapt to the heterogeneous multi-core architecture,making some applications get more than 10 times of I/O performance improvement.

Key words: big data, high performance computing, Sunway TaihuLight, heterogeneous, parallel storage

中图分类号: 

No Suggested Reading articles found!