Hadoop分布式文件系统的模型分析

doi:10.3969/j.issn.1000-0801.2010.12.025

Abstract

Abstract:

Hadoop distributed file system（HDFS）is inspired by the Google file system paper.It gets great attention and is applied widely.In this paper，from a theoretical perspective of system design，we study user requirements and modeling method of distributed file system，and create a model for analyzing the architecture characteristics of HDFS.Through the comparison of HDFS and traditional distributed file system，several advantages of HDFS（e.g.high scalability and high availability）are identified.This paper is helpful to research the design principles of HDFS more deeply，and provides some advices for design and implementation of Cloud Computing-oriented distributed file system.

Key words: hadoop distributed file system, system modeling, cloud computing

Feng Wang,Baohua Lei. Modeling and Analysis of Hadoop Distributed File System[J]. Telecommunications Science, 2010, 26(12): 95-99.

Figures/Tables 5

		NFS	HDFS
系统透明性	访问透明	提供与传统的本地文件系统访问类似的接口，实现访问透明	未提供与传统的POSIX完全兼容的接口，访问透明性不高
	位置透明	远程文件系统的目录直接挂接在客户端文件系统上，支持位置透明	通过文件文本名字和统一资源标识符访问文件，支持位置透明
	移动透明	没有统一的名字空间，文件不能在服务器侧移动，不支持移动透明	Namenode节点管理和维护统一的文件名字空间，实现移动透明
	性能透明	提供服务器侧和客户端的缓存机制，文件访问性能较高	支持多个文件数据块的并行访问，文件访问性能高
	扩展透明	需要用户手工设置远程文件访问的挂接位置，扩展不够透明	根据负载规模增删Datanode，无需用户参与，实现扩展透明
文件并发更新		使用网络文件锁管理机制进行并发控制，但会受NFS协议的影响	不支持文件的并发更新，采用租期机制限定对文件数据块的访问
文件内容复制		读文件可被复制到多台服务器上，但是不支持可写文件的复制	文件块被自动复制为多份拷贝并被分散存放到多个Datanode上
文件数据一致		采用单一拷贝机制，定时审验客户端缓存中的数据信息	采用简单的一致性协议，通过原子化操作对文件进行修改
文件服务容错		采用NFS无状态协议，异常处理和容错功能完全由客户端实现	在Datanode、文件、数据块等多个层次上提供冗余，实现容错
文件服务兼容		支持具有多种操作系统和硬件平台的客户端和服务器	支持运行Linux操作系统的标准PC服务器作为客户端和服务器
文件服务安全		采用安全的远程过程调用机制保证用户权限识别及数据安全传输	采用基于安全套接字的网络安全机制实现文件的安全访问

需求	描述
访问透明	用户程序不必考虑文件的分布问题，能够通过完全相同的编程接口对本地文件和远程文件进行访问和操作
位置透明	用户程序能够看到统一的文件名字空间，对文件进行访问的路径名与其在远程服务器中的存放位置无关
移动透明	用户程序无需关心文件的存放位置发生移动所产生的影响，这一需求的满足与位置透明性的实现相关
性能透明	用户程序始终能够获得高效的文件访问能力，即使远程服务器上的文件服务负载在一定范围内发生了变化
扩展透明	用户程序导致的文件服务需求的增大或者网络规模的扩张等问题，能够通过系统的自动扩展得到有效的解决

1	Hadoop community . Hadoop distributed file system，， 2010
2	George C ， Jean D ， Tim K . Distributed systems：concepts and design（3rd Edition）. Addison-Wesley Publishers Limited， 2000
3	Russel S ， David G ， Steve K ，et al. Design and implementation of the Sun network file system. Artech House， 1988
4	Wang F ， Qiu J ， Yang J ，et al. Hadoop high availability through metadata replication.In： Proceeding of the First International Workshop on Cloud Data Management， Hong Kong，China， November 2009
5	Gluster community . Gluster file system，， 2010

[1]	Xue HAN. Damage toughness assessment method of power backbone communication network based on power big data [J]. Telecommunications Science, 2023, 39(5): 136-143.
[2]	Jiangxing WU. Revolution of the development paradigm of network technology system—network of networks [J]. Telecommunications Science, 2022, 38(6): 3-12.
[3]	Wenjuan XING, Bo LEI, Qianying ZHAO. Development status and trend prospect of computing power infrastructure [J]. Telecommunications Science, 2022, 38(6): 51-61.
[4]	Pingke DENG, Tongxu ZHANG, Nanxiang SHI, Tong ZHANG, Tianzhu SHAO, Shaowen ZHENG. Computing satellite networks—the novel development of computing-empowered space-air-ground integrated networks [J]. Telecommunications Science, 2022, 38(6): 71-81.
[5]	Kun MA, Lingyu XU, Xiaoping SHEN, Zhicheng GONG, Jianping LAN, Shuangxi CHEN, Jun QIAN. Virtual machine scheduling model based on Shapley value modified genetic algorithm in cloud computing [J]. Telecommunications Science, 2022, 38(12): 1-10.
[6]	Zhiyong LIU, Zhongjiang HE, Yilong RUAN, Junfeng SHAN, Chao ZHANG. Big data security features and operation practices [J]. Telecommunications Science, 2021, 37(5): 160-169.
[7]	Jian XU,Li JIN. A quantifiable cloud computing platform security evaluation model [J]. Telecommunications Science, 2020, 36(7): 163-167.
[8]	Sining LUO,Hualong WANG,Hongyu LI,Wei PENG. Improved ant colony algorithm based cloud computing user task scheduling algorithm [J]. Telecommunications Science, 2020, 36(2): 95-100.
[9]	Xiaorong XU,Minghang SUN,Linhui SHEN,Yingbiao YAO,Wei FENG. Key technologies and research progresses in cognitive SWIPT [J]. Telecommunications Science, 2020, 36(10): 87-101.
[10]	Bo LEI, Zengyi LIU, Xuliang WANG, Mingchuan ·ANG, ·unqing CHEN. Computing network:a new multi-access edge computing [J]. Telecommunications Science, 2019, 35(9): 44-51.
[11]	Yongmei GAO,Guanjie CHENG. Data-intensive service deployment based on edge computing [J]. Telecommunications Science, 2019, 35(7): 87-99.
[12]	Zelin LU,Qiangzhi LI. Evolutionary logic and operational mechanism of E-commerce platform [J]. Telecommunications Science, 2019, 35(7): 152-158.
[13]	Zhilan HUANG,Yongbing FAN,Ning FAN,Nan CHEN,Linze WU,Baohong LIN. Solution and key technology of integrating vFW in cloud platform [J]. Telecommunications Science, 2019, 35(5): 140-148.
[14]	Youkang ZHU,Guangxue YUE,Xiaohui YANG,Jiansheng LIU. A survey on edge computing offloading [J]. Telecommunications Science, 2019, 35(4): 74-94.
[15]	Guowei WU,Ning FAN,Laifu WANG,Shuai WANG,Jun SHEN,Huamin JIN. Accelerated algorithm in attribute-based encryption over cloud environment [J]. Telecommunications Science, 2019, 35(11): 101-107.

Modeling and Analysis of Hadoop Distributed File System

RichHTML

PDF下载

Knowledge

Cited

Abstract

Cite this article

share this article

Figures/Tables 5

References 5

Related Articles 15

Metrics

Recommended 0