基于Kademlia的云存储系统数据冗余方案研究

doi:10.3969/j.issn.1000-0801.2011.02.012

摘要/Abstract

摘要：

云存储是分布式存储技术与虚拟化技术结合的产物，是分布式存储技术的最新发展。云存储意味着存储可以作为一种服务，通过互联网提供给用户。相对于当前各种C/S计算模式的GFS、HDFS、Sector 等云存储文件系统，提出了一种基于对等计算模式的云存储文件系统通用模型，并采用Kademlia算法构建了原型系统MingCloud。主要进行了存储系统冗余方案的选择与设计，实验中采用Cauchy Codes作为编码算法，分别从K 桶大小对系统可用性影响、文件分块数量对系统可用性的影响、相同的冗余度下系统可用性比较等方面对系统进行了评估。与完全副本冗余模式相比，纠删码冗余模式能获得较理想的系统可用性，更适合应用于提议的对等云存储系统。

关键词: 云计算, 云存储, MingCloud, Kademlia, 冗余, 纠删码

Abstract:

As the latest development of the distributed storage technology, cloud storage is the product of the integration of distributed storage and virtualization technologies. Cloud storage is a method that allows you to use storage facilities available on the Internet. Relative to the current C/S computing model cloud storage file systems, including GFS, HDFS, Sector, we proposed a general model of peer-to-peer based Cloud Storage file system, and constructed a prototype system named MingCloud based on Kademlia algorithm. Focused on redundancy scheme selection and design of the storage system, our experiment used the Cauchy Codes as coding algorithm to assess the system from many aspects, including the impact of K bucket size on system availability, the impact of File block number on system availability, comparison of system availability under same redundancy. Compared with the redundant mode of complete copy, the Erasure Code model provided more satisfactory system availability, and more suitable to apply to the P2P cloud storage system.

Key words: cloud computing, cloud storage, MingCloud, Kademlia, redundancy, erasure code

吴吉义,章剑林,傅建庆,平玲娣. 基于Kademlia的云存储系统数据冗余方案研究[J]. 电信科学, 2011, 27(2): 68-73.

Jiyi Wu,Jianlin Zhang,Jianqing Fu,Lingdi Ping. Study on Data Redundancy Scheme in Kademlia Cloud Storage System[J]. Telecommunications Science, 2011, 27(2): 68-73.

图/表 6

图1

表1

表2

图2

图3

图4

参考文献 31

1	吴吉义，平玲娣，潘雪增等．云计算：从概念到平台．电信科学， 2009,25（12）:23～30
2	Sanjay Ghemawat , Howard Gobioff , Shun-Tak Leung . The Google file system.In: Proc of the 19th ACM SOSP, New York, 2003
3	Kelly Sims . IBM introduces ready-to-use cloud computing collaboration services get clients started with cloud computing, , 2010
4	Dhruba Borthaku . The hadoop distributed file system: architecture and design, , 2010
5	Hbase Development Team. Hbase: bigtable-like structured storage for hadoop hdfs, , 2010
6	Amazon. Amazon simple storage service（S3）, , 2010
7	Kosmos file system（KFS）, , 2010
8	Gu Yunhong , Robert L Grossman . Sector and sphere:the design and implementation of a high-performance data cloud. Philosophical Transactions of the Royal Society, A(2009)367:2429～2445
9	Robert L Grossman , Gu Yunhong . Data mining using high performance data clouds: experimental studies using sector and sphere.In: Proc of the 14th ACM SIGKDD, 2008
10	吴吉义，傅建庆，张明西等．云数据管理研究综述．电信科学， 2010,26（5）:34～41
11	James Broberg , Rajkumar Buyya , Zahir Tari . Creating aˊcloud storageˊmashup for high performance, low cost content delivery.In: Proc.of ICSOC 2008, LNCS 5472, 2009
12	James Broberg , Zahir Tari . MetaCDN:harnessing storage clouds for high performance content delivery.In: Proc of ICSOC 2008,LNCS 5364, 2008
13	Kevin D , Bowers , Ari Juels , Alina Oprea . Hall: a high-availability and integrity layer for cloud storage. Cryptology ePrint Archive,Report 2008/489, , 2010
14	David Tarrant , Tim Brody Leslie Carr . From the desktop to the cloud:leveraging hybrid storage architectures in your repository, , 2009
15	Li Jin . Erasure resilient codes in peer-to-peer storage cloud.In: Proc of ICASSP 2006
16	Corsair, , 2010
17	Liu Likun , Wu Yongwei , Yang Guangwen , et al. Zettads:a light-weight distributed storage system for cluster.In: Proc of the 3rd ChinaGrid Annual Conference, 2008
18	武永卫，黄小猛云存储．中国计算机学会通讯， 2009,5（6）:44～52
19	侯孟书，王晓斌，卢显良等．一种新的动态副本管理机制．计算机科学， 2006,33（9）:50～51
20	Weatherspoon H , Kubiatowicz J . Erasure coding vs replication:quantitative comparison.In: Proc of the 1st Intˊl Workshop Peer-to-Peer Systems, 2002
21	Bhagwan R , Savage S , Voelker G . Replication strategies for highly available peer-to-peer storage systems. Technical Report,CS2002-0726,UCSD, 2002
22	Lin W , Chiu D , Lee Y . Erasure code replication revisited.In: Proc of the 4th Intˊl Conf on Peer-to-Peer Computing, 2004
23	Rodrigues R , Liskov B . High availability in DHTs:erasure coding vs replication.In: Proc of the 4th Intˊl Workshop on Peer-to-Peer Systems, 2005
24	田敬，代亚非． P2P 持久存储研究．软件学报， 2007,18（6）:1379～1399
25	Reed I S , Solomon G . Polynomial codes over certain finite fields. Journal of the Society for Industrial and Applied Mathematics, 1960（8）:300～304
26	James S P . Optimizing cauchy reed-solomon codes for fault-tolerant storage applications. Technical Report CS-OS-569 Deparment of CoScience University of Tenessee, Dec 2005
27	Luby M , Mitzenmacher M , Shokrollahi A , et al. Efficient erasure correcting codes. IEEE Transactions on Information Theory, 2001,47（2）:569～584
28	肖哲锋．基于Erasure Code的高可用分布式存储系统的设计与实现．湖南：国防科学技术大学， 2006
29	李勤． P2P内容分发系统中编码解码算法设计与实现．四川：电子科技大学， 2008
30	John W B , Michael L , Michael M . A digital fountain approach to reliable distribution of bulk data. ACM SIGCOMM Computer Communication Review, 1998,28（4）:56～67
31	Ranjita , Bhagwan Kiran , Kiran Tati , et al. Total recall:system support for automated availability management, , 2010

名称	优点	缺点
流式编码	主要适用于流媒体的信息传输	存储节点能拥有完整的文件
Shamir门限方案	应用于密钥处理	数据处理能力很弱
Vandermonde	具有纠错功能，能恢复出大小一定的源文件	较难找到合适的生成矩阵来构造 Erasure Code，因此对要处理的文件大小要求严格
Cauchy Codes	通过增加冗余包来抵御数据包的丢失，处理数据量大并且便捷	处理速度不是当前最快的
Tornado Codes	简单，处理速度快	公开的算法资料比较少

参数名	含义
p	节点可用性
r	冗余度
n	采用完全副本冗余时的文件分块
m	采用纠删码冗余时的基本文件分块
k	Kademlia协议中的K桶大小

[1]	韩雪. 基于电力大数据的电力骨干通信网络毁伤韧性评估方法[J]. 电信科学, 2023, 39(5): 136-143.
[2]	李正杰, 刘顺兰, 张旭. 一种基于分段CRC码级联Hash极化码的设计[J]. 电信科学, 2022, 38(7): 96-105.
[3]	邢文娟, 雷波, 赵倩颖. 算力基础设施发展现状与趋势展望[J]. 电信科学, 2022, 38(6): 51-61.
[4]	邓平科, 张同须, 施南翔, 张童, 邵天竺, 郑韶雯. 星算网络——空天地一体化算力融合网络新发展[J]. 电信科学, 2022, 38(6): 71-81.
[5]	邬江兴. 论网络技术体制发展范式的变革——网络之网络[J]. 电信科学, 2022, 38(6): 3-12.
[6]	时书锋, 于游洋. 多接入、无线、有线融合架构演进及流量调度机制[J]. 电信科学, 2022, 38(4): 146-155.
[7]	马焜, 徐玲玉, 沈晓萍, 龚志城, 蓝建平, 陈双喜, 钱钧. 云计算中基于Shapley值改进遗传算法的虚拟机调度模型[J]. 电信科学, 2022, 38(12): 1-10.
[8]	刘志勇, 何忠江, 阮宜龙, 单俊峰, 张超. 大数据安全特征与运营实践[J]. 电信科学, 2021, 37(5): 160-169.
[9]	王燕, 刘顺兰, 奚珍珍, 包建荣. 一种基于增强奇偶校验码级联极化码的新型编译码方法[J]. 电信科学, 2021, 37(3): 125-132.
[10]	许剑,靳莉. 一种可量化的云计算平台安全评估模型[J]. 电信科学, 2020, 36(7): 163-167.
[11]	陈利跃,孙歆,成天晟,吴春明,陈双喜. 面向Web隐藏后门技术的防御[J]. 电信科学, 2020, 36(5): 39-46.
[12]	秦俊宁,韩嘉佳,周升,吴春明,陈双喜,赵若琰,张江瑜. 基于异构冗余架构的拟态防御建模技术[J]. 电信科学, 2020, 36(5): 31-38.
[13]	罗斯宁,王化龙,李弘宇,彭蔚. 基于改进蚁群算法的云计算用户任务调度算法[J]. 电信科学, 2020, 36(2): 95-100.
[14]	雷波, 刘增义, 王旭亮, 杨明川, 陈运清. 基于云、网、边融合的边缘计算新方案：算力网络[J]. 电信科学, 2019, 35(9): 44-51.
[15]	高永梅,程冠杰. 基于边缘计算的数据密集型服务部署[J]. 电信科学, 2019, 35(7): 87-99.