Abstract:

In the current HDFS (Hadoop Distributed File System) key management system, the encryption zone keys are all loaded into the memory during startup of key service.With the increase of the key resource, the occupied memory space also grows, bringing the bottleneck of memory space and key indexing.There are three challenges induced: how to organize cached data and efficiently handle queries with missed keys, how to adjust key resources in the cache, and how to accurately predict the use of keys.In order to achieve fine-grained and efficient caching and improve the efficiency of key use, key caching optimization was considered from three aspects: key index data structure, key replacement algorithm, and key prefetching strategy.An architecture of key cache replacement module was designed, and then a key replacement algorithm based on the query frequency was set.Specifically, from the perspective of heat computing and key replacement, the potential influencing factors affecting the popularity of key cache were analyzed which considered the file system and user of key management system.Besides, the basic model of key usage popularity was constructed.The hash table and minheap linked list was combined to maintain the heat of the key in use, and the elimination algorithm was set based on heat identification.The key in the cache was dynamically updated, and key usage was adjusted by the time controller, so as to realize key replacement according to the key heat.For key prefetching, key usage rules were obtained through log mining and periodical usage analyzing of key provisioning policies, which considered business processes and the time period dimension of user accessing.Experimental results show that the proposed key replacement algorithm can effectively improve the hit rate of cache queries, reduce memory usage, and ameliorate the impact of key file I/O interaction on query performance.

Key words: key management, cache management, usage frequency calculation, key replacement algorithm