知识增强策略引导的交互式强化推荐系统

doi:10.11959/j.issn.2096-0271.2022033

摘要/Abstract

摘要： 推荐系统是解决社会媒体信息过载问题的重要手段。为了解决传统推荐系统无法优化用户长期体验的问题，研究者们提出了交互式推荐系统，并尝试使用深度强化学习优化推荐策略。但是，强化推荐算法面临着反馈稀疏、从零学习损害用户体验、物品空间大等问题。为了解决上述问题，本文提出一种改进的知识增强策略引导的交互式强化推荐模型KGP-DQN。该模型构建行为知识图谱表示模块，将用户历史行为和知识图谱结合，解决反馈稀疏问题；构建策略初始化模块，根据用户历史行为为强化推荐系统提供初始化策略，解决从零学习损害用户体验问题；构建候选集筛选模块，根据行为知识图谱上的物品表示进行动态聚类从而减少物品空间，解决动作空间大的问题。本文在三个真实数据集上进行了实验，实验结果表明，KGP-DQN方法可以快速有效地对强化推荐系统进行训练，并取得不错的推荐效果。

Abstract: The recommendation system is an important means to solve the problem of information overload in social media. In order to solve the problem that traditional recommendation systems cannot optimize the long-term user experience, researchers have proposed the interactive recommendation system and tried to use deep reinforcement learning to optimize strategy of recommendation. However, the reinforcement recommendation algorithm faces problems such as sparse feedback, learning from zero which damages the user experience, and large item space. In order to solve the above problems, this paper proposes an improved interactive reinforcement recommendation model KGP-DQN. The model constructs a behavioral knowledge graph representation module, which combines user historical behavior and knowledge graph to solve the problem of sparse feedback; constructs a strategy initialization module to provide an initialization strategy for the reinforcement recommendation system based on user historical behaviors to solve the problem of learning from zero; constructs the candidate select module which creates candidate by dynamic clustering based on the item representation on the behavioral knowledge graph to solve the problem of large action space. This paper conducts experiments on three real-world datasets. The experimental results show that the KGP-DQN method can quickly and effectively train the reinforcement recommendation system and achieve good recommendation results.

Key words: interactive recommendation system, deep reinforcement learning, knowledge graph, policy initialization, candidate select

张宇奇, 黄晓雯, 桑基韬. 知识增强策略引导的交互式强化推荐系统[J]. 大数据, doi: 10.11959/j.issn.2096-0271.2022033.

Zhang yuqi, huang xiaowen, sang jitao. Knowledge-enhanced Policy-guided Interactive Reinforcement Recommendation System[J]. Big Data Research, doi: 10.11959/j.issn.2096-0271.2022033.

参考文献

[1] Wang H, Wu Q, Wang H. Factorization bandits for interactive recommendation[C]//Thirty-First AAAI Conference on Artificial Intelligence. 2017.
[2] Mahmood T , Ricci F . Learning and adaptivity in interactive recommender systems[C]// Proceedings of the 9th International Conference on Electronic Commerce: The Wireless World of Electronic Commerce, 2007, University of Minnesota, Minneapolis, MN, USA, August 19-22, 2007. DBLP, 2007.
[3] Zhao X, Zhang L, Ding Z, et al. Recommendations with negative feedback via pairwise deep reinforcement learning[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018: 1040-1048.
[4] Zhou S, Dai X, Chen H, et al. Interactive recommender system via knowledge graph-enhanced reinforcement learning[C]//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020: 179-188.
[5] Wang P, Fan Y, Xia L, et al. KERL: A knowledge-guided reinforcement learning model for sequential recommendation[C]//Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2020: 209-218.
[6] Goldberg D, Nichols D, Oki B M, et al. Using collaborative filtering to weave an information tapestry[J]. Communications of the ACM, 1992, 35(12): 61-70.
[7] Rendle S. Factorization machines[C]//2010 IEEE International conference on data mining. IEEE, 2010: 995-1000.
[8] Hidasi B, Karatzoglou A, Baltrunas L, et al. Session-based recommendations with recurrent neural networks[J]. arXiv preprint arXiv:1511.06939, 2015.
[9] Zhou G, Zhu X, Song C, et al. Deep interest network for click-through rate prediction[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018: 1059-1068.
[10] Chen H, Dai X, Cai H, et al. Large-scale interactive recommendation with tree-structured policy gradient[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2019, 33(01): 3312-3320..
[11] Zheng G, Zhang F, Zheng Z, et al. DRN: A deep reinforcement learning framework for news recommendation[C]//Proceedings of the 2018 World Wide Web Conference. 2018: 167-176.
[12] Zou L, Xia L, Ding Z, et al. Reinforcement learning to optimize long-term user engagement in recommender systems[C]//Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2019: 2810-2818.
[13] Zhao X, Zhang L, Ding Z, et al. Recommendations with negative feedback via pairwise deep reinforcement learning[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018: 1040-1048.
[14] Dulac-Arnold G, Evans R, van Hasselt H, et al. Deep reinforcement learning in large discrete action spaces[J]. arXiv preprint arXiv:1512.07679, 2015.
[15] Xian Y, Fu Z, Muthukrishnan S, et al. Reinforcement knowledge graph reasoning for explainable recommendation[C]//Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval. 2019: 285-294.
[16] Van Hasselt H, Guez A, Silver D. Deep reinforcement learning with double q-learning[C]//Proceedings of the AAAI conference on artificial intelligence. 2016, 30(1).
[17] Koren Y , Bell R , Volinsky C . Matrix factorization techniques for recommender systems. IEEE, Computer Journal, 42(8), 30-37[J]. Computer, 2009, 42(8):30-37.
[18] Davies D L , Bouldin D W . A Cluster Separation Measure[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1979, PAMI-1(2):224-227.