基于动态动作覆盖的深度强化学习新闻推荐

doi:10.11959/j.issn.2096-0271.2023069

摘要/Abstract

摘要： 新闻推荐系统对于新媒体新闻传播有着重要作用，本文提出了一种以深度强化学习为基础的旨在结合神经网络的表征能力和强化学习的策略选择能力以提升新闻推荐效果的推荐系统。本文使用动态动作掩码加强对用户短期兴趣的判断能力，使用优化缓存机制提升经验缓存的使用效率，通过区域遮蔽性质的奖励设计加快模型训练，提高推荐系统在新闻推荐领域的表现。实验表明，本文所提模型在新闻数据集上推荐准确率与现今主流的神经网络推荐方法相当，且在排序性能上优于当前先进的推荐算法。

关键词: 新闻推荐, 强化学习, 动态掩码, 优势缓存, 内在奖励

Abstract: News recommendation system plays an important role in news dissemination of new media. This paper proposes a recommendation system based on deep reinforcement learning, which aims to combine the representation ability of neural network and the strategy selection ability of reinforcement learning to improve the effect of news recommendation. This paper uses dynamic action masks to enhance the ability to judge users' short-term interests, uses the optimization cache mechanism to improve the efficiency of experience cache use, and accelerates model training through the reward design of regional masking nature to improve the performance of the recommendation system in the field of news recommendation. Experimental results show that the accuracy of the proposed model in news data sets is comparable to the current mainstream neural network recommendation methods, and its ranking performance is more than others.

Key words: , news recommendation, reinforcement learning, dynamic mask, advantage cache, internal reward

中图分类号:

TP311.5

董相宏, 安俊秀. 基于动态动作覆盖的深度强化学习新闻推荐[J]. 大数据, doi: 10.11959/j.issn.2096-0271.2023069.

DONG Xianghong, AN Junxiu. Deep reinforcement learning news recommendation based on dynamic action coverage[J]. Big Data Research, doi: 10.11959/j.issn.2096-0271.2023069.

参考文献

[1]LIN CHEN, XIE RUNQUAN, GUAN XINJUN, et al. Personalized news recommendation via implicit social experts[J]. Information Sciences, 2014, 254: 1-18.

[2]WU CHUHAN, WU FANGZHAO, QI TAO, et al. Is News Recommendation a Sequential Recommendation Task? [C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2022: 2382-2386.

[3]HIDASI B, KARATZOGLOU A, BALTRUNAS L, et al. Session-based recommendations with recurrent neural networks[J]. arXiv preprint arXiv:1511.06939, 2015.

[4]LIN GUANYU, GAO CHEN, LI YIFENG, et al. Dual contrastive network for sequential recommendation[C]//Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 2022: 2686-2691.

[5]Zhao, Q. RESETBERT4Rec: A Pre-training Model Integrating Time And User Historical Behavior for Sequential Recommendation[C]//Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval.2022. DOI:10.1145/3477495.3532054

[6]IJNTEMA W, GOOSSEN F, FRASINCAR F, et al. Ontology-based news recommendation[C]//Proceedings of the 2010 EDBT/ICDT Workshops. 2010: 1-6.

[7]OKURA S, TAGAMI Y, ONO S, et al. Embedding-based news recommendation for millions of users[C]//Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2017: 1933-1942.

[8]KARVELIS P, GAVRILIS D, GEORGOULAS G, et al. Topic recommendation using Doc2Vec[C]//2018 International Joint Conference on Neural Networks (IJCNN). IEEE, 2018: 1-6.

[9]CASELLES-DUPRÉ H, LESAINT F, ROYO-LETELIER J. Word2vec applied to recommendation: Hyperparameters matter[C]//Proceedings of the 12th ACM Conference on Recommender Systems. 2018: 352-356.

[10]ZHANG JIADONG, CHOW CHIYIN, LI YANHUA. iGeoRec: A personalized and efficient geographical location recommendation framework[J]. IEEE Transactions on Services Computing, 2014, 8(5): 701-714.

[11] KARATZOGLOU A, HIDASI B. Deep learning for recommender systems[C]//Proceedings of the eleventh ACM conference on recommender systems. 2017: 396-397.

[12]DEVLIN S M, KUDENKO D. Dynamic potential-based reward shaping[C]//Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems. IFAAMAS, 2012: 433-440.

[13]LI L, CHU W, LANGFORD J, et al. A contextual-bandit approach to personalized news article recommendation[C]//Proceedings of the 19th international conference on World wide web. 2010: 661-670.

[14]ZHENG G, ZHANG F, ZHENG Z, et al. DRN: A deep reinforcement learning framework for news recommendation[C]//Proceedings of the 2018 world wide web conference. 2018: 167-176.

[15]YUE Y, JOACHIMS T. Interactively optimizing information retrieval systems as a dueling bandits problem[C]//Proceedings of the 26th Annual International Conference on Machine Learning. 2009: 1201-1208.

[16]XIAOCONG C, LINA Y,et al.Locality-Sensitive State-Guided Experience Replay Optimization for Sparse Rewards in Online Recommendation[C]// Proceedings of the 45th international ACM SIGIR conference on research and development in information retrieval. 2022: 1316–1325.

[17]刘全,翟建伟,章宗长,等.深度强化学习综述[J].计算机学报,2018,41(01):1-27.

LIU Quan, ZHAI Jian-Wei, ZHANG Zong-Zhang, ZHONG Shan, ZHOU Qian, ZHANG Peng, XU Jin, A Survey on Deep Reinforcement Learning, 2017,Vol.40

[18]YUYAN Z, XIAYAO S, YONG L. A novel movie recommendation system based on deep reinforcement learning with prioritized experience replay[C]//2019 IEEE 19th International Conference on Communication Technology (ICCT). IEEE, 2019: 1496-1500.

[19]LI YUQI, CHEN WEIZHENG, YAN HONGFEI. Learning graph-based embedding for time-aware product recommendation[C]//Proceedings of the 2017 ACM on Conference on Information and Knowledge Management. 2017: 2163-2166.

[20]LIU Q, ZENG Y, MOKHOSI R, et al. STAMP: short-term attention/memory priority model for session-based recommendation[C]//Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018: 1831-1839.

[21]蔡丽娇,秦进,陈双.远离旧区域和避免回路的强化探索策略[J/OL].计算机工程:1-11[2023-01-04].DOI:10.19678/j.issn.1000-3428.0065296.

CAI Lijia, QIN Jin, CHEN Shuang. Reinforcement Exploration Strategy to Keep Away from an Area and Avoid a Loop[J/OL]. Computer Engineering:1-11.

[22]ZHAO X, ZHANG L, DING Z, et al. Recommendations with negative feedback via pairwise deep reinforcement learning[C]//Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018: 1040-1048.

[23]GONG SHANSAN, KENNY Q ZHU. Positive, Negative and Neutral: Modeling Implicit Feedback in Session-based News Recommendation[J]. arXiv preprint arXiv:2205.06058, 2022.

[24]刘树栋,张可,陈旭.基于多维度兴趣注意力和用户长短期偏好的新闻推荐[J].中文信息学报,2022,36(09):102-111.

LIU Shudong, ZHANG Ke, CHEN Xu. Multi-Dimensional Interest-Attention-based News Recommendation with Long and Short-term User Preferences[J].Journal of Chinese Information Processing,2022,36(09):102-111.

[25]陈希亮,曹雷,李晨溪等.基于重抽样优选缓存经验回放机制的深度强化学习方法[J].控制与决策,2018,33(04).

CHEN Xi-liangy, CAO Lei, LI Chen-xi, et.al. Deep reinforcement learning via good choice resampling experience replay memory[J]. Control and Decision,2018,33(04).

[26]KOREN Y. Factorization meets the neighborhood: a multifaceted collaborative filtering model[C]//Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. 2008: 426-434.

[27]HE XIANGNAN, LIAO LIZI, ZHANG HANWANG, et al. Neural collaborative filtering[C]//Proceedings of the 26th international conference on world wide web. 2017: 173-182.