基于关联规则的气象服务智能推荐

doi:10.11959/j.issn.2096-0271.2018020

摘要/Abstract

摘要：

针对气象公众服务存在的专业化程度不高、针对性不强等问题，提出了一种使用关联规则向用户推荐气象服务产品的方法。将用户访问的Web日志数据经过清洗、转换后进行关联规则挖掘，同时设计了一种基于气象产品浏览总次数计算产品相似度的算法，对挖掘出来的关联规则进行筛选。共进行了5组实验，并对用户浏览产品的行为进行了简单预测。实验结果表明，挖掘出的规则在降水天气和雾霾天气中的正确率比较高，结合气象服务产品的相似度能过滤掉10%的无效规则，从而提升规则的正确性。

Abstract:

To overcome the problem in which public meteorological services are rarely optimized for personalization,a recommendation method incorporating improved association rules and collaborative filtering was presented.First,FPGrowth algorithm was applied on pre-processed Web log data to generate association rules.Meanwhile,a customized collaborative filtering algorithm was used to calculate pairwise similarities between meteorological products based on users' browsing records.A total of five experiments were conducted.The experimental results show that the rules are relatively accurate for precipitation and haze weather,because pairwise similarities between meteorological products which are calculated by the proposed algorithm could reduce the total number of invalid or irrelevant association rules by 10%.

Key words: intelligent recommendation, association rule, Web log, public meteorological service, parallel computing, similarity, collaborative filtering

中图分类号:

TP183

赵文芳, 刘亚楠, 余东昌. 基于关联规则的气象服务智能推荐[J]. 大数据, 2018, 4(2): 72-85.

Wenfang ZHAO, Yanan LIU, Dongchang YU. Intelligent recommendation of meteorological service based on association rules[J]. Big Data Research, 2018, 4(2): 72-85.

图/表 14

图1

表1

表2

表3

图2

表4

图3

表5

表6

图4

图5

表7

图6

图7

参考文献 22

[1]	纪淑娟, 王理, 梁永全 ,等. 基于神经网络的用户视频评分自动获取方法[J]. 计算机科学, 2015,42(11): 96-100.
	JI S J , WANG L , LIANG Y Q ,et al. Neuralnetwork-based method for automatic acquisition of user’s video rating[J]. Computer Science, 2015,42(11): 96-100.
[2]	王涛, 覃锡忠, 贾振红 ,等. 基于相似度和信任度的关联规则微博好友推荐[J]. 计算机应用, 2016,36(8): 2262-2267.
	WANG T , QIN X Z , JIA Z H ,et al. Association rules recommendation of microblog friend based on similarity and trust[J]. Journal of Computer Applications, 2016,36(8): 2262-2267.
[3]	LIU J G , ZHOU T , WANG B H . Personalized recommender systems:a survey of the state-of-the-art[J]. Progress in Natural Science, 2009,19(1): 1-15.
[4]	LIN W , ALVAREZ S A , RUIZ C . Efficient adaptive-support association rule mining for recommendation systems[J]. Data Mining and Knowledge Discovery, 2002,6(1): 83-105.
[5]	李杰, 徐勇, 王云峰 ,等. 面向个性化推荐的强关联规则挖掘[J]. 系统工程理论与实践, 2009,29(8): 144-152.
	LI J , XU Y , WANG Y F ,et al. Strongest association rules mining for personalized recommendation[J]. System EngineeringTheory ＆Practice, 2009,29(8): 144-152.
[6]	易芝, 汪琳琳, 王练 . 基于关联规则相关性分析的Web 个性化推荐研究[J]. 重庆邮电大学学报(自然科学版), 2007,19(2): 234-237.
	YI Z , WANG L L , WANG L . Research on Web personalized recommendation based on correlation analysis of association rule[J]. Journal of Chongqing University of Posts and Communications (Natural Science), 2007,19(2): 234-237.
[7]	鲍玉斌, 王大玲, 于戈 . 关联规则和聚类分析在个性化推荐中的应用[J]. 东北大学学报(自然科学版), 2003,24(12): 1149-1188.
	BAO Y B , WANG D L , YU G . Application of association rules and clustering analysis to personalized recommendation[J]. Journal of Northeastern University (Natural Science), 2003,24(12): 1149-1188.
[8]	张忠林, 田苗凤, 刘宗成 . 大数据环境下关联规则并行分层挖掘算法研究[J]. 计算机科学, 2016,43(1): 286-289.
	ZHANG Z L , TIAN M F , LIU Z C . Parallel hierarchical association rule mining in big data environment[J]. Computer Science, 2016,43(1): 286-289.
[9]	周诗慧, 殷建 . Hadoop 平台下的并行Web 日志挖掘算法[J]. 计算机工程, 2013,39(6): 43-46.
	ZHOU S H , YIN J . Parallel web log mining algorithm in Hadoop platform[J]. Computer Engineering, 2013,39(6): 43-46.
[10]	刘帅, 杨英杰, 常德显 ,等. 改进的模糊关联规则及其挖掘算法[J]. 计算机工程与设计, 2015,36(4): 942-947.
	LIU S , YANG Y J , CHANG D X ,et al. Improved fuzzy association rule and its mining algorithm[J]. Computer Engineer and Design, 2015,36(4): 942-947.
[11]	程广, 王晓峰 . 基于MapReduce的并行关联规则增量更新算法[J]. 计算机工程, 2016,42(2): 21-26.
	CHENG G , WANG X F . Incremental updating algorithm of parallel association rule based on MapReduce[J]. Computer Engineer, 2016,42(2): 21-26.
[12]	黄晓雯, 严明, 桑基韬 ,等. 基于关联规则挖掘的跨网络知识关联及协同应用[J]. 计算机科学, 2016,43(7): 51-57.
	HUANG X W , YAN M , SANG J T ,et al. Association rules mining based crossnetwork knowledge association and collaborative application[J]. Computer Science, 2016,43(7): 51-57.
[13]	陈平华, 陈传瑜, 洪英 . 一种结合关联规则的协同过滤推荐算法[J]. 小型微型计算机系统, 2016,37(2): 287-293.
	CHEN P H , CHEN C Y , HONG Y . Incorporating association rules for collaborative filtering recommendation algorithm[J]. Journal of Chinese Computer Systems, 2016,37(2): 287-293.
[14]	胡波, 黄宁, 仵伟强 . 基于业务路径和频度矩阵的关联规则挖掘算法[J]. 计算机科学, 2016,43(12): 146-154.
	HU B , HUANG N , WU W Q . Algorithm for mining association rules based on application paths and frequency matrix[J]. Computer Science, 2016,43(12): 146-154.
[15]	石岩, 邓敏, 刘启亮 ,等. 海陆气候事件关联规则挖掘方法[J]. 地理信息科学, 2014,16(2): 182-191.
	SHI Y , DENG M , LIU Q L ,et al. Discovering sequential association rules between single ocean climate index and land abnormal climate events[J]. Journal of Geo-information Science, 2014,16(2): 182-191.
[16]	刘娟, 宋安军 . 改进FP-growth 算法在气象预报中的应用[J]. 计算机系统应用, 2016,25(10): 199-205.
	LIU J , SONG A J . Application of an improve FP-growth algorithm in meteorological forecast[J]. Application of Computer System, 2016,25(10): 199-205.
[17]	BINZENH?FER A , TUTSCHKU K , GRABEN B A D ,et al. A P2P-based framework for distributed network management[M]. Berlin: Springer Berlin Heidelberg, 2005.
[18]	CHEVUL S , BINZENHFER A , SCHMID M ,et al. A self-organizing concept for distributed end-to-end quality monitoring[R]. Germany:University of Wurzburg Institute, 2006.
[19]	董长青, 任女尔, 张庆余 ,等. 基于HBase+ElasticSearch的海量交通数据实时存取方案设计[J]. 大数据, 2017(1): 80-89.
	DONG C Q , REN N E , ZHANG Q Y ,et al. Design scheme of massive traffic data real-time access based on HBase and Elastic Search[J]. Big Data Research, 2017(1): 80-89.
[20]	李大中, 刘剑, 邓景文 . 大数据能力开放平台创新和发展[J]. 大数据, 2017(1): 72-80.
	LI D Z , LIU J , DENG J W . Innovation and development of big data ability open platform[J]. Big Data Research, 2017(1): 72-80.
[21]	石勇, 孟凡 . 信用评分基本理论及其应用[J]. 大数据, 2017(1): 19-27.
	SHI Y , MENG F . Credit scoring:basic theory and applications[J]. Big Data Research, 2017(1): 19-27.
[22]	刘岩, 王华, 秦叶阳 ,等. 智慧城市多源异构大数据处理框架[J]. 大数据, 2017(1): 51-60.
	LIU Y , WANG H , QIN Y Y ,et al. Multisource heterogeneous big data processing architecture in smart city[J]. Big Data Research, 2017(1): 51-60.

字段ID	字段含义
idvisitor	用户ID
id_url	页面ID
server_time	用户访问时间
time_spent_ref_action	页面停留时间
action_type	用户行为

字段ID	字段含义
id_url	页面ID
id_name	页面URL
id_re_name	去掉时间参数后的URL

字段ID	字段含义
id_url	页面ID
id_name	页面URL

字段ID	字段含义
id_url	页面ID
sim_url	相似度页面ID列表，ID之间用“$”分隔
sim_url_value	相似度列表，值之间用“$”分隔

操作系统	Red Hat 7.1
CPU	Intel(R) CPU E5-2680 v3，主频2.60 GHz，8核
内存	32 GB
硬盘	SAS 300 GB×2做的RAIDl，用于操作系统和软件；STAT l TB×3用于Hadoop 数据存储