Along with the constant growth of massive online social networks such as Facebook,Twitter,Weixin and Weibo,a tremendous amount of network data sets are generated.How to represent the data is an important aspect when we apply machine learning techniques to analyze network data sets.Firstly,the research background was introduced and the definitions of NRL (network representation learning) were related.According to the categories of different algorithms,five kinds of primary NRL algorithms were introduced.Particularly,a detailed introduction to NRL algorithms based deep learning techniques was given emphatically.Then the evaluation methods and application scenarios of NRL were discussed.Finally,the research prospect of NRL in the future was discussed.
Based on the development of foreign government data opening,the important characteristics of foreign government data opening were summarized,including the perfect laws and regulations system,professional data open web site and application needs guidance.The necessity of opening China government data was analyzed from requirement of data opening in the big data era,improving the governance capability and promoting the innovation development.The problems of government data opening in our country in the aspects of law,regulation,system,consciousness and management were summarized.The countermeasures and suggestions were proposed from the aspects of building code and standard system,data opening and sharing mode,government data opening management,data security and data application.
Big data has become a new technology, which has drawn much attention of media and public. Widely applications of big data indicated that the information age will enter into a new stage. However, the understanding of big data is a process of deepening. The big data from the height of “new information age stage”, data culture and epistemology was expounded. Then how to correctly understand the value and benefit of big data through the explanation of driving effect and wisdom in cyberspace was discussed. The challenges for the research and application of big data technology from the angle of the complexity were analyzed. Finally, some views on avoiding the pitfalls when developing big data technologies were proposed.
The clear property right is the fundamental precondition of trade.However,the question of the data property right is still far not reached a consensus in the current.Especially in the circumstance of removal of personal identity attribute data,who has the property right of the anonymous data,the data subject or the enterprise that record the data? The author puts forward an exploratory opinion that for the original personal data,the data subject owns his data.For the fully anonymous data sets based on the raw data,the enterprise that record the data should be entitled a restricted ownership on it.
DNA sequence data generated by high-throughput sequencing technology is short in length, and the amount of data is enormous. The challenges and opportunities of the big data in high-throughput sequencing environment were analyzed. The data compression, the assembly of metagenomic sequence data, and algorithms and tools of metagenomic sequence data analysis also were summarized and discussed. Finally, the future of the study on short read DNA sequence data in high-throughput sequencing environment was discussed.
In recent years, advanced persistent threat (APT) has become the chief threat to enterprise users.The traditional security protection methods, such as signature-based detection and perimeter protection, are insufficient in dealing with APT.Therefore, the status of network anomaly behavior detection method was described.The technology roadmap and system architecture of abnormal behavior detection based on statistical learning were introduced.The feature extract method and statistical modeling methods were proposed.The characteristic of abnormal behavior detection based on big data was concluded and the direction of future research was proposed.
To achieve efficient big data machine learning, we need to construct a unified big data machine learning system to support both machine learning algorithm design and big data processing. Designing an efficient, scalable and easy-to-use big data machine learning system still faces a number of challenges. Recently, the upsurge of big data technology has promoted rapid development of big data machine learning, making big data machine learning system to become a research hotspot. The basic concepts, research issues, technical characteristics, categories, and typical systems for big data machine learning system, were reviewed. Then a unified and cross-platform big data machine learning system, Octopus, was presented.
Privacy disclosure issue becomes more and more serious due to big data analysis.Privacy-preserving techniques should be conductive to the big data applications while preserving data privacy.Since big data has the characteristics of huge scale,numerous sources and dynamic update,most traditional privacy preserving technologies are not suitable any more.Therefore,the concept of privacy and life cycle protection model of big data era were introduced firstly.Technical state of big data privacy preservation was elaborated from the points of view of four stages in big data life cycle,i.e.data publishing,storage,analysis and use.The relative merits and scope of application of each technology were investigated as well.Finally,some important direction and tendency of privacy preservation technologies for big data were suggested.
Making data flow like liquid is at the heart of this so-called “Data Revolution” age.Open data movement,influenced by open source movement,advocates that government data but also private-sector data,third-sector data,and even particular type of personal data should be opened up for public interest.Therefore,what is open data,how the movement advanced thus far,the global status of the movement,potential for business,and status and challenges of the movement in China were reviewed and summarized.
Starting with big data and big data in healthcare,firstly,challenges and improvements of big data in healthcare were elaborated.Then the background of big data and healthcare industry was presented.Finally,big data applications in healthcare were illustrated,and analysis on the systems of big data in healthcare and their key technologies were made.
The technical characteristics, tasks, and difficulties of data mining in big data era were introduced.The system architecture of large-scale data mining was analyzed.Then, the developed FIU-Miner which is a fast, integrated, and user-friendly system for data mining, was introduced.FIU-Miner supports user-friendly rapid data mining task configuration, flexible cross-language program integration, and effective resource management in heterogeneous environments.Finally three successful real-world applications of FIU-Miner: advanced manufacturing data mining, spatial data mining, and business intelligence data mining, were presented to demonstrate its efficacy and effectiveness.
Urban traffic big data has such characteristics as various types, heterogeneity, large temporal and spatial scales, highly dynamic and random, localization and short life cycle.How to effectively collect and utilize the traffic big data to serve the real-time traffic administrative supervision, efficient operations of transportation enterprise, traffic services and other application needs, are unprecedented opportunities and challenges of urban transportation and smart cities.Several big data core technologies for urban transportation were summarized, and an intelligent application system solution was proposed, and some typical application was listed.In short, big data research and its application cases in the field of urban transportation and smart city were initially discussed.
The government big data is a large data source with high density value.Making good use of the government big data is of great benefit to promote the economic development and improve the public service.The enterprise which could propose solutions is a bridge between the government big data and the public demands.Open government big data and develop big data business market is an effective way for the government to release the value of big data.Taking the government big data of Ningbo under its smart city construction as an example,the obstacles and challenges of the government big data market utilization were analyzed,and the mode and countermeasures of the government big data market utilization were also discussed.
Big data is still a buzzword，and more and more people are talking about it with various kinds of different explanations.Based on writer’s understanding，the big data，big data strategy and “internet plus” initiative will be discussed here.The database philosophy was revisited，for understanding the development of data management is meaningful to catch the good opportunities in big data era.Moreover，from the point of view of a senior IT professional，the development paradigm for IT has been shifted in the past decade.The change was described，and three systems and their development and deployment were presented.A new concept，sharing database，was proposed to catch up the notion behind the block chain.
In big data era,the timeliness of data has become one of the most important factors,and the streaming feature of data has become more obvious.More and more applications need to be deployed in stream computing platforms.Big data stream computing as a major form of big data computing has become more and more important.The features of big data stream computing application were systematically analyzed.The principle strategies to build a big data stream computing system were given from the perspective of system architecture.Combined with some typical big data stream computing systems,some technology challenges in big data stream computing environments were focused,such as resource scheduling in online environments,fault tolerance strategy in node-dependence environments.
Since the authorities conduct deep reform of commercial auto insurance,Chinese insurance companies are able to get more autonomy in product pricing and the insurance companies are required to improve their capabilities of delicacy risk management and risk analysis.The point of location-based big data to discuss applications of mobile positioning methods in auto insurance risk control was proposed.The implementation path and service recommendations to insurance underwriting and claim settlement based on grid-based methods of risk assessment and calculation were given.
In big data era,recommendation system is the key means to tackle the issue of “information overload”.Recommendation system has been widely applied to many domains.The most typical and promising domain is the e-commence.Recently,with the rapid development of e-commence,recommendation system becomes more and more important and is promoted as a hot research field.The history and development of recommendation system,its domain requirements and system architecture,its characteristics and challenges under big data environment,its key techniques,open source big data recommendation systems were introduced.And at last,the open research problems and future trends of bid data recommendation system were discussed.
Beyond transparency, open data has been considered as new type of tool to stimulate innovation and change economic structure. The rise of civic technology also reflects that open data is fundamental to today’s co-governance and co-creation. Building upon these, it is important to recognize that the evolution of the concept, mechanism and form of open data has significant impact on how multiple-stakeholders are involved in governance and innovation. From this perspective, the path of open data development was reflected and relevant challenges and new opportunities were discussed.
Text content analysis is an effective way to understand and acquire the “value” of big fata.The challenges and research results were investigated in the three hot topics: topic modeling for short texts,word embedding and learning to rank for web pages.In the end,some remaining problems in this area were proposed.
Both big data and high performance computing (HPC) are based on the computer technologies. The main methodology of HPC is simulation, which is called the third paradigm of scientific discovery. Big data explore data for correlations even without much knowledge on the object of study, which is called the fourth paradigm of scientific discovery. Big data and HPC with several aspects were compared, such as the research paradigm, main application domain and underlying hardware/software systems.
Big data computing is a necessary way to acquire the “great value” behind the big data, and a computing system is an effective tool for big data computing. Big data computing from a system perspective was reviewed. Based on the fact that big data has the macro characteristics of huge volume, growing fast, complex structure, and quality disparity, the typical features of big data computing by analyzing batch computing, stream computing, and graph computing respectively, were discussed. These features may bring technical challenges to the design and implementation of big data computing system. The related works for overcoming these challenges were further categoried. In the end, some prospective research directions of big data computing from the system perspective were listed.
Agricultural big data center (ABDC) is the premise and basis for developing agricultural big data,China should actively push for the construction of ABDC.The strategic demands for ABDC were analyzed from the view of the development of modern agriculture and the data basis of ABDC building was also introduced.At last,the main components of ABDC were proposed and the application prospect of ABDC was discussed.
Abnormal groups can be found in a wide range of areas.Together with clustering and outlier detection,their goals are all to partition a data set according to data similarity.However,abnormal group mining (AGM) is different in problem definition,algorithm design and applications.To the best of our knowledge,the abnormal group mining problem was investigated systematically.The differences among AGM,clustering and outlier detection were analyzed.The formalized definitions on AGM and a framework algorithm were presented,and several interesting applications were particularized.
Data circulation is the decisive factor realizing big data value.The benefits of big data in social management and economic development are highly restricted because of immature of data exchange market of China.Based on the contrast analysis of the data circulation market,awareness was proposed that the data exchange was mainly hindered by the integrity of the value chain as well as the fear of business secret and personal privacy leakage.Finally,solutions were advised from the following aspects:the commercialization of data,the establishment of social cognition,the protection of the rights and interests of market entities.
The importance of the big data has been widely recognized since it is the main content of the“internet plus”action plan.Agriculture is an important application field of big data.Big data brings new opportunities for development for agricultural information monitoring and early warning.The problems of traditional agricultural development in China were introduced.The important role of the internet plus agriculture was expounded.The importance and specific measures of developing modern agriculture were analyzed.The proposed recommendations can improve the agricultural comprehensive production capacity,reduce the agricultural resources consumption,and provide reference for the construction of the modern agriculture based on the internet and big data technology.
In the age of big data,data scientist has become a hot occupation,supplanting traditional information scientist and big data engineer.Big data boom has been pushing data science research into fast development phase.How to train data scientists has been paid widespread attentions.Many universities launched data science degree training plans.The current situations in data scientists training were analyzed.The achievements of training data scientists in Fudan University were summarized.A systematical data scientists training plan was proposed.
The big data development has been escalated to the strategic altitude by the “Executive Outlines on Facilitating Big Data Development” publicized by the General Office of State Council,stressing the development of business applications such as big data for agriculture and rural areas.The modern agriculture faced lot of realistic predicaments.These obstacles urged big data applications to propel the transformation and upgrade of our agricultural.Aimed at the existing problems,the development status of agricultural data in Shandong province was analyzed.The necessity of agricultural big data development,the corresponding major tasks and safeguard measures of this development were proposed.This article can provide preference for the related policy making.
Big data application is still in its primary stage,and agricultural big data is the integrated products of many different subjects,so the insufficiency of big data specialists or experts is an important bottle-neck of big data application.The concept of agricultural big data,the processing technology and its application were briefly introduced.Combining the characteristics of the agricultural big data,aiming at the situation of agricultural big data talent lack,the importance of graduate students majored in agriculture big data was discussed.On this basis,the essence of training scheme and the training mode of graduate students were discussed.
According to the framework of “one cloud,one engine,four theme library,one portal,three typical applications” from the point of data asset,one big data open sharing platform based the cloud infrastructure was built,and one unified management engine of big data was realized.Four big data theme libraries as government information,public information and etc were created.The big data security access control model was also described.Through the unified cross-language platform interface,the big data could be accessed by the third party application developers.One portal for smart city for data or applications customization and evaluation for data providers,application developers and users was created,and one industrial ecology of big data demand,application development and capital providing was aimed to create.
Mobile crowdsensing (MCS) has become an emerging paradigm for large-scale sensing.It empowers ordinary citizens to contribute data sensed or generated from their mobile devices (e.g.,smartphones,wearable devices),aggregates and fuses the data in the cloud for crowd intelligence extraction and human-centric service delivery.The data contributed by the crowd in MCS systems presents the features such as multi-modal,rich-content,spatio-temporal,and human-centric.The key challenges and techniques about crowdsensing big data were discussed.The recent progress of our group in this promising research area was described.
Big data contains important value information.Text big data as an important part of big data is the main carrier of human knowledge.Feature represents the inherent law of the data.Mapping the text big data to its feature space which reflects the nature of data is an important method to understand the semantic meaning of the text.Text big data feature representations and feature learning were reviewed.Then the progress of feature learning used in text content understanding was presented.Finally,the future development trends of big text data content understanding were discussed.
To promote the continuous development of big data and improve the government and enterprises’ awareness of data management,the organizational data management of the eight process areas were extracted by analyzing and summarizing the data management capabilities,combining with data lifecycle management at all stages of the characteristics and the theory from domestic and abroad.Otherwise,each capability was divided into two process areas and development levels,introduced the functions and formulated the standardization of assessment.
Industrial big data is an important asset of industrial enterprises.It is a crucial factor for an industrial enterprise to realize transformation and upgrading.The main sources and characteristics of industrial big data were analyzed,and reference architecture of industrial big data with three dimensions was proposed.Three aspects of the realization of business innovation and transformation of industrial enterprise based on industrial big data respectively were discussed,which included the typical application and business innovation of enterprise,cyber (physical) systems deployed at all levels of the enterprise,and the business architecture,information systems architecture and IT architecture that guide the implementation of application of industrial big data.Finally,the architecture and technology of a typical application of industrial big data were analyzed,which verified the validity of the proposed architecture.
Large graphcomputing has been a fundamental computing pattern in both academic and industry field,and it was applied to a lot of practical big data applications,such as social network analysis,web page search,and goods recommendation.In general,most of large graphs scale to billions of vertices,and corresponding to hundreds billions of edges,which brings us challenges of efficient graph processing.Therefore,the basic feature and challenges of current large graph computing,typical computing models,and representative distributed,and single machine large graph processing systems were introduced.Then,some key technologies employed in large graph computing were summarized.Finally,some research directions in large graph computing from a system perspective were given.
Big data and the related applications which derived from the internet originally, are now expanding to other industries, and becoming the key driving force of their innovation and transition. The evolvement of the search engine driven by big data technologies was described, based on Baidu’s innovations and practices in the big data area over the years. Baidu big data engine and its explorations in other industries were introduced. Finally, a vision was discussed that big data and artificial intelligence will be prospected in the future information communication technology.
In recent years, P2P lending industry in China has appeared a lot of escape events in the process of its rapid development.Bases on deep analysis of the related concepts for P2P lending and big data, combining innovatively the risk pre-warning of platform with big data, an effective risk pre-warning model of P2P lending platform was constructed according to the collection of huge amounts of data, big data technology including Spark distributed computation and machine learning.Based on the establishment of multi-dimensional risk assessment, the model can be achieved on real-time, accurate, comprehensive monitoring for the risk of P2P lending, thus effectively reducing the frequency of financial fraud, escape malicious event, so as to the majority of investors’ money to maintain security and social stability.
It is a basic work to discover high quality internet information sources automatically for scientific and technological intelligence.The technology of website/webpage information sources discovery was presented based on the co-citation relationship, and the technology of Twitter information sources discovery was presented based on the following relationship and content analysis.Then, the application forms of automatic discovery of information sources were discussed.Three kinds of application scenarios were presented based on the analysis of the requirements of scientific and technological intelligence.
Electronic medical records from hospitals record the patient's disease,diagnosis and treatment information.It forms the basis of clinical data.Mining such data can assist doctors in clinical research and clinical diagnosis and treatment.Firstly,challenges encountered in the process of big data mining on EMR were raised,then the core process was summarized.The process includes tasks such as clinical data integration,the construction of clinical specialist disease database based on knowledge graph,the quality assessment methods on EMR,and comparative effectiveness and risk prediction of diseases as the core of clinical big data applications.A solution for each task was proposed,and the experimental results were given.Finally,the future directions of technologies and applications of big data mining on healthcare were presented.
Scientific Data Resource Integration and Sharing Project is one of the 5 major informatization-specific projects of CAS for the 12th Five-Year Plan period.The overall construction of the project ideas,construction,technical innovation and service innovation,etc.,was summarized.By the end of the project,a distributed mass storage environment with storage capacity of 52 PB was built.At the same time,it provided users with a strong connection between scientific data and literature and a rich visual display platform.The project has initially achieved a multi-level,cross information service system that included the infrastructure cloud service,research data cloud service and data application cloud service.It has gradually become a national science and technology data center for open sharing and service innovation.
The data selected contained 900 million query records in the latest 36 months from the personal credit reference system database.The model was subdivided according to different volatility characteristics of each user’s query behavior,and four types of real-time anomaly inquiry monitoring models were discussed and developed.Results indicate that the anomaly inquiry monitoring model is feasible to apply on predicting anomaly query behaviors and showed positive effects.The successful application and constant perfection of the model would definitely exert deterrent effect on illegal query behaviors,force commercial banks to strengthen internal management,protect individual’s private information and right,and promote the healthy development of the credit reference market.
