Abstract:

ETL is a very important step in the implementation process of data warehouse.A good ETL flow is important,which can effectively process the telecom big data and improve the acquisition efficiency of the operation platform.Firstly,the main data content of the big data platform was expounded.Secondly,in order to improve the efficiency of massive data collection,Hadoop and Oracle mashup solution was suggested.Subsequently,a dynamic triggered ETL scheduling flow and algorithm was proposed.Compared with timer start ETL scheduling method,it could effectively shorten waiting time and avoid the phenomenon of resources to seize and congestion.Finally,according to the running log of Hadoop platform and Oracle database,the relationship between acquisition efficiency and data quantity was analyzed comparatively.Furthermore,practice result shows that the hybrid data structure of the big data platform complement each other and can effectively enhance the timeliness of data collection and access better application effect.

Key words: big data, ETL, Hadoop, scheduling process, mashup architecture