Discovering Approximate Functional Dependencies from Distributed Big Data
Approximate Functional Dependencies (AFDs) discovered from database relations have proven to be useful for various tasks,such as knowledge discovery,query optimization.Previous research has proposed different algorithms to discover AFDs from a centralized relational database.However,none of the proposed algorithms is designed to discover AFDs from distributed data.In this paper,we devise a scalable and efficient approach to discover AFDs from distributed big data and not tied to main memory requirements.To improve the efficiency of AFDs discovery,statistics of local data in each site are collected to filter and prune the candidate AFDs set at first.The AFDs are discovered in parallel after data redistribution.We balance the load as much as possible before the redistribution of data and prune the candidate AFDs set quickly after the redistribution of data.We evaluate our approach using real and synthetic big datasets and the results show that our approach is more efficient and scalable on large relations and the number of nodes.
Distributed data Big data Knowledge discovery
Weibang Li Zhanhuai Li Qun Chen Tao Jiang Zhilei Yin
Northwestern Polytechnical University,Xian 710072,China
国际会议
International Asia-Pacific Web Conference(第18届国际亚太互联网大会)
苏州
英文
289-301
2016-09-23（万方平台首次上网日期，不代表论文的发表时间）