DATA TRANSMISSION BETWEEN RELATIONAL DATABASE AND HDFS

摘要：

　　The functions of data exchange provide a series of methods,transfer data from a storage system to another storage system.HDFS is a Distributed File System realized by Hadoop,which has the character of high fault-tolerance,at the same time it provides a high transfer rate to access the data of applications,and is suitable for those applications with large data set.Traditionally,the large data is stored in FTP servers or SQL databases.We use Hadoop distributed framework for large-scale data calculation,which will certainly need to transfer data from FTP servers or SQL databases to HDFS.This paper mainly discussed the problem of parallel data exchange between SQL database and HDFS,introduced the performance of Hadoop data exchange functions:DBInputFormat/ DBOutputFormat,and put forward some strategies to improve the performance.

关键词： Hadoop HDFS RDBMS Data exchange DBInputFormat/DBOutputFormat

作者: Bin Wu Yu Jia Xinxin Ge

作者单位: Beijing Key Laboratory of Intelligent Telecommunications Software and Multimedia Beijing University of Posts and Telecommunications,Beijing 100876,China

会议类型: 国际会议

会议名称: 2012 2nd IEEE International Conference on Cloud Computing and Intelligence Systems (2012年第2届IEEE云计算与智能系统国际会议(IEEE CCIS2012))

会议地点: 杭州

会议语种:英文

页码: 419-423

在线出版日期: 2012-10-30（万方平台首次上网日期，不代表论文的发表时间）