A Scalable Crawler Framework for FLOSS Data
Free/Libre/Open Source Software (FLOSS) data,such as bug reports,mailing lists and related webpages,contains valuable information for reusing open source software projects.Before conducting further experiment on FLOSS data,researchers often need to download these data into a local storage system.We refer to this pre-process as FLOSS data retrieval,which in many cases can be a challenging task.In this paper,we proposed a crawler framework to ease the process of FLOSS data retrieval.To cope with various types of FLOSS data scattered on the Internet,we designed the framework in a scalable manner where a crawler program can be easily plugged into the system to extend its functionality.Researchers can perform the retrieval process on datasets of various types and sources simply by adding new configurations to the system.We have implemented the framework and provided basic functions via web-based interfaces.We presented the usage of the system by a detailed case study where we retrieved various types of datasets related to Apache Lucene project using our framework.
FLOSS project data retrieval crawler scalable
Lingxiao Zhang Yanzhen Zou Bing Xie
Software Institute,School of Electronics Engineering and Computer Science,Peking University;Key Laboratory of High Confidence Software Technologies,Ministry of Education Beijing 100871,P.R. China
国际会议
长沙
英文
73-79
2013-10-23（万方平台首次上网日期，不代表论文的发表时间）