Unsupervised Query Segmentation Using Generative Language Models and Wikipedia
In this paper, we propose a novel unsupervised approach to query segmentation, an important task inWeb search. We use a generative query model to recover a query’s underlying concepts that compose its original segmented form. The model’s parameters are estimated using an expectation-maximization (EM) algorithm, optimizing the minimum description length objective function on a partial corpus that is specific to the query. To augment this unsupervised learning, we incorporate evidence from Wikipedia. Experiments show that our approach dramatically improves performance over the traditional approach that is based on mutual information, and produces comparable results with a supervised method. In particular, the basic generative language model contributes a 7.4％ improvement over the mutual information based method (measured by segment F1 on the Intersection test set). EM optimization further improves the performance by 14.3％. Additional knowledge from Wikipedia provides another improvement of 24.3％, adding up to a total of 46％ improvement (from 0.530 to 0.774).
Query segmentation concept discovery
Bin Tan Fuchun Peng
Department of Computer Science University of Illinois at Urbana-Champaign Urbana, IL 61801 Yahoo! Inc. 701 First Avenue Sunnyvale, CA 94089
国际会议
第十七届国际万维网大会(the 17th International World Wide Web Conference)(WWW08)
北京
英文
2008-04-21（万方平台首次上网日期，不代表论文的发表时间）