How Unsupervised Learning Affects Character Tagging based Chinese Word Segmentation: A Quantitative Investigation

摘要：

Integrating global information of unsupervised segmentation into Conditional Random Fields (CRF) learning has been proved effective to enhance the performance of the character tagging based Chinese Word Segmentation. By comparing CRF models with and without unsupervised learning enhancement, we investigate how unsupervised learning affects the performance. Especially, two kinds of segmented words, in-vocabulary and out-of-vocabulary words, are separately analyzed case by case to see what part of those words are affected by unsupervised learning. In addition, the cost of the additional features derived from unsupervised segmentation are also taken into account and evaluated.

关键词： Unsupervised learning Chinese word segmentation in-vocahulary words out-of-vocabulary words frequent substring eztraction

作者: YAN SONG CHUNYU KIT RUIFENG XU HAI ZHAO

作者单位: Department of Chinese, Translation and Linguistics City University of Hong Kong, 83 Tat Chee Ave., Kowloon, Hong Kong

会议类型: 国际会议

会议名称: 2009 International Conference on Machine Learning and Cybernetics(2009机器学习与控制论国际会议)

会议地点: 保定

会议语种:英文

页码: 3481-3486

在线出版日期: 2009-07-12（万方平台首次上网日期，不代表论文的发表时间）