Automatic Extraction and Filtration of Multiword Units1
we use five statistical models including Dice coefficient (Dice), Φ 2 coefficient (Φ2), log likelihood ratio (LLR), symmetrical conditional probability (SCP), and normalized expectation(NE) to extract multiword unit candidates from patent corpus. We compare the results from five models and find the number of multiword unit candidates using NE is the most and the precision of Dice is the maximal, but the number of multiword unit candidates using Dice is the least and the precision of SCP is the minimum. Next the multiword unit candidates are filtrated using these filtration strategies including stop words, the threshold, higher frequency, first stop words, last stop words, and context entropy. After filtration, the number of multiword units using NE is the most and the precision of Dice is the maximal, but the number of multiword units using Dice is the least and the precision of SCP is the minimum. Each filtration strategy all help to identify the wrong or unreasonable multiword units and improve the precision of multiword units.
multiword unit Dice Φ2 SCP NE LLR extract filtrate
Ying Liu Zheng Tie
Department of Chinese Language and Literature, Tsinghua University Beijing, China, 100084
国际会议
上海
英文
2651-2655
2011-07-26（万方平台首次上网日期，不代表论文的发表时间）