Inter- Annotator Agreement and the Upper Limit on Machine Performance: Evidence from Biomedical Natural Language Processing

摘要：

　　Human-annotated data is a fundamental part of natural language processing system development and evaluation. The quality of that data is typically assessed by calculating the agreement between the annotators. It is widely assumed that this agreement between annotators is the upper limit on system performance in natural language processing: if humans cant agree with each other about the classification more than some percentage of the time, we dont expect a computer to do any better. We trace the logical positivist roots of the motivation for measuring inter-annotator agreement, demonstrate the prevalence of the widely-held assumption about the relationship between inter-annotator agreement and system performance, and present data that suggest that interannotator agreement is not, in fact, an upper bound on language processing system performance.

关键词： Natural Language Processing Supervised Machine Learning Evaluation Studies

作者: Mayla Boguslav Kevin Bretonnel Cohen

作者单位: Computational Bioscience Program,University Colorado School of Medicine,Aurora,CO,USA

会议类型: 国际会议

会议名称: 第十六届世界医药健康信息学大会((MEDINFO2017)、第二届世界医药健康信息学华语论坛(WCHIS 2017)、第15届全国医药信息学大会(CMIA 2017)

会议地点: 苏州

会议语种:英文

页码: 298-302

在线出版日期: 2017-08-21（万方平台首次上网日期，不代表论文的发表时间）