Light Pre-Trained Chinese Language Model for NLP Tasks

摘要：

　　We present the results of shared-task 1 held in the 2020 Conference on Natural Language Processing and Chinese Computing(NLPCC):Light Pre-Trained Chinese Language Model for NLP tasks.This shared-task examines the performance of light language models on four common NLP tasks:Text Classification,Named Entity Recognition,Anaphora Resolution and Machine Reading Comprehension.To make sure that the models are light-weight,we put restrictions and require-ments on the number of parameters and inference speed of the partic-ipating models.In total,30 teams registered our tasks.Each submis-sion was evaluated through our online benchmark system(https://www.cluebenchmarks.com/nlpcc2020.html),with the average score over the four tasks as the final score.Various ideas and frameworks were explored by the participants,including data enhancement,knowledge distillation and quantization.The best model achieved an average score of 75.949,which was very close to BERT-base(76.460).We believe this shared-task highlights the potential of light-weight models and calls for further research on the development and exploration of light-weight models.

关键词： Chinese language processing Pre-trained language models Model lighting

作者: Junyi Li Hai Hu Xuanwei Zhang Minglei Li Lu Li Liang Xu

作者单位: CLUE Team,Shenzhen,China CLUE Team,Shenzhen,China;Indiana University,Bloomington,USA CLUE Team,Shenzhen,China;iQIYI Inc.,Beijing,China CLUE Team,Shenzhen,China;Speech and Language Innovation Lab,Huawei Cloud and AI,Shenzhen,China CLUE Team,Shenzhen,China;Central China Normal University,Wuhan,China

会议类型: 国际会议

会议名称: 9th CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC 2020)

会议地点: 郑州

会议语种:英文

页码: 1418-1429

在线出版日期: 2020-10-14（万方平台首次上网日期，不代表论文的发表时间）