The Solution of Huawei Cloud&Noah's Ark Lab to the NLPCC-2020 Challenge:Light Pre-Training Chinese Language Model for NLP Task

摘要：

　　Pre-trained language models have achieved great success in natural lan-guage processing.However,they are difficult to be deployed on resource-restricted devices because of the expensive computation.This paper introduces our solution to the Natural Language Processing and Chinese Computing(NLPCC)challenge of Light Pre-Training Chinese Language Model for the Natural Language Process-ing(http://tcci.ccf.org.cn/conference/2020/)(https://www.cluebenchmarks.com/NLPCC.html).The proposed solution uses a state-of-the-art method of BERT knowledge distillation(TinyBERT)with an advanced Chinese pre-trained lan-guage model(NEZHA)as the teacher model,which is dubbed as TinyNEZHA.In addition,we introduce some effective techniques in the fine-tuning stage to boost the performances of TinyNEZHA.In the official evaluation of NLPCC-2020 challenge,TinyNEZHA achieves a score of 77.71,ranking 1 st place among all the participating teams.Compared with the BERT-base,TinyNEZHA obtains almost the same results while being 9x smaller and 8x faster on inference.

关键词： Pre-trained language model Knowledge distillation TinyNEZHA

作者: Yuyang Zhang Jintao Yu Kai Wang Yichun Yin Cheng Chen Qun Liu

作者单位: Huawei Noah's Ark Lab,Beijing,China;Huawei Cloud and AI,Shenzhen,China

会议类型: 国际会议

会议名称: 9th CCF International Conference on Natural Language Processing and Chinese Computing (NLPCC 2020)

会议地点: 郑州

会议语种:英文

页码: 1375-1384

在线出版日期: 2020-10-14（万方平台首次上网日期，不代表论文的发表时间）