Depth Growing for Neural Machine Translation

Published:

Please cite:
@inproceedings{wu2019depth,
title={Depth Growing for Neural Machine Translation},
author={Wu, Lijun and Wang, Yiren and Xia, Yingce and Tian, Fei and Gao, Fei and Qin, Tao and Lai, Jianhuang and Liu, Tie-Yan},
booktitle={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
pages={5558--5563},
year={2019}
}

Abstract

While very deep neural networks have shown effectiveness for computer vision and text classification applications, how to increase the network depth of the neural machine translation (NMT) models for better translation quality remains a challenging problem. Directly stacking more blocks to the NMT model results in no improvement and even drop in performance. In this work, we propose an effective two-stage approach with three specially designed components to construct deeper NMT models, which result in significant improvements over the strong Transformer baselines on WMT14 English→German and English→French translation tasks.

[PDF] [CODE]