Efficient Sequence Learning with Group Recurrent Networks

Published:

Please cite:
@inproceedings{gao2018efficient,
title={Efficient sequence learning with group recurrent networks},
author={Gao, Fei and Wu, Lijun and Zhao, Li and Qin, Tao and Cheng, Xueqi and Liu, Tie-Yan},
booktitle={Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,
Volume 1 (Long Papers)},
pages={799--808},
year={2018}
}

Abstract

Recurrent neural networks have achieved state-of-the-art results in many artificial intelligence tasks, such as language modeling, neural machine translation, speech recognition and so on. One of the key factors to the sesuccesses is big models. However, training such big models usually takes days or even weeks of time even if using tens of GPU cards. In this paper, we propose an efficient architecture to improve the efficiency of such RNN model training, which adopts the group strategy for recurrent layers, while exploiting the representation rearrangement strategy between layers as well as time steps. To demonstrate the advantages of our models, we conduct experiments on several datasets and tasks. The results show that our architecture achieves comparable or better accuracy comparing with baselines, with a much smaller number of parameters and at a much lower computational cost.

[PDF]