Context information plays an important role in human language understanding, and it is also useful for machines to learn vector representations of language. In this paper, we explore an asymmetric encoder-decoder structure for unsupervised context-based sentence representation learning. As a result, we build an encoder-decoder architecture with an RNN encoder and a CNN decoder, and we show that neither an autoregressive decoder nor an RNN decoder is required.   We further combine a suite of effective designs to significantly improve model efficiency while also achieving better performance. Our model is trained on two different large unlabeled corpora, and in both cases transferability is evaluated on a set of downstream language understanding tasks. We empirically show that our model is simple and fast while producing rich sentence representations that excel in downstream tasks.
We proposed an RNN-CNN encoder-decoder model for fast unsupervised sentence representation learning. Modifications to the skip-thought framework for learning sentence embeddings. This paper presents a new RNN encoder–CNN decoder hybrid design for use in pretraining, which does not require an autoregressive decoder when pretraining encoders. The authors extend Skip-thought by decoding only one target sentence using a CNN decoder.