Support for BPE vocabs + denoising autoencoder in PyTorch Translate (#362)
Support for BPE vocabs + denoising autoencoder in PyTorch Translate (#362)

Summary:
Pull Request resolved: https://github.com/pytorch/fairseq/pull/362

Pull Request resolved: https://github.com/pytorch/translate/pull/254

This actually uses the fairseq logic which supports BPE cont / end word marker suffixes.

Reviewed By: xianxl

Differential Revision: D12952766

fbshipit-source-id: 35a1bbc38240e4145bec0fc419f2d0a6a73ae2e5

pytorch/fairseq,pytorch/fairseq,pytorch/fairseq