State-of-the-art neural machine translation methods employ massive amounts of parameters. Drastically reducing computational costs of such methods without affecting performance has been up to this point unsolved. In this work, we propose a quantization strategy tailored to the Transformer architecture. We evaluate our method on the WMT14 EN-FR and WMT14 EN-DE translation tasks and achieve state-of-the-art quantization results for the Transformer, obtaining no loss in BLEU scores compared to the non-quantized baseline. We further compress the Transformer by showing that, once the model is trained, a good portion of the nodes in the encoder can be removed without causing any loss in BLEU.
We fully quantize the Transformer to 8-bit and improve translation quality compared to the full precision model. An 8-bit quantization method to quantize the machine translation model Transformer, proposing to use uniform min-max quantization during inference and bucketing weigts before quantization to reduce quantization error. A method for reducing the required memory space by a quantization technique, focused on reducing it for Transformer architecture.