use utterance level sampling strategy for stella giga data training
use utterance level sampling strategy for stella giga data training

Summary:
Before, we would use batch level sampling with fairseq's MultiCorpusSampledDataset. This means we would sample a batch of portal data, a batch of video data, etc.

The main disadvantage of this is that the implementation for this requires the sampling to happen **after** splitting up the data into batches. This means we don't know the underlying data of each batch until right before the forward call for that batch, which means when splitting up the data for batching fairseq overestimates the tokens per item by just taking the max across all datasets from which the sample could come from. This causes batching inefficiency because it overestimates the actual batch size.

In D21887303 (https://github.com/pytorch/fairseq/commit/8570277f91d6bed03d71cc9c8326f096cd06b0d2) we added utterance level sampling, which samples **before** splitting up the data for batching on an utterance level. This is achieved through a new dataset introduced in that diff and refreshing the batch iterator every epoch. This enables accurate batch size calculation.

Reviewed By: jay-mahadeokar

Differential Revision: D22280839

fbshipit-source-id: 8d6c8b82267184558fe3ec9e1b77fe6f24e0376f

pytorch/fairseq,pytorch/fairseq,pytorch/fairseq