Fix multiGPU peak gpu memory test (#2254)
Fix multiGPU peak gpu memory test (#2254)

This test was failing on a multigpu machine i use:

```
$ py.test -k "trainer_test"
=============================================================================================== test session starts ================================================================================================
platform linux -- Python 3.6.5, pytest-3.6.2, py-1.5.3, pluggy-0.6.0
rootdir: /home/nfliu/Documents/Github/allennlp, inifile: pytest.ini
plugins: cov-2.5.1, flaky-3.4.0
collected 1161 items / 1140 deselected

allennlp/tests/training/trainer_test.py .............F.......                                                                                                                                                [100%]

===================================================================================================== FAILURES =====================================================================================================
__________________________________________________________________________________ TestTrainer.test_trainer_can_run_multiple_gpu ___________________________________________________________________________________

self = <allennlp.tests.training.trainer_test.TestTrainer testMethod=test_trainer_can_run_multiple_gpu>

    @pytest.mark.skipif(torch.cuda.device_count() < 2,
                        reason="Need multiple GPUs.")
    def test_trainer_can_run_multiple_gpu(self):

        class MetaDataCheckWrapper(Model):
            """
                Checks that the metadata field has been correctly split across the batch dimension
                when running on multiple gpus.
                """
            def __init__(self, model):
                super().__init__(model.vocab)
                self.model = model

            def forward(self, **kwargs) -> Dict[str, torch.Tensor]:  # type: ignore # pylint: disable=arguments-differ
                assert 'metadata' in kwargs and 'tags' in kwargs, \
                    f'tokens and metadata must be provided. Got {kwargs.keys()} instead.'
                batch_size = kwargs['tokens']['tokens'].size()[0]
                assert len(kwargs['metadata']) == batch_size, \
                    f'metadata must be split appropriately. Expected {batch_size} elements, ' \
                    f"got {len(kwargs['metadata'])} elements."
                return self.model.forward(**kwargs)

        multigpu_iterator = BasicIterator(batch_size=4)
        multigpu_iterator.index_with(self.vocab)
        trainer = Trainer(MetaDataCheckWrapper(self.model), self.optimizer,
                          multigpu_iterator, self.instances, num_epochs=2,
                          cuda_device=[0, 1])
        metrics = trainer.train()
        assert 'peak_cpu_memory_MB' in metrics
        assert isinstance(metrics['peak_cpu_memory_MB'], float)
        assert metrics['peak_cpu_memory_MB'] > 0
        assert 'peak_gpu_0_memory_MB' in metrics
>       assert isinstance(metrics['peak_gpu_0_memory_MB'], float)
E       AssertionError: assert False
E        +  where False = isinstance(549, float)

allennlp/tests/training/trainer_test.py:128: AssertionError
----------------------------------------------------------------------------------------------- Captured stderr call -----------------------------------------------------------------------------------------------
12/27/2018 09:34:40 - INFO - allennlp.common.checks -   Pytorch version: 0.4.1
0it [00:00, ?it/s]12/27/2018 09:34:40 - INFO - allennlp.data.dataset_readers.sequence_tagging -   Reading instances from lines in file at: /home/nfliu/Documents/Github/allennlp/allennlp/tests/fixtures/data/sequence_tagging.tsv
4it [00:00, 12409.18it/s]
12/27/2018 09:34:40 - INFO - allennlp.data.vocabulary -   Fitting token dictionary from dataset.
100%|██████████| 4/4 [00:00<00:00, 71392.41it/s]
12/27/2018 09:34:40 - INFO - allennlp.common.from_params -   instantiating class <class 'allennlp.models.simple_tagger.SimpleTagger'> from params {'text_field_embedder': {'token_embedders': {'tokens': {'type': 'embedding', 'embedding_dim': 5}}}, 'encoder': {'type': 'lstm', 'input_size': 5, 'hidden_size': 7, 'num_layers': 2}} and extras {'vocab': <allennlp.data.vocabulary.Vocabulary object at 0x7fe36c0f5438>}
12/27/2018 09:34:40 - INFO - allennlp.common.from_params -   instantiating class <class 'allennlp.modules.text_field_embedders.text_field_embedder.TextFieldEmbedder'> from params {'token_embedders': {'tokens': {'type': 'embedding', 'embedding_dim': 5}}} and extras {'vocab': <allennlp.data.vocabulary.Vocabulary object at 0x7fe36c0f5438>}
12/27/2018 09:34:40 - INFO - allennlp.common.from_params -   instantiating class <class 'allennlp.modules.token_embedders.token_embedder.TokenEmbedder'> from params {'type': 'embedding', 'embedding_dim': 5} and extras {'vocab': <allennlp.data.vocabulary.Vocabulary object at 0x7fe36c0f5438>}
12/27/2018 09:34:40 - INFO - allennlp.common.from_params -   instantiating class <class 'allennlp.modules.seq2seq_encoders.seq2seq_encoder.Seq2SeqEncoder'> from params {'type': 'lstm', 'input_size': 5, 'hidden_size': 7, 'num_layers': 2} and extras {'vocab': <allennlp.data.vocabulary.Vocabulary object at 0x7fe36c0f5438>}
12/27/2018 09:34:40 - WARNING - allennlp.training.trainer -   Multiple GPU support is experimental not recommended for use. In some cases it may lead to incorrect results or undefined behavior.
12/27/2018 09:34:40 - INFO - allennlp.training.trainer -   Beginning training.
12/27/2018 09:34:40 - INFO - allennlp.training.trainer -   Epoch 0/1
12/27/2018 09:34:40 - INFO - allennlp.training.trainer -   Peak CPU memory usage MB: 2121.256
12/27/2018 09:34:40 - INFO - allennlp.training.trainer -   GPU 0 memory usage MB: 533
12/27/2018 09:34:40 - INFO - allennlp.training.trainer -   GPU 1 memory usage MB: 10
12/27/2018 09:34:40 - INFO - allennlp.training.trainer -   Training
  0%|          | 0/1 [00:00<?, ?it/s]12/27/2018 09:34:40 - DEBUG - allennlp.data.iterators.data_iterator -   Batch padding lengths: {'tokens': {'tokens_length': 4, 'num_tokens': 4}, 'tags': {'num_tokens': 4}}
12/27/2018 09:34:40 - DEBUG - allennlp.data.iterators.data_iterator -   Batch size: 4
loss: 0.7543 ||: 100%|██████████| 1/1 [00:02<00:00,  2.04s/it]
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -                       Training |  Validation
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   loss            |     0.754  |       N/A
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   gpu_1_memory_MB |    10.000  |       N/A
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   gpu_0_memory_MB |   533.000  |       N/A
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   cpu_memory_MB   |  2121.256  |       N/A
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   Epoch duration: 00:00:02
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   Estimated training time remaining: 0:00:02
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   Epoch 1/1
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   Peak CPU memory usage MB: 3892.764
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   GPU 0 memory usage MB: 549
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   GPU 1 memory usage MB: 549
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   Training
  0%|          | 0/1 [00:00<?, ?it/s]12/27/2018 09:34:42 - DEBUG - allennlp.data.iterators.data_iterator -   Batch padding lengths: {'tokens': {'tokens_length': 4, 'num_tokens': 4}, 'tags': {'num_tokens': 4}}
12/27/2018 09:34:42 - DEBUG - allennlp.data.iterators.data_iterator -   Batch size: 4
loss: 0.7522 ||: 100%|██████████| 1/1 [00:00<00:00, 59.00it/s]
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -                       Training |  Validation
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   loss            |     0.752  |       N/A
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   gpu_1_memory_MB |   549.000  |       N/A
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   gpu_0_memory_MB |   549.000  |       N/A
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   cpu_memory_MB   |  3892.764  |       N/A
12/27/2018 09:34:42 - INFO - allennlp.training.trainer -   Epoch duration: 00:00:00
------------------------------------------------------------------------------------------------ Captured log call -------------------------------------------------------------------------------------------------
09:34:40 - INFO - allennlp.common.checks - Pytorch version: 0.4.1
09:34:40 - INFO - allennlp.data.dataset_readers.sequence_tagging - Reading instances from lines in file at: /home/nfliu/Documents/Github/allennlp/allennlp/tests/fixtures/data/sequence_tagging.tsv
09:34:40 - INFO - allennlp.data.vocabulary - Fitting token dictionary from dataset.
09:34:40 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.models.simple_tagger.SimpleTagger'> from params {'text_field_embedder': {'token_embedders': {'tokens': {'type': 'embedding', 'embedding_dim': 5}}}, 'encoder': {'type': 'lstm', 'input_size': 5, 'hidden_size': 7, 'num_layers': 2}} and extras {'vocab': <allennlp.data.vocabulary.Vocabulary object at 0x7fe36c0f5438>}
09:34:40 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.modules.text_field_embedders.text_field_embedder.TextFieldEmbedder'> from params {'token_embedders': {'tokens': {'type': 'embedding', 'embedding_dim': 5}}} and extras {'vocab': <allennlp.data.vocabulary.Vocabulary object at 0x7fe36c0f5438>}
09:34:40 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.modules.token_embedders.token_embedder.TokenEmbedder'> from params {'type': 'embedding', 'embedding_dim': 5} and extras {'vocab': <allennlp.data.vocabulary.Vocabulary object at 0x7fe36c0f5438>}
09:34:40 - INFO - allennlp.common.from_params - instantiating class <class 'allennlp.modules.seq2seq_encoders.seq2seq_encoder.Seq2SeqEncoder'> from params {'type': 'lstm', 'input_size': 5, 'hidden_size': 7, 'num_layers': 2} and extras {'vocab': <allennlp.data.vocabulary.Vocabulary object at 0x7fe36c0f5438>}
09:34:40 - WARNING - allennlp.training.trainer - Multiple GPU support is experimental not recommended for use. In some cases it may lead to incorrect results or undefined behavior.
09:34:40 - INFO - allennlp.training.trainer - Beginning training.
09:34:40 - INFO - allennlp.training.trainer - Epoch 0/1
09:34:40 - INFO - allennlp.training.trainer - Peak CPU memory usage MB: 2121.256
09:34:40 - INFO - allennlp.training.trainer - GPU 0 memory usage MB: 533
09:34:40 - INFO - allennlp.training.trainer - GPU 1 memory usage MB: 10
09:34:40 - INFO - allennlp.training.trainer - Training
09:34:40 - DEBUG - allennlp.data.iterators.data_iterator - Batch padding lengths: {'tokens': {'tokens_length': 4, 'num_tokens': 4}, 'tags': {'num_tokens': 4}}
09:34:40 - DEBUG - allennlp.data.iterators.data_iterator - Batch size: 4
09:34:42 - INFO - allennlp.training.trainer -                     Training |  Validation
09:34:42 - INFO - allennlp.training.trainer - loss            |     0.754  |       N/A
09:34:42 - INFO - allennlp.training.trainer - gpu_1_memory_MB |    10.000  |       N/A
09:34:42 - INFO - allennlp.training.trainer - gpu_0_memory_MB |   533.000  |       N/A
09:34:42 - INFO - allennlp.training.trainer - cpu_memory_MB   |  2121.256  |       N/A
09:34:42 - INFO - allennlp.training.trainer - Epoch duration: 00:00:02
09:34:42 - INFO - allennlp.training.trainer - Estimated training time remaining: 0:00:02
09:34:42 - INFO - allennlp.training.trainer - Epoch 1/1
09:34:42 - INFO - allennlp.training.trainer - Peak CPU memory usage MB: 3892.764
09:34:42 - INFO - allennlp.training.trainer - GPU 0 memory usage MB: 549
09:34:42 - INFO - allennlp.training.trainer - GPU 1 memory usage MB: 549
09:34:42 - INFO - allennlp.training.trainer - Training
09:34:42 - DEBUG - allennlp.data.iterators.data_iterator - Batch padding lengths: {'tokens': {'tokens_length': 4, 'num_tokens': 4}, 'tags': {'num_tokens': 4}}
09:34:42 - DEBUG - allennlp.data.iterators.data_iterator - Batch size: 4
09:34:42 - INFO - allennlp.training.trainer -                     Training |  Validation
09:34:42 - INFO - allennlp.training.trainer - loss            |     0.752  |       N/A
09:34:42 - INFO - allennlp.training.trainer - gpu_1_memory_MB |   549.000  |       N/A
09:34:42 - INFO - allennlp.training.trainer - gpu_0_memory_MB |   549.000  |       N/A
09:34:42 - INFO - allennlp.training.trainer - cpu_memory_MB   |  3892.764  |       N/A
09:34:42 - INFO - allennlp.training.trainer - Epoch duration: 00:00:00
===Flaky Test Report===


===End Flaky Test Report===
================================================================================================= warnings summary =================================================================================================
allennlp/tests/training/trainer_test.py::TestTrainer::test_trainer_can_run_multiple_gpu
  /home/nfliu/Documents/Github/allennlp/allennlp/modules/encoder_base.py:116: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().
    module_output, final_states = module(packed_sequence_input, initial_states)
  /home/nfliu/Documents/Github/allennlp/allennlp/modules/encoder_base.py:116: UserWarning: RNN module weights are not part of single contiguous chunk of memory. This means they need to be compacted at every call, possibly greatly increasing memory usage. To compact weights again call flatten_parameters().
    module_output, final_states = module(packed_sequence_input, initial_states)

-- Docs: http://doc.pytest.org/en/latest/warnings.html
======================================================================== 1 failed, 20 passed, 1140 deselected, 2 warnings in 26.92 seconds =========================================================================
```

Pretty sure it should be checking for `int` instead of `float`, since `common.util.gpu_memory_mb()` returns `Dict[int, int]`. See https://github.com/allenai/allennlp/blob/3f0953d19de3676ea82e642659fc96d90690e34d/allennlp/common/util.py#L327
allenai/allennlp,allenai/allennlp,allenai/allennlp,allenai/allennlp