Add BatchNormalizationLayer We use the cuDNN v4 release candidate implementation of BatchNormalization (cudnnBatchNormalizationForwardTraining, cudnnBatchNormalizationForwardInference, and cudnnBatchNormalizationBackward). There are three things to be aware of:
Add BatchNormalizationLayer
We use the cuDNN v4 release candidate implementation of BatchNormalization (cudnnBatchNormalizationForwardTraining, cudnnBatchNormalizationForwardInference, and cudnnBatchNormalizationBackward). There are three things to be aware of:

1) We do not actually use cudnnBatchNormalizationForwardInference because we believe there is a bug in its implementation. We use cudnnBatchNormalizationForwardTraining, but set all the parameters related to maintaining running statistics to NULL
2) cudnnBatchNormalization* do not work with FP16, so you must use float or double
3) If a BatchNormalizationLayer has multiple in responses, it must satisfy the following requirements:
  - The number of in and out responses must be the same
  - The dimensions of each in response must be the same
If these are met, then the running statistics are calculated as follows: in the forward pass, compute the running statistics (mean and variance) for the first in response. We copy these statistics over to the running statistics for the second in response and then compute the running statistics for the second in response using the first in response. And so on. Because we store the running statistics for each in response separately, we can use the appropriate running statistics in the backward pass. Once we finish the backward pass, we set all the running statistics to the running statistics for the last in response.

This mechanism allows us to benefit from the running statistics for each input, but also lets us match the appropriate statistics between forward and backward passes. HOWEVER!! This means that BatchNormalizationLayer is not appropriate for multiple in responses for other uses cases that may require them. You have been warned.

PrincetonVision/marvin,PrincetonVision/marvin,PrincetonVision/marvin,PrincetonVision/marvin