LayerNorm initialization
LayerNorm initialization

The LayerNorm gamma and beta should be initialized by .fill_(1.0) and .zero_().

reference links:

https://github.com/tensorflow/tensorflow/blob/989e78c412a7e0f5361d4d7dfdfb230c8136e749/tensorflow/contrib/layers/python/layers/layers.py#L2298

https://github.com/tensorflow/tensorflow/blob/989e78c412a7e0f5361d4d7dfdfb230c8136e749/tensorflow/contrib/layers/python/layers/layers.py#L2308
huggingface/transformers,huggingface/transformers,huggingface/transformers,huggingface/pytorch-transformers,huggingface/transformers