<jupyter_start><jupyter_text>PyTorch TutorialMILA, November 2017By <NAME>  An introduction to the PyTorch neural network library `torch.nn` & `torch.optim`<jupyter_code>import numpy as np
from __future__ import print_function
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.init as init
import torch.nn.functional as F
from torch.autograd import Variable<jupyter_output><empty_output><jupyter_text>torch.nnNeural networks can be constructed using the `torch.nn` package.Provides pretty much all neural network related functionalities such as :1. Linear layers - `nn.Linear`, `nn.Bilinear`2. Convolution Layers - `nn.Conv1d`, `nn.Conv2d`, `nn.Conv3d`, `nn.ConvTranspose2d`3. Nonlinearities - `nn.Sigmoid`, `nn.Tanh`, `nn.ReLU`, `nn.LeakyReLU`4. Pooling Layers - `nn.MaxPool1d`, `nn.AveragePool2d`4. Recurrent Networks - `nn.LSTM`, `nn.GRU`5. Normalization - `nn.BatchNorm2d`6. Dropout - `nn.Dropout`, `nn.Dropout2d`7. Embedding - `nn.Embedding`8. Loss Functions - `nn.MSELoss`, `nn.CrossEntropyLoss`, `nn.NLLLoss`Instances of these classes will have an `__call__` function built-in that can be used to run an input through the layer.  Linear, Bilinear & Nonlinearities<jupyter_code>x = Variable(torch.randn(32, 10))
y = Variable(torch.randn(32, 30))

sigmoid = nn.Sigmoid()

linear = nn.Linear(in_features=10, out_features=20, bias=True)
output_linear = linear(x)
print('Linear output size : ', output_linear.size())

bilinear = nn.Bilinear(in1_features=10, in2_features=30, out_features=50, bias=True)
output_bilinear = bilinear(x, y)
print('Bilinear output size : ', output_bilinear.size())<jupyter_output>Linear output size :  torch.Size([32, 20])
Bilinear output size :  torch.Size([32, 50])<jupyter_text>Convolution, BatchNorm & Pooling Layers<jupyter_code>x = Variable(torch.randn(10, 3, 28, 28))

conv = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=(3, 3), stride=1, padding=1, bias=True)
bn = nn.BatchNorm2d(num_features=32)
pool = nn.MaxPool2d(kernel_size=(2, 2), stride=2)

output_conv = bn(conv(x))
outpout_pool = pool(conv(x))

print('Conv output size : ', output_conv.size())
print('Pool output size : ', outpout_pool.size())<jupyter_output>Conv output size :  torch.Size([10, 32, 28, 28])
Pool output size :  torch.Size([10, 32, 14, 14])<jupyter_text>Recurrent, Embedding & Dropout Layers<jupyter_code>inputs = [[1, 2, 3], [1, 0, 4], [1, 2, 4], [1, 4, 0], [1, 3, 3]]
x = Variable(torch.LongTensor(inputs))

embedding = nn.Embedding(num_embeddings=5, embedding_dim=20, padding_idx=1)
drop = nn.Dropout(p=0.5)
gru = nn.GRU(input_size=20, hidden_size=50, num_layers=2, batch_first=True, bidirectional=True, dropout=0.3)

emb = drop(embedding(x))
gru_h, gru_h_t = gru(emb)

print('Embedding size : ', emb.size())
print('GRU hidden states size : ', gru_h.size())
print('GRU last hidden state size : ', gru_h_t.size())<jupyter_output>Embedding size :  torch.Size([5, 3, 20])
GRU hidden states size :  torch.Size([5, 3, 100])
GRU last hidden state size :  torch.Size([4, 5, 50])<jupyter_text>torch.nn.functionalUsing the above classes requires defining an instance of the class and then running inputs through the instance.The functional API provides users a way to use these classes in a `functional` way. Such as`import torch.nn.functional as F`1. Linear layers - `F.linear(input=x, weight=W, bias=b)`2. Convolution Layers - `F.conv2d(input=x, weight=W, bias=b, stride=1, padding=0, dilation=1, groups=1)`3. Nonlinearities - `F.sigmoid(x), F.tanh(x), F.relu(x), F.softmax(x)`4. Dropout - `F.dropout(x, p=0.5, training=True)`  A few examples of the functional API<jupyter_code>x = Variable(torch.randn(10, 3, 28, 28))
filters = Variable(torch.randn(32, 3, 3, 3))
conv_out = F.relu(F.dropout(F.conv2d(input=x, weight=filters, padding=1), p=0.5, training=True))

print('Conv output size : ', conv_out.size())<jupyter_output>Conv output size :  torch.Size([10, 32, 28, 28])<jupyter_text>torch.nn.initProvides a set of functions for standard weight initialization techniques`import torch.nn.init as init`1. Calculate the gain of a layer based on the activation function - `init.calculate_gain('sigmoid')`2. Uniform init - `init.uniform(tensor, low, high)`3. Xavier uniform - `init.xavier_uniform(tensor, gain=init.calculate_gain('sigmoid'))`4. Xavier normal - `init.xavier_normal(tensor, gain=init.calculate_gain('tanh'))`5. Orthogonal - `init.orthogonal(tensor, gain=init.calculate_gain('tanh'))`6. Kaiming normal - `init.kaiming_normal(tensor, mode='fan_in')`  Initializing convolution kernels<jupyter_code>conv_layer = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=(3, 3), padding=1)
for k,v in conv_layer.named_parameters():
    if k == 'weight':
        init.kaiming_normal(v)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup