<jupyter_start><jupyter_text>* The empty ipynb for you to start from in the repo: https://github.com/grantmlong/itds2019  Setting the stage<jupyter_code>import sys
sys.version # 3.6
# if running on colab, pytorch is already installed.
# if running locally, conda or pip install this in your conda environment:
# conda install pytorch torchvision -c pytorch
# OR
# pip3 install torch torchvision

# I'll be assuming python >=3.6 and torch 1.0.1 which are the colab defaults.
import torch
import torch.nn as nn
import torchvision.datasets as dsets
import torchvision.transforms as transforms
from torch.utils.data import TensorDataset, DataLoader
import torch.nn.functional as F
print(torch.__version__) # 1.0.1

%matplotlib inline

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import statsmodels.api as sm
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn import preprocessing<jupyter_output>1.0.0.dev20181130<jupyter_text>Torch and autograd basics Torch is a package that defines vectors, matrices, or in general "tensors". If you know numpy, you will not be surprised by any of these:<jupyter_code>a = torch.ones(3,3)
a
b = torch.arange(9).float().view(3,3)
b
(a+b)**2
b[:,0]
a.zero_() # operations with an underscore modify the Tensor in place.
a<jupyter_output><empty_output><jupyter_text>You can slice and dice tensors and they have roughly all tensor operations you expect equivalently to numpy, but with a bit more low level control. If you need more intro: https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.htmlsphx-glr-beginner-blitz-tensor-tutorial-pySo what's the big deal about pytorch?**autograd = automatic differentiation**.Every `torch.Tensor`, let's say `x`, has an important flag `requires_grad`. If this flag is set to True, pytorch will keep track of the graph of operations that happen with this tensor.When we finally arrive at some output (a scalar variable based on a sequence of operations on `x`), we can call `.backward()` on this output, to compute the gradient `d(output) / dx`. This gradient will end up in `x.grad`.<jupyter_code>x = torch.randn(2,2, requires_grad=True)
x
y=(x**2 + x)
z = y.sum()
z<jupyter_output><empty_output><jupyter_text>We know from high school math that the derivative `dz / dx[i,j]` = 2*x +1<jupyter_code>z.backward()
x.grad
2*x+1<jupyter_output><empty_output><jupyter_text>What about the intermediate variable y? Does it require a gradient?<jupyter_code>y.requires_grad<jupyter_output><empty_output><jupyter_text>However the gradient of y is not exposed, since it is an intermediary variable, the result of an operation on leaf variables. Leaf variables are inputs to the operations: the data X or the `Parameter`s of a neural network.More about autograd in the tutorial https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.htmlsphx-glr-beginner-blitz-autograd-tutorial-py and the docs https://pytorch.org/docs/stable/autograd.html In the lecture we talked about how derivatives and backpropagation (based on the chain rule of differentiation) play a central role in deep learning. You can now start to see how this autograd will be massively powerful to define neural networks with weights `W` and optimize them based on the gradients in `W.grad`.  Linear regression on a toy problem Let's try this for a simple linear mapping `y = W x `, where we want to optimize W:<jupyter_code>torch.manual_seed(23801)
x1 = torch.Tensor([1, 2, 3, -3, -2])
y = torch.Tensor ([3, 6, 9, -9, -6]).view(5,1)
x2 = torch.randn(5)
x = torch.stack([x1, x2], dim=1) # 5 x 2 input. 5 datapoints, 2 dimensions.
W = torch.randn(1,2, requires_grad=True)
# we start W at random initialization, the gradient will point us in the right direction.
print('x:\n', x)
print('y:\n', y)
print('W at random initialization: ', W)<jupyter_output>x:
 tensor([[ 1.0000, -0.9665],
        [ 2.0000, -0.4228],
        [ 3.0000,  0.2336],
        [-3.0000, -1.6804],
        [-2.0000,  1.6508]])
y:
 tensor([[ 3.],
        [ 6.],
        [ 9.],
        [-9.],
        [-6.]])
W at random initialization:  tensor([[-0.4394, -0.2656]], requires_grad=True)<jupyter_text>Take a look at x and y. What is the (linear) relationship?A: clearly the true relationship is y  = 3*x1 + 0 * x2.  So we can see that the optimal W = [3,0] Note that in notation of previous lecture 5, * the features are still $X_1, X_2$ concatenated as columns (rows = datapoints)* $W = [\beta_1, \beta_2]$* there is no $\beta_0$ or "intercept term" or in deep learning speak, "bias" term<jupyter_code>ypred = x @ W.t() # matrix multiply; (N x 2) * (2 x 1) -> N x 1
print('ypred:\n', ypred)
loss = ((ypred-y)**2).mean() # mean squared error = MSE
print('mse loss: ', loss.item())
loss.backward()
print('W grad:\n', W.grad)
# let's move W in that direction
W.data -= 0.1 * W.grad
# Now we will reset the gradient to zero.
# Do an in-place operation tensor operation `.zero_()` on the gradient W.grad.
W.grad.zero_()
print('W:\n', W)<jupyter_output>ypred:
 tensor([[-0.1827],
        [-0.7665],
        [-1.3802],
        [ 1.7644],
        [ 0.4404]], grad_fn=<MmBackward>)
mse loss:  64.20298767089844
W grad:
 tensor([[-37.2121,  -1.5780]])
W:
 tensor([[ 3.2818, -0.1078]], requires_grad=True)<jupyter_text>You can re-execute this cell above a couple of times and see how W goes close towards the optimal value of `[3,0]` (it may be a bit off because of the added noise).torch defines `Module`s which do two things: (a) they contain the learnable weight, and (b) define how they operate on an input tensor to give an output.In this case this would be a `Linear` layer, reducing 2D datapoints `x` to 1D output `y`. Ok doing this manually gives you insight what happens down to the gradienst. But usually we do not do these things manually, it would become very cumbersome if the net becomes more complex than the simple linear layer. pytorch gives us primitives to do the same: `net.zero_grad()` to clear the gradients, and for optimization you can do `optimizer.step()` to do a step of SGD.Again we will do 1 epoch.<jupyter_code>linear = nn.Linear(2,1, bias=False)
linear.weight.data.copy_(W) # we re-initialize the linear layer with the W we just found
ypred = linear(x)
ypred<jupyter_output><empty_output><jupyter_text>Now with a `torch.nn.Linear` layer you can do the same thing we just did:* forward propagate (=compute prediction), * compute MSE loss, and * backprop, then * update the linear layer's weight according to the gradient.Let's also change two things* do this in a loop, instead of re-executing the cell many times which was silly* use an "optimizer" which takes care of adding the gradient to the weight in `linear.weight`* (homework: break this down to the way we did it above, and check it matches.)<jupyter_code>torch.manual_seed(23801)
linear = nn.Linear(2,1, bias=False)
optimizer = torch.optim.SGD(linear.parameters(), lr=0.1) # do updates with `optimizer.step()`
# x, y defined above. In a real problem we would typically get different x, y "minibatches"
# of samples from a dataloader.
for i in range(10): # 10 optimization steps (gradient descent steps)
    ypred = linear(x)
    loss = ((ypred-y)**2).mean() # mean squared error = MSE
    print('mse loss:', loss.item())
    linear.zero_grad()
    loss.backward()
    optimizer.step()<jupyter_output>mse loss: 51.05780792236328
mse loss: 0.5673346519470215
mse loss: 0.12921813130378723
mse loss: 0.0681183710694313
mse loss: 0.03650308772921562
mse loss: 0.019565189257264137
mse loss: 0.01048668660223484
mse loss: 0.00562070170417428
mse loss: 0.0030126385390758514
mse loss: 0.0016147240530699492<jupyter_text>Now the real stuff: MNIST classification MNIST is a dataset of 50k handwritten digits (0-9) which is very commonly used in the deep learning community.It is small enough to work with locally (on your laptop) and without much hassle, and complex enough to do something interesting with. The loss we will optimize is  $$\mathcal{L(\theta)} = \sum_{x,y_{t} \in D} \ell(x,y_t; \theta)$$with $$\ell(x,y_t; \theta) = \text{NLL}(\hat{y}(x; \theta) || y_t)$$ the negative log likelihood loss. https://pytorch.org/docs/stable/nn.htmltorch.nn.NLLLossNow we'll set up a small neural network to learn a good $\hat{y}(x; \theta)$. The variable $\theta$ simply means all the neural network weights together, which we will update on each step. So per sample, the neural network output $\hat{y}(x; \theta)$ is the model's prediction of the probability of each class.  This is being compared to the true label $y_t$.The loss value is a metric of how far off the predictions are.We want to minimize this loss with SGD (stochastic gradient descent) We will follow the typical training procedure for a neural network which is as follows:* Define the neural network that has some learnable parameters (or weights) $\theta$.* Iterate over a dataset of inputs* Process input through the network* Compute the loss (how far is the output from being correct)* Propagate gradients back into the network’s parameters* Update the weights of the network, typically using SGD, i.e. the simple update rule: weight = weight - learning_rate * gradient<jupyter_code># let's download the MNIST data, if you do this locally and you downloaded before,
# you can change data paths to point to your existing files
train_dataset = dsets.MNIST(root='./MNISTdata',
                           train=True,
                           transform=transforms.ToTensor(),
                           download=True)

test_dataset = dsets.MNIST(root='./MNISTdata',
                           train=False,
                           transform=transforms.ToTensor())<jupyter_output>Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Processing...
Done!<jupyter_text>Dataset and DataLoader are abstractions to help us iterate over the data in random order. Let's look at the digits and their labels<jupyter_code>ix=129
x,y = train_dataset[ix]
plt.imshow(x.squeeze().numpy())
plt.colorbar()
print('label: y={}'.format(y))<jupyter_output>label: y=6<jupyter_text>Now let's define the dataloaders and train simple neural network like before.You'll recognize that the core is exactly the same: we do a forward pass, compute a loss, backpropagate the loss to compute the gradients, then let the optimizer update the weights.<jupyter_code># The neural network hyperparameters.
input_size    = 784   # The MNIST image size = 28 x 28 = 784
hidden_size   = 100   # The number of nodes at the hidden layer
num_classes   = 10    # The number of output classes. In this case, from 0 to 9
num_epochs    = 5     # The number of times entire dataset is trained
batch_size    = 100   # The number of samples per minibatch
learning_rate = 1.0   # SGD step size
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                          batch_size=batch_size,
                                          shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)
len(train_loader)
device = torch.device('cpu') # if on gpu-enabled machine, change to torch.device('cuda')

#  define simple neural network with 1 hidden layer.
class Net(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(Net, self).__init__()                    # Inherited from the parent class nn.Module
        self.fc1 = nn.Linear(input_size, hidden_size)  # 1st Full-Connected Layer: 784 (input data) -> 500 (hidden node)
        self.relu = nn.ReLU()                          # Non-Linear ReLU Layer: max(0,x)
        self.fc2 = nn.Linear(hidden_size, num_classes) # 2nd Full-Connected Layer: 500 (hidden node) -> 10 (output class)
    
    def forward(self, x):                              # Forward pass: stacking each layer together
        x = x.view(x.size(0), -1) # flatten (bs x 1 x 28 x 28) -> (bs x 784)
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out
def test(net, dl):
    right, tot = 0, 0
    net.eval() # Set dropout and possibly other modules in eval mode.
    for x,y in dl:
        ypred = net(x).argmax(dim=1) # select index of maximal score
        right += (ypred == y).sum().item()
        tot   += x.size(0)
    return 1.* right / tot
# create the net based on this class definition
net = Net(input_size, hidden_size, num_classes).to(device)
# define the optimizer
optimizer = torch.optim.SGD(net.parameters(), lr=learning_rate)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.5)
print('Before training: {:.1f}% test accuracy'.format(100*test(net, test_loader)))
for epoch in range(num_epochs):
    scheduler.step()
    for i, (x,y_target) in enumerate(train_loader):
        x, y_target = x.to(device), y_target.to(device)
        y_probs = F.log_softmax(net(x), dim=1)
        output = F.nll_loss(y_probs, y_target)
        output.backward()
        optimizer.step()
        net.zero_grad()
    print('After epoch {}: {:.1f}% test accuracy'.format(epoch, 100*test(net, test_loader)))
print('End of training: {:.1f}% train accuracy'.format(100*test(net, train_loader)))
# show the prediction on some samples.
ix=1234
x,y = test_dataset[ix]
plt.imshow(x.squeeze().numpy())
print('Ground truth label: y={}'.format(y))
y_probs = F.softmax(net(x), dim=1)
print ('Model probabilities: ') 
print(' / '.join(['{}: {:.3f}'.format(k,v) 
                  for k,v in zip(range(10), y_probs.squeeze().tolist()) ] ))
print('Model prediction: ', y_probs.argmax(1))<jupyter_output>Ground truth label: y=8
Model probabilities: 
0: 0.047 / 1: 0.000 / 2: 0.206 / 3: 0.018 / 4: 0.000 / 5: 0.012 / 6: 0.027 / 7: 0.000 / 8: 0.690 / 9: 0.000
Model prediction:  tensor([8])<jupyter_text>Voila and that's how it's done. You can play around to figure out the answer to some more questions:* What do we get back from `net.parameters()`: which trainable weights and biases does the network have now? * How many total parameters?* Can you add another hidden layer to the class definition of `Net`? How does the performance change (watch both test & train)?* Can you add a dropout layer after the first and second hidden layer? How does performance change now? See https://pytorch.org/docs/master/nn.htmltorch.nn.Dropout Some more fundamental questions for you to understand what's going on in the optimization loop:* what does softmax do? (compute softmax of a random vector, then sum the output, it will sum to 1)* what is its purpose? (read the docs)* what does nll_loss compute? can you manually compute it? Important questions to revisit after you learned about overfitting:* How many parameters (weights) does our classifier neural network have? How many  datapoints did we train on?* Old-skool machine learning rule of thumb is: you can optimize about as many parameters (weights) as you have datapoints before you can memorize the dataset (thus overfit heavily). Are we close to the limit?* Does the model overfit? We are measuring this by splitting off a heldout set after loading the data on which you do not train, and measure performance on the heldout set.* In deep neural networks you can easily have way more parameters than datapoints. Is overfitting an issue for neural networks in general? Now we used a simple flat neural network which looks at the image as a flat vector, without awareness of the 2D structure or which pixels neighbor each other.A convolutional neural network is an architecture that takes the 2D structure of the image into account by sliding a kernel over all the different locations in the image. This kind of neural network has been very succesful in image recognition [1] and speech recognition [2,3].Pytorch and other deep learning toolboxes are designed to deal with this kind of data and with convolutional neural networks just as easily as with flat data. Try swapping out the network above for a convolutional neural network, see for example the pytorch tutorial [4].[1] https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks[2] http://www.cs.toronto.edu/~asamir/papers/icassp13_cnn.pdf[3] https://arxiv.org/abs/1509.08967[4] https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.htmlsphx-glr-beginner-blitz-neural-networks-tutorial-py[5] https://colab.research.google.com/drive/1jxUPzMsAkBboHMQtGyfv5M5c7hU8Ss2c<jupyter_code># have fun exploring<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup