<jupyter_start><jupyter_text>![MLU Logo](../data/MLU_Logo.png)  Machine Learning Accelerator - Tabular Data - Lecture 3 MXNet and Gluon1. MXNet: NDarrays and Autograd2. Gluon: Building a Neural Network  1. MXNet: NDarrays and AutogradGo to topThis tutorial is following the notebooks under the MXNet crash course [here](https://mxnet.apache.org/api/python/docs/tutorials/packages/ndarray/01-ndarray-intro.html).To get started, let’s import the ndarray package (nd is shortform) from MXNet.<jupyter_code>from mxnet import nd<jupyter_output><empty_output><jupyter_text>Next, let’s see how to create a 2D array (also called a matrix) with values from two sets of numbers: 1, 2, 3 and 4, 5, 6. This might also be referred to as a tuple of a tuple of integers.<jupyter_code>nd.array(((1,2,3),(5,6,7)))<jupyter_output><empty_output><jupyter_text>We can also create a very simple matrix with the same shape (2 rows by 3 columns), but fill it with 1s.<jupyter_code>x = nd.ones((2,3))
x<jupyter_output><empty_output><jupyter_text>Often we’ll want to create arrays whose values are sampled randomly. For example, sampling values uniformly between -1 and 1. Here we create the same shape, but with random sampling.<jupyter_code>y = nd.random.uniform(-1,1,(2,3))
y<jupyter_output><empty_output><jupyter_text>You can also fill an array of a given shape with a given value, such as 2.0.<jupyter_code>x = nd.full((2,3), 2.0)
x<jupyter_output><empty_output><jupyter_text>As with NumPy, the dimensions of each NDArray are accessible by accessing the .shape attribute. We can also query its size, which is equal to the product of the components of the shape. In addition, .dtype tells the data type of the stored values.<jupyter_code>(x.shape, x.size, x.dtype)<jupyter_output><empty_output><jupyter_text>OperationsNDArray supports a large number of standard mathematical operations. Such as element-wise multiplication:<jupyter_code>x * y<jupyter_output><empty_output><jupyter_text>Exponentiation:<jupyter_code>y.exp()<jupyter_output><empty_output><jupyter_text>And grab a matrix’s transpose to compute a proper matrix-matrix product:<jupyter_code>nd.dot(x, y.T)<jupyter_output><empty_output><jupyter_text>IndexingMXNet NDArrays support slicing in all the ridiculous ways you might imagine accessing your data. Here’s an example of reading a particular element, which returns a 1D array with shape (1,).<jupyter_code>y[1,2]<jupyter_output><empty_output><jupyter_text>Read the second and third columns from y.<jupyter_code>y[:,1:3]<jupyter_output><empty_output><jupyter_text>and writing to a specific element<jupyter_code>y[:,1:3] = 2
y<jupyter_output><empty_output><jupyter_text>Multi-dimensional slicing is also supported.<jupyter_code>y[1:2,0:2] = 4
y<jupyter_output><empty_output><jupyter_text>Automatic differentiation with autogradWe train models to get better and better as a function of experience. __Usually, getting better means minimizing a loss function__. To achieve this goal, we often iteratively compute the gradient of the loss with respect to weights and then update the weights accordingly. While the gradient calculations are straightforward through a chain rule, for complex models, working it out by hand can be a pain.__Before diving deep into the model training, let’s go through how MXNet’s autograd package expedites this work by automatically calculating derivatives.____Basic usage__Let’s first import the autograd package.<jupyter_code>from mxnet import nd
from mxnet import autograd<jupyter_output><empty_output><jupyter_text>As a toy example, let’s say that we are interested in differentiating a function $f(x) = 0.6x^2$ with respect to parameter $x$. We can start by assigning an intial value of x.<jupyter_code>x = nd.array([[1, 2], [3, 4]])
x<jupyter_output><empty_output><jupyter_text>Once we compute the gradient of $f(x)$ with respect to $x$, __we’ll need a place to store it__.In MXNet, we can tell an NDArray that we plan to store a gradient by invoking its attach_grad method.<jupyter_code>x.attach_grad()<jupyter_output><empty_output><jupyter_text>Now we’re going to define the function $y=f(x)$. To let MXNet store 𝑦, so that we can compute gradients later, we need to put the definition inside a autograd.record() scope.<jupyter_code>with autograd.record():
    y = 0.6 * x * x<jupyter_output><empty_output><jupyter_text>Let’s invoke back propagation (backprop) by calling y.backward(). When 𝑦 has more than one entry, y.backward() is equivalent to y.sum().backward().<jupyter_code>y.backward()<jupyter_output><empty_output><jupyter_text>Now, let’s see if this is the expected output. Note that $y=0.6x^2$ and $dx/dy = 1.2x$ which should be [[1.2, 2.4],[3.6, 4.8]]. Let’s check the automatically computed results:<jupyter_code>x.grad<jupyter_output><empty_output><jupyter_text>2. Gluon: Building a Neural NetworkGo to top  Implement a network with sequential mode Let's implement a simple neural network with two hidden layers of size 64 and 128 using the sequential mode. We will have 5 inputs, 1 output and some dropouts between the layers.<jupyter_code>from mxnet import nd
from mxnet.gluon import nn

net = nn.Sequential()

net.add(nn.Dense(64 ,activation='relu'),    # Layer 1
        nn.Dropout(.4),                     # Apply random 40% dropout to layer 1
        nn.Dense(128, activation='relu'),   # Layer 2
        nn.Dropout(.3),                     # Apply random 30% dropout to layer 2
        nn.Dense(1, activation='sigmoid'))  # Output layer
net<jupyter_output><empty_output><jupyter_text>Initialize weights<jupyter_code>net.initialize()<jupyter_output><empty_output><jupyter_text>Let's look at our layers and dropouts on them. We can easily access them with net[index]<jupyter_code>print(net[0])
print(net[1])
print(net[2])
print(net[3])
print(net[4])<jupyter_output>Dense(None -> 64, Activation(relu))
Dropout(p = 0.4, axes=())
Dense(None -> 128, Activation(relu))
Dropout(p = 0.3, axes=())
Dense(None -> 1, Activation(sigmoid))<jupyter_text>Let's send a batch of data to this network (batch size is 4 in this case)__Important note:__ Weights are initialized after you pass some data through network. This is because network input size is learned from your input data.<jupyter_code># Input shape is (batch_size, data lenght)
x = nd.random.uniform(shape=(4, 5))
y = net(x)
print("Random input data with shape" , x.shape)
print(x)
print("Output shape:", y.shape)
print("Network output: ", y)<jupyter_output>Output shape: (4, 1)
Network output:  
[[0.50008506]
 [0.4998414 ]
 [0.5009381 ]
 [0.5004806 ]]
<NDArray 4x1 @cpu(0)><jupyter_text>We can also see the initialized weights for each layer.<jupyter_code>print(net[0].weight.data().shape, net[0].bias.data().shape)
print(net[0].weight.data(), net[0].bias.data())<jupyter_output>(64, 5) (64,)

[[ 0.05958354  0.04705103 -0.06005495 -0.02276454 -0.0578019 ]
 [ 0.02074406 -0.06716943 -0.01844618  0.04656678  0.06400172]
 [ 0.03894195 -0.05035089  0.0518017   0.05181222  0.06700657]
 [-0.00369488  0.0418822   0.0421275  -0.00539289  0.00286685]
 [ 0.03927409  0.02504314 -0.05344158  0.03088857  0.01958894]
 [ 0.01148278 -0.04993054  0.00523225  0.06225365  0.03620619]
 [ 0.00305876 -0.05517294 -0.01194733 -0.00369594 -0.03296221]
 [-0.04391347  0.03839272  0.03316854 -0.00613896 -0.03968295]
 [ 0.00958075 -0.05106945 -0.06736943 -0.02462026  0.01646897]
 [-0.04904552  0.0156934  -0.03887501  0.01637076 -0.01589154]
 [ 0.06212472  0.05636378  0.02545484 -0.007007   -0.0196689 ]
 [ 0.01582889 -0.00881553  0.0563288   0.02766836 -0.05610075]
 [-0.06156844  0.06577327  0.02334734  0.0214396  -0.01161692]
 [ 0.06960588  0.03084543  0.06055803 -0.06998399 -0.05206258]
 [-0.02767344  0.06986568 -0.04945417 -0.03694754 -0.0570726 ]
 [-0.0144787  -0.04392357 -0.01569249 -0[...]<jupyter_text>Implement the network flexibly:In nn.Sequential, MXNet will automatically construct the forward function that sequentially executes added layers. Now let’s introduce another way to construct a network with a flexible forward function. This second approach gives you more control and you can do things like having parallel branches, skip connections and different types of connections between layers. These are all advanced concepts that we won't cover in this class.To do it, we create a subclass of nn.Block and implement two methods:* __init()__ create the layers* __forward()__ define the forward function.<jupyter_code>class MixMLP(nn.Block):
    def __init__(self, **kwargs):
        # Run `nn.Block`'s init method
        super(MixMLP, self).__init__(**kwargs)
        self.blk = nn.Sequential()
        self.blk.add(nn.Dense(64, activation='relu'),   # Layer 1
                     nn.Dropout(.4),                    # Apply random 40% dropout to layer 1
                     nn.Dense(128, activation='relu'),  # Layer 2
                     nn.Dropout(.3),                    # Apply random 30% dropout to layer 2
                     nn.Dense(1, activation='sigmoid')  # Output layer
                    )
    def forward(self, x):
        y = self.blk(x)
        return y

net = MixMLP()
net<jupyter_output><empty_output><jupyter_text>In the sequential chaining approach, we can only add instances with nn.Block as the base class and then run them in a forward pass. In this example, we used print to get the intermediate results and nd.relu to apply relu activation. So this approach provides a more flexible way to define the forward function.The usage of net is similar as before.<jupyter_code>net.initialize()
# Input shape is (batch_size, data lenght)
x = nd.random.uniform(shape=(4, 5))
net(x)
# CLean up work space
!rm -r ../../__MACOSX<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup