<jupyter_start><jupyter_text>*Data Science Unit 4 Sprint 2 Assignment 1*Use TensorFlow Keras & a sample of the [Quickdraw dataset](https://github.com/googlecreativelab/quickdraw-dataset) to build a sketch classification model. The dataset has been sampled to only 10 classes and 10000 observations per class.  Objective- Build a baseline classification model then run a few experiments with different optimizers and learning rates. - Don't forgot to [**switch to GPU on if you're running your notebook on Colab!**](https://colab.research.google.com/notebooks/gpu.ipynb)  Load Your Data<jupyter_code># imports in first cell 
import seaborn as sns
import pandas as pd
import numpy as np
import tensorflow as tf 
import matplotlib.pyplot as plt
from sklearn.utils import shuffle

# use Sequential to build out your model
from tensorflow.keras.models import Sequential

# Dense layer is used for Fully Connected Forward Feeding networks
from tensorflow.keras.layers import Dense
# load in data set
data = np.load('quickdraw10.npz')
X = data['arr_0']
y = data['arr_1']


print(X.shape)
print(y.shape)
class_names = ['apple',
             'anvil',
             'airplane',
             'banana',
             'The Eiffel Tower',
             'The Mona Lisa',
             'The Great Wall of China',
             'alarm clock',
             'ant',
             'asparagus']
# these are your 10 unique images
plt.figure(figsize=(10,5))
start = 0

for num, name in enumerate(class_names):
    plt.subplot(2,5, num+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(X[start].reshape(28,28), cmap=plt.cm.binary)
    plt.xlabel(name)
    start += 10000
plt.show()<jupyter_output><empty_output><jupyter_text>Say hello to the Quickdraw dataset. You'll be using this dataset a lot this week as an alternative to the mnist which we'll be using in the guided projects. The nice thing about this dataset is that it's simple, which allows us to focus on our model, it's various components, and gradually coming to a better understand of how to build neural networks without worrying about cleaning and preping our image data too much.<jupyter_code># always a good idea to shuffle your dataset 
X, y = shuffle(X, y)<jupyter_output><empty_output><jupyter_text>----- Build Your Baseline ModelMake sure that you- **Determine** the dimensionality of your input data by investigating **X**- **Normalize** your input data to values between 0 and 1 - **Determine** the number of neurons in your output layer by investigating **Y**- **Select** `sparse_categorical_crossentropy` as your loss function.- **Select** `sgd` as your optimizer.- **Add** 3 hidden layers to your model with the following number of nodes    - h1 has 500 nodes    - h2 has 250 nodes    - h3 has 100 nodes    - **Set** epochs to 20 - **Use** the `validation_split` command to automatically create a training / validation dataset from within the model, so you don't have to do it yourself.    -  Specify a percentage such as .2 in your fit statement.  Not sure what the various parameters are for or what what values to assign to them?- Reference the guided project notebook for Sprint 2 Module 1- Reference the [**Keras documentation**](https://keras.io/api/)- Google other examples- Discuss your results with classmates<jupyter_code>len(np.unique(y))
# get dim of image row vectors and save to imput_dim

# get number of unique labels and save to n_output_nodels

# normalize image data to values between 0 and 1 (by dividing by max pixel value)


max_pixel_val = X.max()

X_scaled = X/max_pixel_val

n_imgs = X_scaled.shape[0]

input_dims = n_imgs

n_output_nodels = len(np.unique(y))

X_scaled.min(), X_scaled.max()
# a check on your data prep 
assert  X_scaled.max(), "Max pixel value should be 1.0, make sure you normalize your data"
# instantiate a sequential object and call it model, then add layers to your model
model = Sequential()

model.add(Dense(500, activation='sigmoid', input_dim=784))
model.add(Dense(250, activation='sigmoid'))
model.add(Dense(100, activation='sigmoid'))
model.add(Dense(n_output_nodels, activation='softmax'))

# add a compile layer but don't fit your model yet 
model.compile(optimizer= 'sgd',
              loss= 'sparse_categorical_crossentropy',
              metrics= ['accuracy'])

# a check on your model architecture 
n_layers = len(model.get_config()["layers"])
assert n_layers == 5, "You should have 5 layers: input, h1, h2, h3, and output"
# check out your model summary 
model.summary()
# fit your model and save training resuts to history 

history = model.fit(X_scaled, y,
                    epochs=20, 
                    validation_split=0.2)<jupyter_output>Epoch 1/20
2500/2500 [==============================] - 11s 4ms/step - loss: 2.3047 - accuracy: 0.1155 - val_loss: 2.2748 - val_accuracy: 0.1884
Epoch 2/20
2500/2500 [==============================] - 9s 3ms/step - loss: 2.2549 - accuracy: 0.2341 - val_loss: 2.1291 - val_accuracy: 0.4333
Epoch 3/20
2500/2500 [==============================] - 9s 3ms/step - loss: 2.0168 - accuracy: 0.4049 - val_loss: 1.7282 - val_accuracy: 0.4612
Epoch 4/20
2500/2500 [==============================] - 9s 3ms/step - loss: 1.6500 - accuracy: 0.4904 - val_loss: 1.4604 - val_accuracy: 0.5295
Epoch 5/20
2500/2500 [==============================] - 9s 3ms/step - loss: 1.4112 - accuracy: 0.5668 - val_loss: 1.2470 - val_accuracy: 0.6340
Epoch 6/20
2500/2500 [==============================] - 9s 3ms/step - loss: 1.2052 - accuracy: 0.6350 - val_loss: 1.0948 - val_accuracy: 0.6733
Epoch 7/20
2500/2500 [==============================] - 9s 3ms/step - loss: 1.0780 - accuracy: 0.6684 - val_loss: 1.0190 - val_accuracy[...]<jupyter_text>---- Visualize the results- Move results in `history` in a dataframe - Use [**Seaborn**](https://seaborn.pydata.org/generated/seaborn.lineplot.html) to create lineplots for both loss and accuracy by epoch. - Analyze the results and write a couple of obsverations. At what point should we have stopped training the model and why?<jupyter_code>df = pd.DataFrame(data=history.history['accuracy'])
epochs = [i for i in range(len(history.history['loss']))]
sns.lineplot(epochs, history.history['loss'], label='train')
sns.lineplot(epochs, history.history['val_loss'], label= 'test');
# a check on our model training
assert df.shape[0] == 20, "df should have the training results from 20 epochs"
# use the plotting method in your dataframe to plot the modeling results 
sns.lineplot(epochs, history.history['loss'], label='train')
sns.lineplot(epochs, history.history['val_loss'], label= 'test');<jupyter_output>/usr/local/lib/python3.7/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.7/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>Observations Based on the plot of the training and validation loss answer the following questions.  **Was our model able to learn over the course of the 20 epochs? Why or why not?** Yes, but very minimal improvement after 12.5 epochs.  The loss did continue to drop but at minimal rates. **Is our model overfitting? Why or why not?** No it is not.  If it were there would be major discrepancies between the training accuracy and test accuracy.  In this case, these are nearly identical in accuracy and loss. **Could the model score benefit from additional epochs? Why or why not?** It could benefit but it would only yield about another 1% or so in accuracy so it probably isn't worth the extra time. ----- Change OptimizersLet's compare model performance between difference optimizers. - Build a new model, identical as the last one but using `adam` for the optimizer. - Visualize the training results just as we did for the last model. - Save modeling results to adam_history so we don't erase the results from the previous model run<jupyter_code># train same model as above but use sgd if you used adam previously ( or use adam if you used sgd previously)
model = Sequential()

model.add(Dense(500, activation='sigmoid', input_dim=784))
model.add(Dense(250, activation='sigmoid'))
model.add(Dense(100, activation='sigmoid'))
model.add(Dense(n_output_nodels, activation='softmax'))

# add a compile layer but don't fit your model yet 
model.compile(optimizer= 'adam',
              loss= 'sparse_categorical_crossentropy',
              metrics= ['accuracy'])

# a check that we're using the correct optimizer in this model
opt_name = model.optimizer.get_config()["name"]
assert opt_name == "Adam", "you need to use adam for the optimizer in this model."
model.summary()
adam_history = model.fit(X_scaled, y,
                    epochs=20, 
                    validation_split=0.2)
sns.lineplot(epochs, adam_history.history['loss'], label='train')
sns.lineplot(epochs, adam_history.history['val_loss'], label= 'test');<jupyter_output>/usr/local/lib/python3.7/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning
/usr/local/lib/python3.7/dist-packages/seaborn/_decorators.py:43: FutureWarning: Pass the following variables as keyword args: x, y. From version 0.12, the only valid positional argument will be `data`, and passing other arguments without an explicit keyword will result in an error or misinterpretation.
  FutureWarning<jupyter_text>Observations This plot should look very different from the previous plot. Based on the plot of the training and validation loss answer the following questions.  **Is our model overfitting? Why or why not?** Yes it is! The val_accuracy actually started to "bounce" at just 5 epochs and after 20 was actually lower than at 5.  It also had a very high val_loss when comparing to the training accuracy and loss. **Could our model's performance improve by training on more than 20 epochs? Why or why not?** No it definitely would not.  It could actually improve by dropping back down to around 3-4 epochs max.  That's where it performed best for both the training and test sets without massive overfitting. **Assuming that you want to stick to this model architecture, what can you do to avoid overfitting?** Drop the number of epochs down to 3 or 4.  This minimizes loss and maximizes accuracy without overfitting. ------ Conclusion What you have observed by comparing the two models that are identitical, except for the optimizer, is that the choice of the optimizer can have a very big influence in the training outcome of ML models in general and in a neural networks in particuar. You might not actually know - why changing the optimizer from Stocastic Gradient Descent over to Adam made a difference.- how Gradient Desecent works - or that Adam is known as Adaptive Gradient Descent (i.e. is it a different version of ordinary Gradient Descent). But that's ok - tomorrow's lesson will be a deep dive into Gradient Descent. You'll learn the theory of Gradient Descent, we'll code up Gradient Descent from scratch, and we'll talk about how Adam is different from ordinary Gradient Descent.  ----- Preparation for Tomorrow In Preparation for tomorrow, you might need to watch a few videos. I say might because you may or may not already be comfortable with multi-variate calculus.  Theory of Calculus The theory of Gradient Descent rest on [**the derivative from Calculus**](https://www.youtube.com/watch?v=WUvTyaaNkzM). If you've never taken a Calculus course before, or are a bit rusty, definitely watch this video in preparation for tomorrow's lesson. The take away here is to conceptually understand the derivative, you won't be asked to actually calculate any derivatives by hand.  The Gradient - a multi-dimensional derivativeOnce you've watched the previous video, you are now set up to  understand how to conceptualize a derivative in an N-dimensional space (where N is any number ranging from 2, 3, 4, ..., all the way to a very large number of dimensions N ) [**This video visually explains the multi-dimensional derivative called the Gradient visually**](https://www.youtube.com/watch?v=GkB4vW16QHI) - and that's the take away. This point of this video is to help you understand the Gradient visually using 3D surfaces. You should also know that the Gradient is made of up partial derivatives, more on this tomorrow.  Contour Maps It is very common to visual 3D surfaces as 2D contour maps - and we'll be making a lot of use of contour maps to understand Gradient Descent. Watch this video in order to [**understand the relationship between 3D surfaces and 2D contour maps**](https://www.youtube.com/watch?v=acdX4YamDtU) The Calculus of Backpropagation Unlike standard Sklearn ML models, neural networks use not just Gradient Descent but also something called Backpropagatin in order to learn from the data. [**In order to understand how backpropagation works, you need to understand the Chain Rule in Calculus**](https://www.youtube.com/watch?v=acdX4YamDtU). The take away here is to understand how a partial derivative can be decomposed into a product of multiple derivatives.  ------ Stretch Goals This section is optional!Only after you've completed the above work and watched the recommended videos, are you then encouraged to experiment with building a few more models and analyze their results. Here are some suggestions to help you get started. - Train the same model using the Sigmoid activation fucntion and the Relu activation function and note the difference in their learning outcomes.  [**Keras docs on activation fucntions**](https://keras.io/api/layers/activations/). We'll cover the Relu activation function in Sprint 2 Module 3.- Train the same model using Normalized and non-Normalized data and note the difference in their learning outcomes. The objective here is to give you an oppertunity to get more practice with the Keras API (i.e. building models) and running a couple of experiments to help set you up for future lessons. In order words, we'll be discussing different activation functions and the affect of Normalized data on model training.<jupyter_code><jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup