<jupyter_start><jupyter_text>Fully Connected Neural Networks This tutorial demonstrates the usage of fully connected neural-networks to classify 2-dimensional synthetic data. We implement the network using Keras and TensorFlow. Keras and TensorFlow are powerful frameworks that allow you to build many  deep-learning-based applications.The goals of this tutorial are:  - Introduce Keras and TensorFlow to implement fully connected neural network.  - Illustrate the non-linear decision rules that multi-layer neural networks can achieve.         Importing Python Libraries<jupyter_code>%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

import tensorflow as tf

physical_devices = tf.config.experimental.list_physical_devices('GPU')
tf.config.experimental.set_memory_growth(physical_devices[0], True)

from tensorflow.keras.utils import to_categorical # Function to convert labels to one-hot encoding
from tensorflow.keras.models import Model 
from tensorflow.keras.layers import Dense, Activation,Input 
from tensorflow.keras.optimizers import SGD

import sys
## Adding dataset_utils to the system path so we can import it
MY_UTILS_PATH = '../Modules/'
if not MY_UTILS_PATH in sys.path:
    sys.path.append(MY_UTILS_PATH)
import dataset_utils<jupyter_output><empty_output><jupyter_text>Loading and Displaying a Synthetic 2D Dataset<jupyter_code>#Generating a synthetic dataset with four classes
nsamples = 300

data = dataset_utils.corners(N = nsamples,scale = 0.5,gapwidth = 3,cornerwidth = 4) # linear dataset
#data = dataset_utils.crescent_full_moon(N = nsamples,r1 = 5,r2 = 10,r3 = 15) #non-linear dataset

# Separating features (X) and labels (Y)
Y = data[:,-1].astype(int) # Labels
X = data[:,:-1] # Features

#Number of classes
k = np.unique(Y).size

colors = np.array(['r','y','b','g'])
#Displaying the dataset
plt.scatter(X[:,0],X[:,1],c=Y)

plt.xlabel("x1")
plt.ylabel("x2")
plt.grid()
plt.title("Dataset")
plt.show()<jupyter_output><empty_output><jupyter_text>Train, Validation and Test Sets Split<jupyter_code># First we will shuffle the samples
indexes = np.arange(X.shape[0])
np.random.shuffle(indexes)
X = X[indexes,:]
Y = Y[indexes]

# Then, we split our data into train/val/test sets
train_split = np.int(0.5*Y.size)
val_split = np.int(0.75*Y.size)

X_train = X[:train_split,:]
Y_train = Y[:train_split]

X_val = X[train_split:val_split,:]
Y_val = Y[train_split:val_split]

X_test = X[val_split:,:]
Y_test = Y[val_split:]<jupyter_output><empty_output><jupyter_text>Data Normalization<jupyter_code># min-max data normalization
x_train_min = X_train.min(axis = 0, keepdims = True)
x_train_max = X_train.max(axis = 0, keepdims = True)

X_train = (X_train - x_train_min)/(x_train_max - x_train_min)
X_val = (X_val - x_train_min)/(x_train_max - x_train_min)
X_test = (X_test - x_train_min)/(x_train_max - x_train_min)


#####################################################################################
#Activity suggestion:
# Change the min-max normalization above by standardization ((X - mean)/(std))
####################################################################################


#Displaying the normalized train set
plt.scatter(X_train[:,0],X_train[:,1],c=Y_train)
plt.xlabel("x1")
plt.ylabel("x2")
plt.grid()
plt.title("Normalized Train Set")
plt.show()<jupyter_output><empty_output><jupyter_text>Representing Labels using one-hot-ecoding<jupyter_code>Y_oh_train = to_categorical(Y_train, k) 
Y_oh_val = to_categorical(Y_val, k) 
Y_oh_test = to_categorical(Y_test, k)
# Displaying the 5 first elemnts
print('Y_train[:5]:')
print(Y_train[:5])
print('\nY_oh_train[:5]=')
print(Y_oh_train[:5])<jupyter_output>Y_train[:5]:
[0 3 3 3 3]

Y_oh_train[:5]=
[[1. 0. 0. 0.]
 [0. 0. 0. 1.]
 [0. 0. 0. 1.]
 [0. 0. 0. 1.]
 [0. 0. 0. 1.]]<jupyter_text>Defining a Fully Connected Neural Network with Keras<jupyter_code>def my_model(k,learning_rate = 0.15):
    inputs = Input(shape=(2,))
    out1 = Dense(k, activation='softmax')(inputs)
    model = Model(inputs=inputs, outputs=out1)
    sgd = SGD(lr=learning_rate)
    model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics = ["accuracy"]) 
    return model
model = my_model(k)
print(model.summary())<jupyter_output>Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, 2)]               0         
_________________________________________________________________
dense (Dense)                (None, 4)                 12        
=================================================================
Total params: 12
Trainable params: 12
Non-trainable params: 0
_________________________________________________________________
None<jupyter_text>Training the Model<jupyter_code>batch_size = train_split
nb_epoch = 100
n = X_train.shape[0] # number of training examples
history = model.fit(X_train, Y_oh_train, validation_data=(X_val,Y_oh_val), \
                    batch_size=batch_size, epochs=nb_epoch,verbose=1)<jupyter_output>Epoch 1/100
1/1 [==============================] - 0s 98ms/step - loss: 1.3479 - accuracy: 0.2027 - val_loss: 1.4047 - val_accuracy: 0.2027
Epoch 2/100
1/1 [==============================] - 0s 16ms/step - loss: 1.3391 - accuracy: 0.2297 - val_loss: 1.3927 - val_accuracy: 0.2162
Epoch 3/100
1/1 [==============================] - 0s 16ms/step - loss: 1.3306 - accuracy: 0.2905 - val_loss: 1.3812 - val_accuracy: 0.2703
Epoch 4/100
1/1 [==============================] - 0s 16ms/step - loss: 1.3225 - accuracy: 0.3716 - val_loss: 1.3703 - val_accuracy: 0.3243
Epoch 5/100
1/1 [==============================] - 0s 17ms/step - loss: 1.3147 - accuracy: 0.4122 - val_loss: 1.3599 - val_accuracy: 0.3649
Epoch 6/100
1/1 [==============================] - 0s 16ms/step - loss: 1.3072 - accuracy: 0.4459 - val_loss: 1.3499 - val_accuracy: 0.3919
Epoch 7/100
1/1 [==============================] - 0s 15ms/step - loss: 1.3001 - accuracy: 0.4595 - val_loss: 1.3404 - val_accuracy: 0.3919
Epoch 8/100
1/1 [===[...]<jupyter_text>Prininting Loss and estimated weights (Ws)<jupyter_code>loss = model.evaluate(X_test, Y_oh_test, verbose=0)
print('Final loss (cross-entropy and accuracy):',loss)
print()
W = model.get_weights()
for ii in range(len(W)//2):
    print("Layer %d" %ii)
    print('Bias:\n', W[2*ii + 1])
    print('W:\n', W[2*ii])
    print()

plt.plot(history.history['loss'], label = "Train loss")
plt.plot(history.history['val_loss'], label = "Val loss")
plt.xlabel("Epoch (iteration)")
plt.ylabel("Loss")
plt.legend()
plt.grid()
plt.show()


plt.plot(history.history['accuracy'], label = "Train accuarcy")
plt.plot(history.history['val_accuracy'], label = "Val accuarcy")
plt.xlabel("Epoch (iteration)")
plt.ylabel("Loss")
plt.legend()
plt.grid()
plt.show()<jupyter_output>Final loss (cross-entropy and accuracy): [0.9351743459701538, 0.9054054021835327]

Layer 0
Bias:
 [-0.3811083   0.5073468  -0.11489844 -0.01133997]
W:
 [[ 0.6958021  -0.13924785  1.4662881  -1.5514398 ]
 [ 0.31671807 -1.2081363  -1.0485553   1.1792164 ]]<jupyter_text>Classifier Decision Boundary<jupyter_code>h = .02  # step size in the mesh
slack = 0.1
# Calcula a grade para o espaço de atributos

x_min, x_max = X_train.min(axis=0) - slack, X_train.max(axis=0) + slack
xx, yy = np.meshgrid(np.arange(x_min[0], x_max[0], h), np.arange(x_min[1], x_max[1], h))
X_grid = np.c_[xx.ravel(), yy.ravel()]

# Prediction of all samples in our grid
Z = model.predict(X_grid).argmax(axis = 1)
Z = Z.reshape(xx.shape)

# Display decision boundary
plt.figure(1, figsize=(6, 5))
plt.pcolormesh(xx, yy, Z, cmap=plt.cm.Paired)

# Display training samples
plt.scatter(X_train[:,0], X_train[:,1], c=colors[Y_train], edgecolors='k', cmap=plt.cm.Paired)
plt.xlabel('x1')
plt.ylabel('x2')
plt.show()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup