<jupyter_start><jupyter_text>Everything you need to know about AIBy <NAME>Within this presentation, I'll give a taster into the world of AI, and focus of 'Machine Learning' with code samples, and an interactive segment to finish!  What am I using to present?The presentation method which I am using is called a ***'Jupyter Notebook'***. I love using Jupyter Notebooks in and out of work as an effective way to explain code, and more complex changes that are being implemented, as it allows for:* Splitting up text into digestable chunks (and uses MarkDown, which would be familiar to you if you use programs such as Notion)* Showing snippets of code (in a variety of languages, but I'll just be using Python in this presentation)* Showing the output of the code in a digestable format* Being incredibly lightweight and easy to useIt is because of all these positive features (plus being free and open source!) that Jupyter Notebooks are the industry standard form of presenting and working on Machine Learning and Data Science projects.<jupyter_code>#!/bin/python
import time


#Being able to seemlessly transition between code and text allows for better presentation!
y = 5
x = 83

print('<NAME> is the best!')

operationOne = y*x
print(operationOne)

operationTwo = x/operationOne
print(operationTwo)

operationThree = operationOne*operationTwo
print(operationThree)<jupyter_output><NAME> is the best!
415
0.2
83.0<jupyter_text>What is 'Artifical Intelligence'?Artificial intelligene refers to:* *"The effort to automate intellectual tasks, normally performed by humans"*AI in it's infancy consisted of a large set of pre-defined 'rules'. These rules are able to define what is and isn't possible (therefore no actual learning).Examples of this 'Classic AI' approach includes AI that is used in common games, such as:* Chess* Tic-Tac-Toe* etc.***'Very good'*** AI would therefore represent a ***'very good'*** set of rules, being more exhaustive and taking into account more situations. Adjusting the length and depth of the rules would therefore dictate how simple/complex the AI would become.***A real world example: Pacman's Ghosts***These ghosts are every simple in how they are programed, with it pre-programmed to find the shortest possible path to 'Pacman' in order to catch it, although it involves a character with slower movement speed.AI has many different layers and complexities that you are able to go into, with contemporary AI including the likes of ***Machine Learning*** and ***Neural Networks***.  What is 'Machine Learning'?***'Machine Learning*** is a field which has garnered a lot of hype over recent years, and is quickly becoming a popular field. I'll give a brief overview here:Machine Learning refers to:* "A method, whereby a program is able to *'create it's own rules'* and develop over time and training into something stronger than what any human could hard-code"***Machine Learning*** is able to take an *'input'*, and the desired outcome to create different rules. This process is referred to as the *'training stage'*. ***Training*** refers the process of a Machine Learning model getting sample data, and the expected values in order to figure out the best way to solve the problem, as accurately as possible. Due to the open ended nature of these rules (as the program makes them itself), Machine Learning models do not have a 100% accuracy.Another very commonly referred to subsection of AI is ***'Deep Learning/Neural Networks'***  What is Deep Learning/Neural Networks?***'Deep Learning'*** would have to be the most hyped section of AI, with it also posing as the most complex. This is a newly emerging field, with the potential to allow for more complex tasks to be solved.Deep Learning/Neural Networks refers to:* "A form of Machine Learning, that uses a layered representation of Data"Deep Learning relies on different *'nodes'* (often referred to as '*neurons*'), which all hold a unique, partially computed state, with the basic logic and data flow between these nodes being regulated via *'weightings'*. This allows for simple logic between nodes, using things such as *'if'* gates, *'and'* gates and *'or'* gates to choose if it will progress through the layer. Changing and adjusting these weightings allows for a neural network to optimise the output and be more accurate.Nodes more specifically changes the data with a set of pre-defined rules, of which can also be adjusted in tandem with the weightings between them. A simple way to visualise this would be to look at it as a large network of 'AI' models communcating with each other.All of this processing occurs within a set of *'hidden layers'*, with inputs coming in through an *'input layer'*, and being collated into a set of meaningful data and ouputed via an *'output layer'*.One confusion (which I would like to make clear within this presentation) is that Neural Networks ***ARE NOT MODELLED AFTER THE BRAIN***. They happen to use similar terminology, although to the best of our knowledge, this is not how the human brain functions.  What is Data?Data is incredibly important, and is used in *every* form of AI, and is the main thing used for analysis.Here's an example of *'data'*, and how it can be manipulated (with some extra terminology specific to Machine Learning and Deep Learning):|Midterm 1|Midterm 2|Final||---------|---------|-----||70       |80       |77   ||60       |90       |84   ||40       |50       |38   |"If I was given a student's Midterm 1 grade and their Final grade, how could I predict their Midterm 2 grade?"*  Any input information given is referred to as a "***feature***"    * In this example, the features are the **Midterm 1** and **Final** datasets* Any information that will be predicted is referred to as a "***Label***"   * In this example, the label will be the **Midterm 2** dataset.* i.e. when feeding a model '***features***' it will output a '***label***' Most machine learning models require a large sample size, as it allows for the model to create the most efficient rules possible.  * This means that the longer you let a model "train", the more efficient and more accurate it becomes (although the law of diminishing returns is very evident here).  * To test a model with the above dataset, you would remove the '***Midterm 2***' dataset, and see how accurate and reliable the output is.  * If you are feeding incorrect data, it can cause a large amoutn of mistakes, as it uses this data to learn more about what is should do. Now to some examples of AI, and a deeper dive into Machine Learning and Deep Learning  What is TensorFlow?TensorFlow is what I will be demonstrating with, and it is a framework developed by Google in order to make different kinds of AI. This framework is completely free and open source, and allows for the development of custom models, and those provided by a third party.  TensorFlow Core Learning Algorithms:There are 4 main learning algorithms that are a part of TensorFlow, these are:* Linear Regression* Classification* Clustering* Hidden Markov ModelsMost Business and Application implementations of Machine Learning use very basic Machine Learning models, as even these very simple models are capable of doing some very powerful things.These fundamental/'*core*' Machine Learning models can be used in almost all basic applications.  Linear Regression:"One of the most basic forms of Machine Learning, and is used to predict numerical values""Line of best fit refers to a line through a scatter plot of data points that best expresses the relationship between those points"**Project:**In this presentation we'll be using Linear Regression to predict the survival rate of passengers from the Titanic* This uses https://www.tensorflow.org/tutorials/estimator/linear heavily  How Linear Regression Works:Linear Regression refers to a way of calculating a 'line of best fit'. If data-points are related linearly, you are able to create a line of best fit, and predict future values.**Here is an example**:<jupyter_code>import plotly.io as pio
import matplotlib.pyplot as plt
import numpy as np

#making a simple dataset:
x = [1, 2, 2.5, 3, 4]
y = [1, 4, 7, 9, 15]

plt.plot(x, y, "ro")
plt.axis([0, 6, 0, 20])<jupyter_output><empty_output><jupyter_text>From this graph, it is evident that the dataset has a linear correspondance, and could therefore benefit from Linear Regression to predict future values.In this example, this will only use one input variable, as it is 2 dimensional, although in larger datasets, it will requite a lot more variables and inputs.An equation for a line in 2d is: $$y=mx+c$$Here is an example of a line of best fit for the above graph:<jupyter_code>import matplotlib.pyplot as plt
import numpy as np

#making a simple dataset:

x = [1, 2, 2.5, 3, 4]
y = [1, 4, 7, 9, 15]

plt.plot(x, y, "ro")
plt.axis([0, 6, 0, 20])
#Original dataset for convenience

plt.plot(np.unique(x), np.poly1d(np.polyfit(x, y, 1))(np.unique(x)))
#Linear regression example

plt.show<jupyter_output><empty_output><jupyter_text>SetupThis section includes most of the imports used:<jupyter_code>#All the needed imports for this example:

from __future__ import absolute_import, division, print_function, unicode_literals

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from IPython.display import clear_output
from six.moves import urllib

import tensorflow.compat.v2.feature_column as fc

import tensorflow as tf<jupyter_output><empty_output><jupyter_text>Data:Data is the **MOST** important part of Machine Learning!In this example, I'll go through exploring, cleaning and selecting appropriate data.This uses the 'Titanic' dataset, which has a large amount of information for each passenger, so I'll go through the data and explore!**Here I'll go through loading the database and how to explore it using built-in tools**<jupyter_code># Load dataset.

dftrain = pd.read_csv('https://storage.googleapis.com/tf-datasets/titanic/train.csv') #training
dfeval = pd.read_csv('https://storage.googleapis.com/tf-datasets/titanic/eval.csv') #testing

y_train = dftrain.pop('survived') # removes this column from the original datasets, and puts it in it's own variable
y_eval = dfeval.pop('survived')<jupyter_output><empty_output><jupyter_text>Using the `pd.read_csv()` method allows for the creation of a 'dataframe' (a bit like a large table), which you can refer to and edit later.    In this example, we used the '**survived**' column and sorted it into a variable.To look at the data, I'll be using ``.head()`` from Pandas. This allows for looking at the '*head*' of the data (i.e. the first 5 lines), a bit like it is in Unix.<jupyter_code>dftrain.head() # Looking at the top 5 columns of this dataset
dfeval.head() #looking at the top 5 columns of this dataset<jupyter_output><empty_output><jupyter_text>To test how many rows there are in each dataset, you can use the ``.shape()`` command.<jupyter_code>dftrain.shape[0], dfeval.shape[0]

# In this case there are 627 lines in dftrain and 264 in dfeval<jupyter_output><empty_output><jupyter_text>If you want to have a more statistical analysis of the data, you can use the ``.describe()`` command. This gives some common statistical values that can be useful.<jupyter_code>dftrain.describe()<jupyter_output><empty_output><jupyter_text>Same thing for the other dataset<jupyter_code>dfeval.describe()<jupyter_output><empty_output><jupyter_text>In order to make more sense of the data, it is valuable to generate some visual representations to look at potential ways you can work with the data.<jupyter_code># A Histogram to denote age:

dftrain.age.hist(bins=20)<jupyter_output><empty_output><jupyter_text>From this histogram, it is quite obvious that most of the passengers in this training set are in their 20s and 30s.<jupyter_code># A bar chart to represent sex in those who travelled:

dftrain.sex.value_counts().plot(kind='barh')<jupyter_output><empty_output><jupyter_text>From this bar chart, it is quite obvious that there were about twice the amount of males compared to females.<jupyter_code># Looking at what class most of the passengers were in:

dftrain['class'].value_counts().plot(kind='barh')<jupyter_output><empty_output><jupyter_text>From this bar graph, it is quite obvious that the majority of passengers were in Third class, with Second and First class having similar numbers.<jupyter_code>#Looking at survival rate and sex (seeing if there is a correlation)

pd.concat([dftrain, y_train], axis=1).groupby('sex').survived.mean().plot(kind='barh').set_xlabel('% survive')<jupyter_output><empty_output><jupyter_text>From this bar graph it is very obvious that if you are female in this example with the Titanic, you have a significantly larger chance of survival. This therefore has a good chance of being a predictive feature for this model.  Training vs Testing data:When creating a Machine Learning model, it is important to use multiple datasets, usually using one to train the Machine Learning model, and another to test and validate the Machine Learning model.In this example, there are 2 datasets, these are:* **Training**  * Usually much larger than the testing dataset, allows for the model to adapt and learn.* **Testing**  * Allows for you to use a different set of data (to avoid the model from potentially 'memorising' the data), to test how well the model is going. This allows for the model to be able to work on '***NEW***' data, therefore requires for there to be seperate datasets for testing and training.This can be seen in the difference in dataset size, as seen below (already seen above, but I'll do it again for convenience).<jupyter_code>Training = dftrain.shape
Evaluation = dfeval.shape

print("Training Dataset size:")
print(Training) #Shows the amount of rows (amount of data points), and the amount of columns (amount of variables per data point)

print("Evaluation Dataset size:")
print(Evaluation) #Shows the amount of rows (amount of data points), and the amount of columns (amount of variables per data point)<jupyter_output>Training Dataset size:
(627, 9)
Evaluation Dataset size:
(264, 9)<jupyter_text>From this, it is evident that the training dataset is significantly larger compared to the testing dataset.  Catagorising Data:There are 2 main types of data, and it is important to separate them (so the Machine Learning algorithm is able to manipulate them accordingly).These are:* **Catagorical Data**  * Catagorical data represents data which is in 'words', and isn't numerical. This usually can be seen as headings on a table.  * In this example, catagorical data represents the headings on the table which represent any text based data.  * Catagorical data is '***encoded***' using an integer value so the Machine Learning algorithm is able to work with the data.* **Numerical Data**  * Numerical data represents anything that relates to numbers.  * In this example, it represents any headings that represent and number based data.  * Numerical data ***does not*** need to be encoded, as it '*already is*'.<jupyter_code># seperating the dataset:

CATEGORICAL_COLUMNS = ['sex', 'n_siblings_spouses', 'parch', 'class', 'deck',
                       'embark_town', 'alone'] #Categorical Data

NUMERIC_COLUMNS = ['age', 'fare'] #Numeric Data

#Some loops to feed in the data into the 
#TensorFlow Linear Regression Estimator Model (aka Feature Columns):

feature_columns = [] #blank list to store all the feature columns

#Categorical Data
for feature_name in CATEGORICAL_COLUMNS:
  vocabulary = dftrain[feature_name].unique() #Goes through each data-frame (such as 'sex'), then gets all the unique values
  feature_columns.append(tf.feature_column.categorical_column_with_vocabulary_list(feature_name, vocabulary))

for feature_name in NUMERIC_COLUMNS:
  feature_columns.append(tf.feature_column.numeric_column(feature_name, dtype=tf.float32))<jupyter_output><empty_output><jupyter_text>Just to explanation's sake, here is some more information:<jupyter_code>#Categorical data example:

dftrain['embark_town'].unique() 

#Adds to the vocabulary, and you can then create a 'numerical' code for it.<jupyter_output><empty_output><jupyter_text>The 'Training Process':Models are loaded in 'batches' (to allow for extremely large training data to fit within RAM). In this example, it will be fed in batches of 32.These batches will be fed into the model a certain amount of times (as dictated by the number of ***epochs***)**Epoch**:  * One stream of the entire dataset (i.e. the entire data set being fed into the model).  * The number of epochs set for the Machine Learning model dictates how well it can react to an unknown set of data (therefore it is optimal to run through multiple epochs)  * E.g. if there are 10 epochs, the Machine Learning model will go through the same dataset 10 times.As you need to load the input data in batches, and multiple times, it is required to make an '*input function*'. This function simply defines how the dataset will be converted into batches at each epoch. **Input Function:**The TensorFlow model which is being used here requires that the data is passed as a ``tf.data.Dataset`` object. This means that an *input function* must be created to convert the current pandas dataframe into that object.This is quite complex, although there should be some comments to make it a bit more understandable.You will never have to code an input function from scratch, you can usually find one in documentation.<jupyter_code>def make_input_fn(data_df, label_df, num_epochs=10, shuffle=True, batch_size=32):
#^^ Make Input Function: import data-dataframe, label-dataframe, at 10 epochs, shuffle the data and with a batch size of 32
  def input_function(): #inner function (this will be returned)
    ds = tf.data.Dataset.from_tensor_slices((dict(data_df), label_df)) # create tf.data.Dataset object with data and its label
    if shuffle:
      ds = ds.shuffle(1000) #randomise the order of the data.
    ds = ds.batch(batch_size).repeat(num_epochs) #split data into batches, and repeat for the number of epochs
    return ds #return a batch of the dataset
  return input_function #return a function object for use

train_input_fn = make_input_fn(dftrain, y_train) #calling the input function for both of these, although there can be some changes for the input values for the function.
eval_input_fn = make_input_fn(dfeval, y_eval, num_epochs=1, shuffle=False)<jupyter_output><empty_output><jupyter_text>Creating the model:In this example, I will be using the 'Linear Estimator' function of TensorFlow in order to use the Linear Regression model.<jupyter_code>linear_est = tf.estimator.LinearClassifier(feature_columns=feature_columns)
#creating a linear estimator by passing through the feature columns that were created earlier

clear_output() #Just to clean up the output, otherwise it takes up a lot of space ;)<jupyter_output><empty_output><jupyter_text>Training the model:All we have to do in this example is pass through the input functions from earlier.<jupyter_code>linear_est.train(train_input_fn) #train
result = linear_est.evaluate(eval_input_fn) #get the model metrics by testing on the testing data

clear_output() #clear console output

print("Accuracy of Model:")
accuracy = (result['accuracy']*100) #result variable is just the metrics, here I am just outputing the accuracy.
percentage = str(accuracy)+"%"
print(percentage)<jupyter_output>Accuracy of Model:
77.27272510528564%<jupyter_text>Getting an answer:From this training, you are able to get a prediction on how many people will survive. You can get this value by looking through the output results, and outputting the specific part of the dictionary you need.This is quite a long script, but it allows for a neater and more digestable output.<jupyter_code>result = list(linear_est.predict(eval_input_fn))

clear_output() # clear console output

f = 0

while f < 4: #small script to loop through the information and predicted change of death (top 7 results from testing dataset)
    print("Passenger number: " + str(f))

    print("")

    print("Information of passenger:")
    print(dfeval.loc[f])

    print("")

    print("Chance of survival:")
    survival = (result[f]['probabilities'][1]*100) #looking through the dictionary, and getting chance of survival
    print(str(survival)+"%")

    print("")

    print("Chance of death:")
    death = (result[f]['probabilities'][0]*100) #looking through the dictionary, and getting chance of death
    print(str(death)+"%")
    
    print("")

    print("Did they survive?:")

    if y_eval.loc[f] == 1:
        print("This person survived")
    else:
        print("This person did not survive")

    print("")
    
    print("\n")

    o = f + 1
    f = o

#From this output, you are able to tell the prediction that the Machine Learning model made, and what the actual result was (Percentage chance of dying and survivng, vs if they actually survived.)<jupyter_output>Passenger number: 0

Information of passenger:
sex                          male
age                          35.0
n_siblings_spouses              0
parch                           0
fare                         8.05
class                       Third
deck                      unknown
embark_town           Southampton
alone                           y
Name: 0, dtype: object

Chance of survival:
6.184752285480499%

Chance of death:
93.81524920463562%

Did they survive?:
This person did not survive



Passenger number: 1

Information of passenger:
sex                          male
age                          54.0
n_siblings_spouses              0
parch                           0
fare                      51.8625
class                       First
deck                            E
embark_town           Southampton
alone                           y
Name: 1, dtype: object

Chance of survival:
31.40166699886322%

Chance of death:
68.59833002090454%

Did they survive?:
This person did not sur[...]
StarcoderdataJupyterStructuredCleanDedup