<jupyter_start><jupyter_text>An Introduction to KNN classifier This an example to use Amazon Sagemaker. SageMaker allows one to build a ML pipeline easily. Building, training and deploying of ML models is less cumbersome with SageMakerIn this example, I will be using a Amazon's marketplace algorithm (KNN).The purpose of the notebook is to explain the usage of sagemaker and not the modeling aspect.The data used in this is mnist and problem is framed as binary classification.Amazon SageMaker's KNN algorithm extends upon typical linear models by training many models in parallel, in a computationally efficient manner. Each model has a different set of hyperparameters, and then the algorithm finds the set that optimizes a specific criteria. This can provide substantially more accurate models than typical linear algorithms at the same, or lower, cost.  Libraries used<jupyter_code>import boto3
import re
import pickle 
import gzip
import numpy as np 
import urllib.request
import json
import os
import io
import sagemaker
import pandas as pd
from sagemaker.predictor import csv_serializer, json_deserializer
from sagemaker.amazon.amazon_estimator import get_image_uri
import sagemaker.amazon.common as smac
from sagemaker import get_execution_role
import matplotlib.pyplot as plt
%matplotlib inline<jupyter_output><empty_output><jupyter_text>This notebook assumes that you have an AWS account and an IAM user setup and using the notebook instance of Amazon SageMaker. For further reference please refer to this notebook https://docs.aws.amazon.com/sagemaker/latest/dg/notebooks.html  Permissions and environment variables This notebook was created and tested on an ml.t2.medium notebook instance.Let's start by specifying:1. The S3 bucket and prefix that you want to use for training and model data. This should be within the same region as the Notebook Instance, training, and hosting.2. The IAM role arn used to give training and hosting access to your data. See the documentation for how to create these.<jupyter_code>bucket = 'test-karan-02'
prefix = 'sagemaker_demo_knn'
role = get_execution_role()<jupyter_output><empty_output><jupyter_text>Data ingestion Next, we read the dataset from an online URL into memory, for preprocessing prior to training. This processing could be done in situ by Amazon Athena, Apache Spark in Amazon EMR, Amazon Redshift, etc., assuming the dataset is present in the appropriate location. Then, the next step would be to transfer the data to S3 for use in training.<jupyter_code>! wget https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data
    
! wget https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.names    
iris_df = pd.read_csv("iris.data", header = None)

iris_df.columns = ['sepal_length', 'sepal_width','petal_length','petal_width', 'class']<jupyter_output><empty_output><jupyter_text>Data inspection Once the dataset is imported, it's typical as part of the machine learning process to inspect the data, understand the distributions, and determine what type(s) of preprocessing might be needed.<jupyter_code>iris_df.head()
import seaborn as sns

sns.countplot(x = "class",data=iris_df)<jupyter_output><empty_output><jupyter_text>Data conversion Since algorithms have particular input and output requirements, converting the dataset is also part of the process that a data scientist goes through prior to initiating training. In this particular case, the Amazon SageMaker implementation of Linear Learner takes recordIO-wrapped protobuf, where the data we have today is a pickle-ized numpy array on disk.Most of the conversion effort is handled by the Amazon SageMaker Python SDK, imported as sagemaker below<jupyter_code>iris_df['class'] = pd.Categorical(iris_df['class'])

iris_df['code_class'] = iris_df['class'].cat.codes
vectors = iris_df.iloc[:,:4].values.astype('float32')
labels = iris_df.iloc[:,5].values.astype('float32')

buf = io.BytesIO()
smac.write_numpy_to_dense_tensor(buf, vectors, labels)
buf.seek(0)<jupyter_output><empty_output><jupyter_text>Upload training data Now that we've created our recordIO-wrapped protobuf, we'll need to upload it to S3, so that Amazon SageMaker training can use it.<jupyter_code>key = 'recordio-pb-data'
boto3.resource('s3').Bucket(bucket).Object(os.path.join(prefix, key)).upload_fileobj(buf)
s3_train_data = 's3://{}/{}/{}'.format(bucket, prefix, key)
print('uploaded training data location: {}'.format(s3_train_data))<jupyter_output>uploaded training data location: s3://test-karan-02/sagemaker_demo_knn/recordio-pb-data<jupyter_text>Let's also setup an output S3 location for the model artifact that will be output as the result of training with the algorithm.<jupyter_code>output_location = 's3://{}/{}/output'.format(bucket, prefix)
print('training artifacts will be uploaded to: {}'.format(output_location))<jupyter_output>training artifacts will be uploaded to: s3://test-karan-02/sagemaker_demo_knn/output<jupyter_text>Training the linear model Once we have the data preprocessed and available in the correct format for training, the next step is to actually train the model using the data. Again, we'll use the Amazon SageMaker Python SDK to kick off training, and monitor status until it is completed. . Despite the dataset being small, provisioning hardware and loading the algorithm container take time upfront.<jupyter_code>container = get_image_uri(boto3.Session().region_name, 'knn')<jupyter_output><empty_output><jupyter_text>Next we'll kick off the base estimator, making sure to pass in the necessary hyperparameters. Notice:1. feature_dim is set to 4, which is the number of columns .2. predictor_type is set to classifier' 3. k is set to 5. It has to be tuned<jupyter_code>sess = sagemaker.Session()
knn = sagemaker.estimator.Estimator(container,
                                       role, 
                                       train_instance_count=1, 
                                       train_instance_type='ml.c4.xlarge',
                                       output_path=output_location,
                                       sagemaker_session=sess)
knn.set_hyperparameters(
    k = 5,
    predictor_type= "classifier",
    sample_size = 10,
    feature_dim= 4)


knn.fit({'train': s3_train_data})<jupyter_output><empty_output><jupyter_text>Sample Output1. 2020-02-26 21:07:45 Starting - Starting the training job...2. 2020-02-26 21:07:46 Starting - Launching requested ML instances......3. 2020-02-26 21:08:53 Starting - Preparing the instances for training......4. 2020-02-26 21:09:48 Downloading - Downloading input data...5. 2020-02-26 21:10:45 Training - Training image download completed. Training in progress..Docker entrypoint called with argument(s): train6. 2020-02-26 21:28:11 Uploading - Uploading generated training model7. 2020-02-26 21:28:11 Completed - Training job completed  Set up hosting for the model Now that we've trained our model, we can deploy it behind an Amazon SageMaker real-time hosted endpoint. This will allow out to make predictions (or inference) from the model dyanamically.<jupyter_code>knn_predictor = knn.deploy(initial_instance_count=1,
                                 instance_type='ml.m4.xlarge')<jupyter_output>---------------!<jupyter_text>Validate the model for use Finally, we can now validate the model for use. We can pass HTTP POST requests to the endpoint to get back predictions. To make this easier, we'll again use the Amazon SageMaker Python SDK and specify how to serialize requests and deserialize responses that are specific to the algorithm.<jupyter_code>knn_predictor.content_type = 'text/csv'
knn_predictor.serializer = csv_serializer
knn_predictor.deserializer = json_deserializer
result = knn_predictor.predict(iris_df.iloc[30,:4])
print(result)<jupyter_output>{'predictions': [{'predicted_label': 0.0}]}<jupyter_text>OK, a single prediction works. We see that for one record our endpoint returned some JSON which contains predictions, including the score and predicted_label. In this case, score will be a categorical value between [0,1,2] representing the class.<jupyter_code>sagemaker.Session().delete_endpoint(knn_predictor.endpoint)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup