<jupyter_start><jupyter_text>ML Pipeline PreparationFollow the instructions below to help you create your ML pipeline. 1. Import libraries and load data from database.- Import Python libraries- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)- Define feature and target variables X and Y<jupyter_code>!pip install cache-magic
cat_count = (df.iloc[:, 4:] != 0).sum()
cat_count
%cache cache1 = model_cv
import dill
dill.dump_session('notebook_env.db')
#change if needed
pd.set_option('display.max_rows', 100)
pd.set_option('display.max_columns', 100)
pd.set_option('display.width', 100)
# import libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sqlalchemy import create_engine
import sys
import cache_magic

#machine learning
from sklearn.pipeline import Pipeline, FeatureUnion
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestClassifier,AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.multioutput import MultiOutputClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, make_scorer, classification_report, fbeta_score
from sklearn.svm import SVC
from scipy.stats.mstats import gmean

from sklearn.base import BaseEstimator, TransformerMixin

#NLP
import re
import pickle
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize, WhitespaceTokenizer
from nltk.stem import WordNetLemmatizer
from nltk.stem.porter import PorterStemmer
from sklearn.pipeline import Pipeline, FeatureUnion
#nltk.download('stopwords')
#nltk.download('punkt')
#nltk.download('wordnet')

%matplotlib inline
# load data from database
engine = create_engine('sqlite:///../data/disaster_response.db')
df = pd.read_sql_table(table_name='message_categories', con=engine)
X = df['message']
Y = df.iloc[:,4:]
df.isna().sum()
display(X, Y)<jupyter_output><empty_output><jupyter_text>2. Write a tokenization function to process your text data<jupyter_code>def tokenize(text):
    '''
    IN: 
        raw text for tokenizing via the following steps: 
            - normalized, punctuation removed, stop words removed, stemmed, and lemmatized
    OUT:
        tokenized text
    '''
    #Normalize text and remove punctuation
    normalized_txt = re.sub(r"[^a-zA-Z0-9]", " ", text.lower())
    
    #tokenize text
    words = word_tokenize(normalized_txt)
 
    #lemmatize
    words = [WordNetLemmatizer().lemmatize(w) for w in words]
    
    #Reduce words to their stems
    words = [PorterStemmer().stem(w) for w in words]

    return words<jupyter_output><empty_output><jupyter_text>3. Build a machine learning pipelineThis machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables.<jupyter_code>#starting pipeline
def pipeline_model():
    pipeline = Pipeline([('vect', CountVectorizer(tokenizer=tokenize)),
                          ('tfidf', TfidfTransformer()),
                          ('clf', MultiOutputClassifier(RandomForestClassifier()))])
    return pipeline
model = pipeline_model()
model.get_params().keys()<jupyter_output><empty_output><jupyter_text>4. Train pipeline- Split data into train and test sets- Train pipeline<jupyter_code>X_train, X_test, Y_train, Y_test = train_test_split(X, Y,random_state = 42)
%%time
model.fit(X_train, Y_train)<jupyter_output>Wall time: 4min 57s<jupyter_text>5. Test your modelReport the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each.<jupyter_code>%%time
#make predictions
y_preds = model.predict(X_test)
def metrics_report(Y_test, y_preds):
    '''
    Create a weighted averages summary dataframe for each label, 
    using the classification_report function 
    IN: 
        Y_test - array of actual values
        y_preds - numpy array of predicted values
    OUT df columns: precision,recall,f1-score,support
    This is a performance metric reference from Github User: rebeccaebarnes
    '''
    results_dict = {}

    for pred, label, col in zip(y_preds.transpose(), Y_test.values.transpose(), Y_test.columns):
        results_dict[col] = classification_report(label, pred, output_dict=True)
        
    weighted_avg = {}
    for key in results_dict.keys():
        weighted_avg[key] = results_dict[key]['weighted avg']

    df_wavg = pd.DataFrame(weighted_avg).transpose()
    return df_wavg
def weighted_avg_report(df):
    '''
    OUT:
        descriptive statistics for the created weighted averages summary df
        upper and lower quantile df slices
    '''
    display(df['f1-score'].describe())
    display('lowest quantile of f scores',df[df['f1-score'] <= df['f1-score'].quantile(0.25)]) # lowest quantile of f scores
    #print(df.sort_values('f1-score').head(n = 10))
    display('highest quantile of f scores', df[df['f1-score'] >= df['f1-score'].quantile(0.75)]) # highest quantile of f scores
weighted_avg_report(df_wavg)
weighted_avg = {}
for key in results_dict.keys():
    weighted_avg[key] = results_dict[key]['weighted avg']

df_wavg = pd.DataFrame(weighted_avg).transpose()
df_wavg
df_wavg['f1-score'].describe()
df_wavg[df_wavg['f1-score'] <= df_wavg['f1-score'].quantile(0.25)]
df_wavg.sort_values('f1-score').head(n = 10)
df_wavg[df_wavg['f1-score'] >= df_wavg['f1-score'].quantile(0.75)]<jupyter_output><empty_output><jupyter_text>---<jupyter_code>df_feats = df.iloc[:,4:]
df_feats_counts = df_feats.sum()
(df_feats_counts/df_feats.shape[0]).sort_values()
df_wavg['f1-score'].sort_values()
df.shape[0]
df_feats.sort_values()<jupyter_output><empty_output><jupyter_text>---  6. Improve your modelUse grid search to find better parameters.<jupyter_code>model.get_params()
parameters = {'vect__ngram_range':[(1,2),(2,2)],
            'clf__estimator__n_estimators':[50, 100]
             }

cv = GridSearchCV(estimator=pipeline, param_grid=parameters)
%%time
#grid search

parameters = {'vect__min_df': [1, 5],
              'tfidf__use_idf':[True, False],
              'clf__estimator__n_estimators':[10, 25, 50, 100], 
              'clf__estimator__min_samples_split':[2, 5, 10]}

model_cv = GridSearchCV(estimator=model, param_grid=parameters, verbose=3)
%%time
model_cv.fit(X_train, Y_train)
model_cv.best_params_
model_cv.best_score_
%%time
y_preds_cv = model_cv.predict(X_test)
cv_results = model_performance(Y_test, y_preds)<jupyter_output>related
              precision    recall  f1-score   support

           0       0.76      0.29      0.42      1569
           1       0.81      0.97      0.88      4938

    accuracy                           0.81      6507
   macro avg       0.79      0.63      0.65      6507
weighted avg       0.80      0.81      0.77      6507

request
              precision    recall  f1-score   support

           0       0.90      0.99      0.94      5403
           1       0.87      0.46      0.60      1104

    accuracy                           0.90      6507
   macro avg       0.88      0.72      0.77      6507
weighted avg       0.89      0.90      0.88      6507

offer
              precision    recall  f1-score   support

           0       1.00      1.00      1.00      6481
           1       0.00      0.00      0.00        26

    accuracy                           1.00      6507
   macro avg       0.50      0.50      0.50      6507
weighted avg       0.99      1.00      0.99      650[...]<jupyter_text>7. Test your modelShow the accuracy, precision, and recall of the tuned model.  Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio! ---Try new model parameters<jupyter_code>%%time
#grid search

parameters_2 = {'vect__min_df': [5, 10],
              'tfidf__use_idf':[True],
              'vect__binary':[True, False],
              'clf__estimator__n_estimators':[100, 500, 1000], 
              'clf__estimator__min_samples_split':[1, 2]}

model_cv_2 = GridSearchCV(estimator=model, param_grid=parameters_2, verbose=3)
#pipeline model 2
def pipeline_model_2():
    '''
    build a pipeline model with the specificed transformers and estimator
    '''
    pipeline = Pipeline([
        ('features', FeatureUnion([

            ('text_pipeline', Pipeline([
                ('vect', CountVectorizer(tokenizer=tokenize)),
                ('tfidf', TfidfTransformer())
            ])),

            #('starting_verb', StartingVerbExtractor()),
            #("word_count", WordCount()),
            #("character_count", CharacterCount()),
            #("noun_count", NounCount()),
            #("verb_count", VerbCount())
        ])),

        ('clf', MultiOutputClassifier(RandomForestClassifier()))
    ])
    #pipeline = Pipeline([('vect', CountVectorizer(tokenizer=tokenize)),
    #                      ('tfidf', TfidfTransformer()),
    #                      ('clf', MultiOutputClassifier(RandomForestClassifier()))])
    return pipeline
avg_accuracy_model_2 = (y_preds_2 == Y_test).mean().mean()
avg_accuracy_model_2
model_2 = pipeline_model_2()
X_train, X_test, Y_train, Y_test = train_test_split(X, Y,random_state = 42)
%%time
model_2.fit(X_train, Y_train)
%%time
#make predictions
y_preds_2 = model_2.predict(X_test)
wavg_df_2 = create_report_summary(y_preds_2, Y_test)
weighted_avg_report(wavg_df_2)
model_cv_2 = GridSearchCV(estimator=model, param_grid=parameters_2, verbose=3)
%%time
model_cv_2.fit(X_train, Y_train)
model_cv_2.best_params_
model_cv_2.best_score_
%%time
y_preds_cv_2 = model_cv_2.predict(X_test)
%%time
cv_2_results = model_performance(Y_test, y_preds_cv_2)<jupyter_output>related
              precision    recall  f1-score   support

           0       0.75      0.30      0.43      1569
           1       0.81      0.97      0.88      4938

    accuracy                           0.81      6507
   macro avg       0.78      0.64      0.66      6507
weighted avg       0.80      0.81      0.77      6507

request
              precision    recall  f1-score   support

           0       0.91      0.98      0.94      5403
           1       0.86      0.51      0.64      1104

    accuracy                           0.90      6507
   macro avg       0.89      0.75      0.79      6507
weighted avg       0.90      0.90      0.89      6507

offer<jupyter_text>count    9.000000mean     0.859255std      0.055942min      0.77475825%      0.82182550%      0.88707775%      0.892294max      0.933391<jupyter_code>#pipeline model 3
def pipeline_model_3():
    pipeline = Pipeline([
        ('features', FeatureUnion([

            ('text_pipeline', Pipeline([
                ('vect', CountVectorizer(tokenizer=tokenize)),
                ('tfidf', TfidfTransformer())
            ])),
        ])),

        ('clf', MultiOutputClassifier(AdaBoostClassifier()))
    ])
    return pipeline
model_3 = pipeline_model_3()
model_3.get_params().keys()
parameters = {'features__vect__min_df': [1, 5],
              'features__tfidf__use_idf':[True, False],
          'clf__estimator__learning_rate':[0.01, 0.1, 0.5],
          'clf__estimator__n_estimators':[100, 200]}
model_cv_3 = GridSearchCV(estimator=model_3, 
                          scoring='f1_weighted', 
                          param_grid=parameters, 
                          verbose=3)
Y.sum()
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, random_state = 42)
%%time
model_3.fit(X_train, Y_train)
%%time
#make predictions
y_preds_3 = model_3.predict(X_test)
wavg_df_3 = metrics_report(Y_test, y_preds_3)
wavg_df_3
weighted_avg_report(wavg_df_3)
metrics_report(Y_test, y_preds_3)<jupyter_output><empty_output><jupyter_text>--- Model 3 Run again, Tokenize has no stemmingEnd result mean of f-score is 0.94, slightly less than when stemming is present. Therefore decided to keep stemming in the tokenize function.<jupyter_code>%%time
#make predictions
y_preds_3 = model_3.predict(X_test)
wavg_df_3 = metrics_report(Y_test, y_preds_3)
wavg_df_3
weighted_avg_report(wavg_df_3)
metrics_report(Y_test, y_preds_3)
from sklearn.metrics import multilabel_confusion_matrix as confusion_matrix
def display_mat(Y_test, y_preds):
    '''
    prints a confusion matrix of each var in each label
    '''
    #accuracy_df = pd.DataFrame(columns=['Accuracy score'])
    actual = np.array(Y_test)
    col_names = list(Y_test.columns.values)
    for i in range(len(Y_test.columns)):
        key = col_names[i]
        confusion_mat = confusion_matrix(actual[:,i], y_preds[:,i])
        print(key, confusion_mat)
    #for pred, actual, col in zip(y_preds.transpose(), Y_test.values.transpose(), Y_test.columns):
        #results_dict[col] = accuracy_score(np.(Y_test[col]), y_preds[col])
    #return accuracy_df
display_mat(Y_test, y_preds_3)<jupyter_output>related [[[4664  274]
  [1020  549]]

 [[ 549 1020]
  [ 274 4664]]]
request [[[ 618  486]
  [ 188 5215]]

 [[5215  188]
  [ 486  618]]]
offer [[[   0   26]
  [   2 6479]]

 [[6479    2]
  [  26    0]]]
aid_related [[[1682 1032]
  [ 536 3257]]

 [[3257  536]
  [1032 1682]]]
medical_help [[[ 143  399]
  [  77 5888]]

 [[5888   77]
  [ 399  143]]]
medical_products [[[ 123  224]
  [  59 6101]]

 [[6101   59]
  [ 224  123]]]
search_and_rescue [[[  45  147]
  [  26 6289]]

 [[6289   26]
  [ 147   45]]]
security [[[   6  127]
  [  12 6362]]

 [[6362   12]
  [ 127    6]]]
military [[[  71  154]
  [  49 6233]]

 [[6233   49]
  [ 154   71]]]
water [[[ 277  153]
  [  84 5993]]

 [[5993   84]
  [ 153  277]]]
food [[[ 514  217]
  [ 126 5650]]

 [[5650  126]
  [ 217  514]]]
shelter [[[ 329  269]
  [  93 5816]]

 [[5816   93]
  [ 269  329]]]
clothing [[[  44   57]
  [  22 6384]]

 [[6384   22]
  [  57   44]]]
money [[[  58  110]
  [  38 6301]]

 [[6301   38]
  [ 110   58]]]
missing_people [[[  12   5[...]<jupyter_text>8. Try improving your model further. Here are a few ideas:* try other machine learning algorithms* add other features besides the TF-IDF<jupyter_code>class StartingVerbExtractor(BaseEstimator, TransformerMixin):
    '''
    Returns True/False boolean value for whether the first word in a string of text is a verb or not
    '''
    def starting_verb(self, text):
        pos_tags_dict = {}
        for ea_msg in nltk.sent_tokenize(msg):
            pos_tags_dict = dict(nltk.pos_tag(tokenize(ea_msg)))
            first_word = list(pos_tags_dict.keys())[0]
            first_tag = pos_tags_dict[first_word]
        #sentence_list = nltk.sent_tokenize(text)
        #for sentence in sentence_list:
            #pos_tags = nltk.pos_tag(tokenize(sentence))
            #first_word, first_tag = pos_tags[0]
            if first_tag in ['VB', 'VBP'] or first_word == 'RT':
                return True
        return False

    def fit(self, X, y=None):
        return self

    def transform(self, X):
        X_tagged = pd.Series(X).apply(self.starting_verb)
        return pd.DataFrame(X_tagged)
#pipeline model 4
def pipeline_model_4():
    pipeline = Pipeline([
        ('features', FeatureUnion([

            ('text_pipeline', Pipeline([
                ('vect', CountVectorizer(tokenizer=tokenize)),
                ('tfidf', TfidfTransformer())
            ])),

            ('starting_verb', StartingVerbExtractor()),
            #("word_count", WordCount()),
            #("character_count", CharacterCount()),
            #("noun_count", NounCount()),
            #("verb_count", VerbCount())
        ])),

        ('clf', MultiOutputClassifier(AdaBoostClassifier()))
    ])
    #pipeline = Pipeline([('vect', CountVectorizer(tokenizer=tokenize)),
    #                      ('tfidf', TfidfTransformer()),
    #                      ('clf', MultiOutputClassifier(RandomForestClassifier()))])
    return pipeline
model_4 = pipeline_model_4()
%%time
y_preds_4 = model_4.fit(X_train, Y_train)
%%time
y_preds_4 = model_4.predict(X_test)#make predictions
wavg_df = metrics_report(Y_test, y_preds_4)
display(weighted_avg_report(wavg_df))
def run_model(pipeline, X, Y):
    X_train, X_test, Y_train, Y_test = train_test_split(X, Y,random_state = 42)
    model = pipeline()
    y_preds = model.fit(X_train, Y_train)
    y_preds = model.predict(X_test)#make predictions
    wavg_df = metrics_report(Y_test, y_preds)
    display(weighted_avg_report(wavg_df))
    return model<jupyter_output><empty_output><jupyter_text>TRY RUNNING A PIPELINE MODEL WITH OBTAINED PARAMETERS<jupyter_code>#pipeline model 5
def pipeline_model_5():
    pipeline = Pipeline([
        ('features', FeatureUnion([

            ('text_pipeline', Pipeline([
                ('vect', CountVectorizer(tokenizer=tokenize, 
                                         min_df=10
                                        )),
                ('tfidf', TfidfTransformer(use_idf=True))
            ])),

            ('starting_verb', StartingVerbExtractor()),
        ])),

        #('clf', MultiOutputClassifier(AdaBoostClassifier()))
        ('clf', MultiOutputClassifier(RandomForestClassifier(min_samples_split=2, 
                                                             random_state=42, 
                                                             n_estimators=100))),
    ])
    return pipeline
%%time
model_5 = run_model(pipeline_model_5, X, Y)
AdaBoostClassifier
#pipeline model 6
def pipeline_model_6():
    pipeline = Pipeline([
        ('features', FeatureUnion([

            ('text_pipeline', Pipeline([
                ('vect', CountVectorizer(tokenizer=tokenize, 
                                         min_df=10
                                        )),
                ('tfidf', TfidfTransformer(use_idf=True))
            ])),

            ('starting_verb', StartingVerbExtractor()),
        ])),

        ('clf', MultiOutputClassifier(AdaBoostClassifier()))
        #('clf', MultiOutputClassifier(RandomForestClassifier(min_samples_split=2, 
                                                             #random_state=42, 
                                                             #n_estimators=100))),
    ])
    return pipeline
%%time
model_6 = run_model(pipeline_model_6, X, Y)
model_6.get_params().keys()
#grid search

parameters_3 = {'vect__min_df': [5, 10],
              'clf__estimator__n_estimators':[100, 500, 1000], 
              'clf__estimator__min_samples_split':[1, 2]}
model_cv_3 = GridSearchCV(estimator=model_3, param_grid=parameters_3, verbose=3)
%%time
model_cv_3.fit(X_train, Y_train)<jupyter_output><empty_output><jupyter_text>model CV 1 best parameters:     {'clf__estimator__min_samples_split': 2,     'clf__estimator__n_estimators': 100,     'tfidf__use_idf': True,     'vect__min_df': 5}model CV 2 best parameters:    {'clf__estimator__min_samples_split': 2,     'clf__estimator__n_estimators': 1000,     'tfidf__use_idf': True,     'vect__binary': False,     'vect__min_df': 10} F-scores    Model 1:    count    35.000000    mean      0.931140    std       0.057876    min       0.771124    25%       0.916096    50%       0.944198    75%       0.971679    max       0.994010    Model CV 1:    count    9.000000    mean     0.851604    std      0.053219    min      0.771124    25%      0.817647    50%      0.872271    75%      0.883507    max      0.917431    Model 2:    count    35.000000    mean      0.930772    std       0.058746    min       0.767192    25%       0.921509    50%       0.940264    75%       0.970957    max       0.994010    Model CV 2:    count    9.000000    mean     0.859255    std      0.055942    min      0.774758    25%      0.821825    50%      0.887077    75%      0.892294    max      0.933391    Model 3:    count    35.000000    mean      0.940781    std       0.057800    min       0.754306    25%       0.933265    50%       0.962092    75%       0.979511    max       0.993857    Model 4:    count    35.000000    mean      0.940778    std       0.057797    min       0.754306    25%       0.933265    50%       0.962092    75%       0.979511    max       0.993857    Model 5:    count    35.000000    mean      0.936367    std       0.056137    min       0.773738    25%       0.929425    50%       0.952569    75%       0.973235    max       0.994010    Model 6:    count    35.000000    mean      0.940143    std       0.058305    min       0.751381    25%       0.928964    50%       0.960996    75%       0.978966    max       0.993703  9. Export your model as a pickle file<jupyter_code>def pickle_file(file_to_pickle, file_name):
    '''
    Saves a file to the data folder with the extension .pkl
    file path: '../data/'+ file_name+'.pkl'
    '''
    pickle.dump(file_to_pickle, open('../data/'+file_name+'.pkl', 'wb'))
%%time
pickle_file(model_3, '../data/model_3')
%%time
pickle_file(model_cv_2, 'model_cv_2.pkl')
model_cv_load = load_pickle('model_cv.pkl')<jupyter_output><empty_output><jupyter_text>Note: the cross-validation models (which are too big for GitHub anyways) were not able to be saved at the time.  I encountered a bug where the tokenize instance doesn't match the named function, possibly because the function was editing while the tests were running. The summary results are saved as outputs in this notebook.<jupyter_code>pickle_file(model_4, 'model_4')
pickle_file(model_5, 'model_5')
pickle_file(model_6, 'model_6')
def load_pickle(pickled_file_name):
    '''
    Loads a pickled file (.pkl) in the data folder. 
    File path: 
    '''
    return pickle.load(open('../data/'+ pickled_file_name+'.pkl', 'rb'))<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup