<jupyter_start><jupyter_text>Data section Build classifiers and retrieve prediction  Set up<jupyter_code>import math
import pandas as pd
import matplotlib.pyplot as plt
%pylab inline<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>Read the data. Make sure that there is a column called "Noshow", with the indicators of no-shows (1 vs 0)<jupyter_code>#import statsmodels.api as sm
df = pd.read_csv('data.csv',sep=',')<jupyter_output><empty_output><jupyter_text>The no-show rate<jupyter_code>df['Noshow'].mean()
X = df.drop(columns=['Noshow'],axis=1) 
Y = df['Noshow']
Y.mean()<jupyter_output><empty_output><jupyter_text>Split data into training and testing<jupyter_code>from sklearn.model_selection import train_test_split
test_size_proportion = 0.2
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=test_size_proportion, random_state=0)<jupyter_output><empty_output><jupyter_text>Find typical variance obtained by probabilistic classifiers Study the distribution of the probabilities obtained by probabilistic classifiers The first thing to do it to obtain a vector of probabilities. We execute a cross-validation and label each instance with the predicted probability obtained when it was out of sample<jupyter_code>import sklearn as sk
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.svm import LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis
from sklearn.neural_network import MLPClassifier
from sklearn.ensemble import BaggingClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import AdaBoostClassifier

iter = 1
nfolds = 10
from sklearn.metrics import roc_curve, auc
X_copy = X_train.copy()


clfs = [sk.ensemble.RandomForestClassifier(n_jobs=-1,random_state=0), sk.naive_bayes.GaussianNB(),
        sk.linear_model.LogisticRegression(n_jobs=-1),
        sk.ensemble.AdaBoostClassifier(random_state=0),\
        MLPClassifier(random_state=0)]
#clfs = [sk.ensemble.AdaBoostClassifier(random_state=0)]
#clfs=[sk.linear_model.LogisticRegression(n_jobs=-1)]
class_df = pd.DataFrame()
classifiers_by_name = {}
for cl in clfs:
    kf = KFold(n_splits=nfolds,random_state=0,shuffle=True)
    clName = str(cl)[:15] 
    classifiers_by_name[clName] = cl
    class_df.loc[clName,'auc'] = 0
    # compute the cross-validated auc
    print(f'Testing {clName}: ')
    for tr_ind,te_ind in kf.split(X_train,Y_train):
        
        tr_X = X_train.iloc[tr_ind]
        tr_Y = Y_train.iloc[tr_ind]
        te_X = X_train.iloc[te_ind]
        te_Y = Y_train.iloc[te_ind]
        print(f'Training in progress for iteration {iter}')
        iter+=1
        cl.fit(tr_X,tr_Y)
        pred_scores = cl.predict_proba(te_X)[:,1]
        X_copy.loc[X_copy.iloc[te_ind].index,'pred_scores_' + (clName)] = pred_scores
        #X_copy.iloc[te_ind].loc[:,'pred_scores_' + (clName)] = pred_scores
        
        # Compute ROC curve and ROC area for each class
        fpr, tpr, _ = roc_curve(te_Y, pred_scores)
        class_df.loc[clName,'auc'] += auc(fpr, tpr)
    class_df.loc[clName,'auc'] /= nfolds<jupyter_output>Testing RandomForestCla: 
Training in progress for iteration 1<jupyter_text>Add columns "pred_probas_"+classifiername<jupyter_code>for clname in class_df.index:
    if clname.startswith('LogisticReg'):
        X_copy['pred_probas_'+clname] = X_copy['pred_scores_' + clname]
    else:
        logreg = sk.linear_model.LogisticRegression(n_jobs=-1)
        logreg.fit(X_copy[['pred_scores_' + clname]],Y_train)
        X_copy['pred_probas_'+clname] = logreg.predict_proba(X_copy[['pred_scores_' + clname]])[:,1]<jupyter_output>c:\program files\python36\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
c:\program files\python36\lib\site-packages\sklearn\linear_model\logistic.py:1544: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = 12.
  " = {}.".format(effective_n_jobs(self.n_jobs)))
c:\program files\python36\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
c:\program files\python36\lib\site-packages\sklearn\linear_model\logistic.py:1544: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = 12.
  " = {}.".format(effective_n_jobs(self.n_jobs)))
c:\program files\python36\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default s[...]<jupyter_text>Find which classifier has the smallest brier score<jupyter_code>from sklearn.metrics import brier_score_loss

best_classifier_name = ''
lowest_brier_loss = 10000
for clname in class_df.index:
    score = brier_score_loss(Y_train,X_copy['pred_probas_'+clname])
    print(f'{clname} obtains a brier score of {score}')
    class_df.loc[clname,'brier'] = score
    if score < lowest_brier_loss:
        lowest_brier_loss = score
        best_classifier_name = clname
class_df<jupyter_output><empty_output><jupyter_text>The best classifier is...<jupyter_code>best_classifier_name<jupyter_output><empty_output><jupyter_text>Calculate predicted probabilities on test set Now we have the brier losses over the training set. Train on the whole training set and label the test set<jupyter_code>best_classifier = classifiers_by_name[best_classifier_name]
best_classifier.fit(X_train,Y_train)
pred_scores = pd.DataFrame(index=X_test.index, 
        data = best_classifier.predict_proba(X_test)[:,1],
        columns = ['score']
           )
X_train_2 = X_train.copy()
X_train_2['score'] = best_classifier.predict_proba(X_train_2)[:,1]
if best_classifier_name.startswith('LogisticReg'):
    pred_probas = pred_scores
    data=pred_probas['score'].sort_values().values
else:
    logreg = sk.linear_model.LogisticRegression(n_jobs=-1)
    logreg.fit(X_train_2[['score']],Y_train)

    pred_probas = logreg.predict_proba(pred_scores)[:,1]
    data = pred_probas<jupyter_output>c:\program files\python36\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
  FutureWarning)
c:\program files\python36\lib\site-packages\sklearn\linear_model\logistic.py:1544: UserWarning: 'n_jobs' > 1 does not have any effect when 'solver' is set to 'liblinear'. Got 'n_jobs' = 12.
  " = {}.".format(effective_n_jobs(self.n_jobs)))<jupyter_text>Let's analyze the no-show probabilities of the best classifier AUC on test set<jupyter_code>import sklearn
sklearn.metrics.roc_auc_score(Y_test, pred_scores)<jupyter_output><empty_output><jupyter_text>Brier's score on the test set<jupyter_code>brier_score_loss(Y_test,pred_probas)
X_test['Y_pred'] = pred_probas#.score
X_test['Y_actual'] = Y_test<jupyter_output>c:\program files\python36\lib\site-packages\ipykernel_launcher.py:1: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  """Entry point for launching an IPython kernel.
c:\program files\python36\lib\site-packages\ipykernel_launcher.py:2: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy<jupyter_text>Write to file<jupyter_code>X_test.to_csv('test_set_with_predictions.csv')<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup