<jupyter_start><jupyter_text>Mini-Lab: Logistic Regression and SVMs (Modeling) __Team Members__- Name 1: <NAME>- Name 2: <NAME>- Name 3: <NAME>- Name 4: <NAME><jupyter_code>import pandas as pd
import numpy as np

hotel_bookings = pd.read_csv('https://raw.githubusercontent.com/olmosjorge28/DS7331_ML1_ASSINGMENTS/main/hotel_bookings.csv') # read in the csv file
hotel_bookings.head()<jupyter_output><empty_output><jupyter_text>Data Importing and Cleaning<jupyter_code>import math

hotel_bookings_transform = hotel_bookings
hotel_bookings_transform= hotel_bookings_transform.drop(columns=['agent','company', 'country'])

hotel_bookings_transform['meal']= hotel_bookings_transform['meal'].apply(lambda x: 'SC' if  x == 'Undefined'  else x)
hotel_bookings_transform['children']= hotel_bookings_transform['children'].apply(lambda x: 0 if  math.isnan(x)  else x)
hotel_bookings_transform['market_segment']= hotel_bookings_transform['market_segment'].apply(lambda x: 'Online TA' if  x == 'Undefined'  else x)
hotel_bookings_transform['distribution_channel']= hotel_bookings_transform['distribution_channel'].apply(lambda x: 'TA/TO' if  x == 'Undefined'  else x)


hotel_bookings_transform['hotel']=hotel_bookings_transform['hotel'].astype('category').cat.codes
hotel_bookings_transform['market_segment']=hotel_bookings_transform['market_segment'].astype('category').cat.codes
hotel_bookings_transform['meal']=hotel_bookings_transform['meal'].astype('category').cat.codes
hotel_bookings_transform['reservation_status']=hotel_bookings_transform['reservation_status'].astype('category').cat.codes
hotel_bookings_transform['distribution_channel']=hotel_bookings_transform['distribution_channel'].astype('category').cat.codes
hotel_bookings_transform['reserved_room_type']=hotel_bookings_transform['reserved_room_type'].astype('category').cat.codes
hotel_bookings_transform['assigned_room_type']=hotel_bookings_transform['assigned_room_type'].astype('category').cat.codes
hotel_bookings_transform['deposit_type']=hotel_bookings_transform['deposit_type'].astype('category').cat.codes
hotel_bookings_transform['customer_type']=hotel_bookings_transform['customer_type'].astype('category').cat.codes

assign_month = {"arrival_date_month": {"January": 1, "February": 2, "March": 3, "April": 4,"May": 5,"June": 6,"July": 7,"August": 8,"September": 9,"October": 10,"November": 11,"December": 12}}
hotel_bookings_transform = hotel_bookings_transform.replace(assign_month)

# Convert Date to Ordinal
import datetime as dt
hotel_bookings_transform['reservation_status_date'] = pd.to_datetime(hotel_bookings_transform['reservation_status_date'])
hotel_bookings_transform['reservation_status_date'] = hotel_bookings_transform['reservation_status_date'].map(dt.datetime.toordinal)<jupyter_output><empty_output><jupyter_text>Select modeling techniques<jupyter_code>## todo:<jupyter_output><empty_output><jupyter_text>Generate test design:   Normalization - SVM tries to maximize the distance between the support vectors and the separating plane. - Prior to conducting SVM, we need to normalize the dataset so we can get rid of bias and dominance of certain features over others. - The result of applying normalized dataset will be having the same influence on the distance metric, thus boosting the accuracy of the model.<jupyter_code>X=hotel_bookings_transform.drop(columns=['is_canceled'])
y=hotel_bookings_transform.is_canceled
from sklearn import preprocessing
scaler = preprocessing.StandardScaler().fit(X)
scaler

scaler.mean_
scaler.scale_

X_scaled = scaler.transform(X)<jupyter_output><empty_output><jupyter_text>Verifying each feature has a mean of zero:<jupyter_code>X_scaled.mean(axis=0)<jupyter_output><empty_output><jupyter_text>Verifying that the standard deviation should be 1:<jupyter_code>X_scaled.std(axis=0)<jupyter_output><empty_output><jupyter_text>Feature Selection  Running initial model to select best parameters to be used for RFE selection:<jupyter_code>from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import ShuffleSplit

cv = ShuffleSplit(n_splits=10, test_size=0.10, random_state=0)
regEstimator = LogisticRegression()

parameters = { 'penalty':['l2']
              ,'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]
              ,'class_weight': ['balanced','none']
              ,'random_state': [10]
              ,'solver': ['lbfgs']
              ,'max_iter':[500,1000]
             }


from sklearn.model_selection import GridSearchCV
regGridSearch = GridSearchCV(estimator=regEstimator
                   , n_jobs=8 # jobs to run in parallel
                   , verbose=1 # low verbosity
                   , param_grid=parameters
                   , cv=cv # KFolds = 10
                   , scoring='accuracy')

regGridSearch.fit(X_scaled, y)<jupyter_output>Fitting 10 folds for each of 28 candidates, totalling 280 fits<jupyter_text>RFE Selection<jupyter_code>from sklearn.feature_selection import RFECV

#Create a regression estimator with best parameters for cross validation
classifierEst = regGridSearch.best_estimator_

rfecv = RFECV(estimator=classifierEst, step=1, cv=cv, scoring='accuracy')
rfecv.fit(X_scaled, y)<jupyter_output>/Users/boyun/opt/anaconda3/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(
/Users/boyun/opt/anaconda3/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logis[...]<jupyter_text>Here we're examining the plot of the categorical values of interest and looking at the optimal number of features:<jupyter_code># Examine categorical variables of interest  
import matplotlib.pyplot as plt
%matplotlib inline

print("Optimal number of features : %d" % rfecv.n_features_)

# Plot number of features VS. cross-validation scores
plt.figure()
plt.xlabel("Number of features selected")
plt.ylabel("Cross validation score (nb of correct classifications)")
plt.plot(range(1, len(rfecv.grid_scores_) + 1), rfecv.grid_scores_)
plt.show()
#Plot the RFE Rankings
import matplotlib. pyplot as plt
plt.style.use('ggplot')

rfe_ft_imp_df = pd.DataFrame({'feature_names':X.columns, 'weights':rfecv.grid_scores_})
rfe_ft_imp_df.sort_values(by='weights', inplace=True, ascending=False )

top_features = rfe_ft_imp_df.head(50)

top_features<jupyter_output><empty_output><jupyter_text>Data split using Cross Validation<jupyter_code>from sklearn.model_selection import StratifiedShuffleSplit 
from sklearn.model_selection import train_test_split
from sklearn import metrics as mt

num_cv_iterations = 3
num_instances = len(y)
cv_object = ShuffleSplit(n_splits=num_cv_iterations,test_size  = 0.2)


# okay, so run through the cross validation loop and set the training and testing variable for one single iteration
for train_indices, test_indices in cv_object.split(X_scaled,y): 
    # I will create new variables here so that it is more obvious what 
    # the code is doing (you can compact this syntax and avoid duplicating memory,
    # but it makes this code less readable)
    X_train = X_scaled[train_indices]
    y_train = y[train_indices]
    
    X_test = X_scaled[test_indices]
    y_test = y[test_indices]
    
    
print("shape of original dataset :", hotel_bookings_transform.shape)
print("shape of input - training set", X_train.shape)
print("shape of output - training set", y_train.shape)
print("shape of input - testing set", X_test.shape)
print("shape of output - testing set", y_test.shape)<jupyter_output>shape of original dataset : (119390, 29)
shape of input - training set (95512, 28)
shape of output - training set (95512,)
shape of input - testing set (23878, 28)
shape of output - testing set (23878,)<jupyter_text>Build model To build our model, after analyzing our feature selection we decided to keep all features that had weights greater than .9  Logistic Regression<jupyter_code># Changing X_train, Y_train array into dataframe

import pandas as pd    
import numpy as np   
Columns=X.columns
X_train = pd.DataFrame(X_train, columns=Columns)
X_train
# Selecting features that are significant from RFE result

X_train = X_train[['hotel',
'arrival_date_year',
'arrival_date_month',
'arrival_date_week_number',
'lead_time',
'arrival_date_day_of_month',
'stays_in_weekend_nights',
'stays_in_week_nights',
'adults',
'children',
'babies',
'meal',
'previous_cancellations',
'previous_bookings_not_canceled',
'distribution_channel',
'is_repeated_guest',
'market_segment',
'reserved_room_type',
'assigned_room_type',
'booking_changes',
'deposit_type',
'customer_type',
'days_in_waiting_list',
'adr']]


# Changing back to array form to be able to fit in Sklearn models
X_train=X_train.to_numpy()<jupyter_output><empty_output><jupyter_text>Doing f1 scoring, which is best for a binary target like is_canceled<jupyter_code>from sklearn.linear_model import LogisticRegression
regEstimator = LogisticRegression()



tuning_parameters = {'penalty':['l2']
              ,'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]
              ,'class_weight': ['balanced', 'none']
              ,'random_state': [10]
              ,'solver': ['lbfgs']
              ,'max_iter':[100,500]
             }


#Doing f1 for accuracy best for binary
#Create a grid search object using the  

from sklearn.model_selection import GridSearchCV
regGridSearch = GridSearchCV(regEstimator
                   , n_jobs=8 # jobs to run in parallel
                   , verbose=1 # low verbosity
                   , param_grid=tuning_parameters
#                    , cv=cv # KFolds = 10
                   , scoring='f1')

#Perform hyperparameter search to find the best combination of parameters for our data
regGridSearch.fit(X_train, y_train)
 regGridSearch.best_params_
regGridSearch.best_score_
 regGridSearch.best_estimator_.intercept_
regGridSearch.best_estimator_.coef_<jupyter_output><empty_output><jupyter_text>Support Vector Machines- Due to a computational issue, we decided to downsample our data (90% reduction).- Stochastic Gradient descent is utilized. - We decided to perform a Non-linear Kernel SVM and compare the result with the linear SVM. - With RBF, tuning hyperparameters, such as gamma(controls the size of the radial basis function kernel) and C(margin hardness), are allowed to find the optimal model setting. - We will explore different combinations of gamma and C to see which one gives us the best accuracy.   Stochastic Gradient Descent  Downsampling data<jupyter_code># Downsample data
hotel_bookings_transform_d=hotel_bookings_transform.sample(n=10000, random_state=1)

X_d=hotel_bookings_transform_d.drop(columns=['is_canceled'])
y_d=hotel_bookings_transform_d.is_canceled


from sklearn import preprocessing
scaler = preprocessing.StandardScaler().fit(X_d)
scaler

scaler.mean_
scaler.scale_

X_scaled_d = scaler.transform(X_d)


from sklearn.model_selection import StratifiedShuffleSplit 
from sklearn.model_selection import train_test_split
from sklearn import metrics as mt


num_cv_iterations = 3
num_instances = len(y_d)
cv_object = ShuffleSplit(n_splits=num_cv_iterations,test_size  = 0.2)


# okay, so run through the cross validation loop and set the training and testing variable for one single iteration
for train_indices, test_indices in cv_object.split(X_scaled_d,y_d): 
    # I will create new variables here so that it is more obvious what 
    # the code is doing (you can compact this syntax and avoid duplicating memory,
    # but it makes this code less readable)
    X_train_d = X_scaled_d[train_indices]
    y_train_d = y_d[train_indices]
    
    X_test_d = X_scaled_d[test_indices]
    y_test_d = y_d[test_indices]
    
    
print("shape of original dataset :", X_scaled.shape)
print("shape of input - training set", X_train_d.shape)
print("shape of output - training set", y_train_d.shape)
print("shape of input - testing set", X_test_d.shape)
print("shape of output - testing set", y_test_d.shape)    
# Import linear svm model
from sklearn import svm

# Create a svm Classifier called "lin"
lin = svm.SVC(kernel='linear') # Linear Kernel

# Train the model using the training sets
lin.fit(X_train_d, y_train_d)

# Make prediction for test dataset
y_pred_lin = lin.predict(X_test_d)
from sklearn import svm


svm_estimator = svm.SVC()

tuned_parameters = [{'kernel': ['rbf'], 'gamma': [1e-3, 1e-4],
                     'C': [1, 10, 100, 1000]},
                    {'kernel': ['linear'], 'C': [1, 10, 100, 1000]}]


svmGridSearch = GridSearchCV(svm_estimator
                   , n_jobs=8 # jobs to run in parallel
                   , verbose=1 # low verbosity
                   , param_grid=tuned_parameters
#                    , cv=cv # KFolds = 10
                   , scoring='accuracy')

svmGridSearch.fit(X_train, y_train)

means = clf.cv_results_['mean_test_score']
stds = clf.cv_results_['std_test_score']


classification_report(y_true, y_pred)<jupyter_output>Fitting 5 folds for each of 12 candidates, totalling 60 fits
StarcoderdataJupyterStructuredCleanDedup