<jupyter_start><jupyter_text>LEAD SCORING CASE STUDY **Business Objective : An education company named X Education sells online courses to industry professionals.Now, although X Education gets a lot of leads, its lead conversion rate is very poor. For example, if, say, they acquire 100 leads in a day, only about 30 of them are converted. The objective is to build a model to identify the hot/potential leads and achieve lead conversion rate to 80%.**  Data Sourcing<jupyter_code># Suppressing Warnings
import warnings
warnings.filterwarnings('ignore')
# Importing Pandas and NumPy
import pandas as pd
import numpy as np

import matplotlib.pyplot as plt
%matplotlib inline

import seaborn as sns

import sklearn
from sklearn.preprocessing import StandardScaler

import statsmodels.api as sm<jupyter_output><empty_output><jupyter_text>Data Reading<jupyter_code>#Importing the dataset to a csv file.
lead = pd.read_csv("C:/Users/hp/Downloads/Lead Scoring Assignment/Leads.csv")
#Displaying the content of the data
lead.head()
#Inspecting the shape
print(lead.shape)
#Inspecting the columns
print(lead.columns)
#inspecting the datatypes of variables
print(lead.dtypes)
#Listing down all the columns along with name ,no of non null values,datatype,memory usage
print(lead.info())
#Describing Mean,std,min,25%,50%,75%,max of the dataset
print(lead.describe())<jupyter_output>Lead Number    Converted  TotalVisits  Total Time Spent on Website  \
count    9240.000000  9240.000000  9103.000000                  9240.000000   
mean   617188.435606     0.385390     3.445238                   487.698268   
std     23405.995698     0.486714     4.854853                   548.021466   
min    579533.000000     0.000000     0.000000                     0.000000   
25%    596484.500000     0.000000     1.000000                    12.000000   
50%    615479.000000     0.000000     3.000000                   248.000000   
75%    637387.250000     1.000000     5.000000                   936.000000   
max    660737.000000     1.000000   251.000000                  2272.000000   

       Page Views Per Visit  Asymmetrique Activity Score  \
count           9103.000000                  5022.000000   
mean               2.362820                    14.306252   
std                2.161418                     1.386694   
min                0.000000                     [...]<jupyter_text>Data Cleaning  **The following steps will be done in data cleaning process :-**    1. "Select" value will be replaced with NaN2. Calculating missing value percentage for each column and dropping score variables.3. Dropping the columns with high percentage of missing values.4. Checking the unique category for each column.5.  If the columns is highly skewed with one category,such columns will be dropped.    Combining different categories of the columns with less percentage values into "Others" category.6. Imputing the column with least missing values percentage.7. Finally Checking for the number of rows retained after performing all the above steps.  Replace all Select values in the dataframe with np.nan **Reason to remove the select variable :** Select variable does not provide any information regarding the data, It just means that the user has not answered anything for the particular question and it is better to treat them as missing values.<jupyter_code>#Inspecting all the columns with Select variable in the dataframe.
colsHavingSelect = lead.isin(["Select"]).any()
print(colsHavingSelect)
#df.replace() will replace all the occurances of the given string to missing values.
lead.replace('Select', np.nan,inplace=True)<jupyter_output><empty_output><jupyter_text>Calculating the missing value percentage in each column<jupyter_code>#Percentage of missing values in each column.
nullseries = round(lead.isnull().sum() * 100 / len(lead),2)
print(nullseries)<jupyter_output>Prospect ID                                       0.00
Lead Number                                       0.00
Lead Origin                                       0.00
Lead Source                                       0.39
Do Not Email                                      0.00
Do Not Call                                       0.00
Converted                                         0.00
TotalVisits                                       1.48
Total Time Spent on Website                       0.00
Page Views Per Visit                              1.48
Last Activity                                     1.11
Country                                          26.63
Specialization                                   36.58
How did you hear about X Education               78.46
What is your current occupation                  29.11
What matters most to you in choosing a course    29.32
Search                                            0.00
Magazine                                          0.00
Newspaper [...]<jupyter_text>Observation : There are high missing value percentages in many columns and we must remove them. **Data Cleaning - Removing (Score variable) all unnecessary information from the dataframe.****Reason to remove score variables :** Score variable are those variables in the dataframe where the sales team has already contacted the lead, since the lead has already been contacted it does not add more value to our analysis, So we drop these variables.**Dropping the below score variable first**- Tags- Lead Quality- Lead Profile- Asymmetrique Activity Index- Asymmetrique Profile Index- Asymmetrique Activity Score- Asymmetrique Profile Score<jupyter_code>#Dropping all score variables in the dataframe.
lead.drop(['Tags','Lead Quality','Lead Profile','Asymmetrique Activity Index','Asymmetrique Profile Index','Asymmetrique Activity Score','Asymmetrique Profile Score'],axis=1,inplace=True)
#When we inspect the column we can see that above score variables are removed.
lead.columns
# Also remove the 'Last Notable Activity' column as this column is recorded once the Sales team is in contact with the lead.
lead.drop('Last Notable Activity', axis = 1, inplace=True)
#Percentage of missing values in each column.
nullseries = round(lead.isnull().sum() * 100 / len(lead),2)
print(nullseries)<jupyter_output>Prospect ID                                       0.00
Lead Number                                       0.00
Lead Origin                                       0.00
Lead Source                                       0.39
Do Not Email                                      0.00
Do Not Call                                       0.00
Converted                                         0.00
TotalVisits                                       1.48
Total Time Spent on Website                       0.00
Page Views Per Visit                              1.48
Last Activity                                     1.11
Country                                          26.63
Specialization                                   36.58
How did you hear about X Education               78.46
What is your current occupation                  29.11
What matters most to you in choosing a course    29.32
Search                                            0.00
Magazine                                          0.00
Newspaper [...]<jupyter_text>Data Cleaning - Treating all missing values in the dataframe (~40%) **Drop columns that has missing value percentage of about 38-40% approximately.**<jupyter_code>#Checking columns having null value greater than or equal to 38%
print(nullseries[nullseries >= 38])
#Dropping all columns having a missing value percentage greater than 38%.
lead = lead.drop(lead.loc[:,list((100*(lead.isnull().sum()/len(lead.index))>=38))].columns, 1)
#Percentage of missing values in each column.
nullseries = round(lead.isnull().sum() * 100 / len(lead),2)
print(nullseries)<jupyter_output>Prospect ID                                       0.00
Lead Number                                       0.00
Lead Origin                                       0.00
Lead Source                                       0.39
Do Not Email                                      0.00
Do Not Call                                       0.00
Converted                                         0.00
TotalVisits                                       1.48
Total Time Spent on Website                       0.00
Page Views Per Visit                              1.48
Last Activity                                     1.11
Country                                          26.63
Specialization                                   36.58
What is your current occupation                  29.11
What matters most to you in choosing a course    29.32
Search                                            0.00
Magazine                                          0.00
Newspaper Article                                 0.00
X Educatio[...]<jupyter_text>Data Cleaning - Checking for unique category - Combining categories  having low values to make a meaningful category<jupyter_code>#Checking uniqueness in all categorical variables
lead['Lead Origin'].value_counts(normalize=True)
#Checking uniqueness in all categorical variables
lead['Lead Source'].value_counts(normalize=True)
#Checking uniqueness in all categorical variables
lead['Do Not Email'].value_counts(normalize=True)
#Checking uniqueness in all categorical variables
lead['Do Not Call'].value_counts(normalize=True)
#Checking uniqueness in all categorical variables
lead['Last Activity'].value_counts(normalize=True)
#Checking uniqueness in all categorical variables
lead['Country'].value_counts(normalize=True)
#Checking uniqueness in all categorical variables
lead['Specialization'].value_counts(normalize=True)
#Checking uniqueness in all categorical variables
lead['What is your current occupation'].value_counts(normalize=True)
#Checking uniqueness in all categorical variables
lead['What matters most to you in choosing a course'].value_counts(normalize=True) 
#Checking uniqueness in all categorical variables
lead['Search'].value_counts(normalize=True) 
#Checking uniqueness in all categorical variables
lead['Magazine'].value_counts(normalize=True) 
#Checking uniqueness in all categorical variables
lead['Newspaper Article'].value_counts(normalize=True) 
#Checking uniqueness in all categorical variables
lead['X Education Forums'].value_counts(normalize=True) 
#Checking uniqueness in all categorical variables
lead['Digital Advertisement'].value_counts(normalize=True) 
#Checking uniqueness in all categorical variables
lead['Through Recommendations'].value_counts(normalize=True) 
#Checking uniqueness in all categorical variables
lead['Receive More Updates About Our Courses'].value_counts(normalize=True) 
#Checking uniqueness in all categorical variables
lead['Get updates on DM Content'].value_counts(normalize=True) 
#Checking uniqueness in all categorical variable
lead['Update me on Supply Chain Content'].value_counts(normalize=True) 
#Checking uniqueness in all categorical variable
lead['I agree to pay the amount through cheque'].value_counts(normalize=True) 
#Checking uniqueness in all categorical variable
lead['A free copy of Mastering The Interview'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>**Data Cleaning - Dropping Highly Skewed Columns** **Dropping highky skewed columns is important because : there is a high data imbalance amongst the columns and it does not add more value to model building.****Following Columns To Be Drop:**- Do Not Email- Do Not Call- Country - What matters most to you in choosing a course - Search- Magazine- Newspaper Article  - X Education Forums- Newspaper- Digital Advertisement - Through Recommendations- Receive More Updates About Our Courses - Update me on Supply Chain Content- Get updates on DM Content - I agree to pay the amount through cheque<jupyter_code>#Using drop to remove columns with high data imbalance 
lead.drop(['Do Not Email','Do Not Call','Country','What matters most to you in choosing a course', 
'Search','Magazine','Newspaper Article', 'X Education Forums','Newspaper','Digital Advertisement','Through Recommendations',
'Receive More Updates About Our Courses', 'Update me on Supply Chain Content','Get updates on DM Content', 
'I agree to pay the amount through cheque'],axis=1,inplace=True)
#Inspecting to check if the columns have been deleted carefully.
lead.columns
#Percentage of missing values in each column.
nullseries = round(lead.isnull().sum() * 100 / len(lead),2)
print(nullseries)
print("Total number of columns :", len(lead.columns))<jupyter_output>Total number of columns : 12<jupyter_text>Data Cleaning - Combining columns having low percentages into one single column. **Following are the columns in which categories are to be combined and form "Others" category :-**- Lead Origin- Lead Source- Last Activity- Specialization- What is your current occupation  **Lead Origin**<jupyter_code>#Inspecting the column again
lead['Lead Origin'].value_counts(normalize=True)
#Combining categories with very low row percentages as it does not make sense to create dummies for such rows.
lead['Lead Origin'] = lead['Lead Origin'].replace(['Lead Import','Quick Add Form'] ,'Others')
#Checking if the category has been correctly replaced. 
lead['Lead Origin'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>**Now, Performing the same for other columns** **Lead Source**<jupyter_code>#Inspecting Lead Source column.
lead['Lead Source'].value_counts(normalize=True) 
#Combining categories
lead['Lead Source'] = lead['Lead Source'].replace(['Welingak Website','Referral Sites','Facebook','bing','google','Click2call', 
                                                   'Press_Release','Social Media','Live Chat','Pay per Click Ads','blog','WeLearn',
                                                   'youtubechannel','testone','NC_EDM' ,'welearnblog_Home'] ,'Others')
#Inspecting Lead Source column.
lead['Lead Source'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>**Last Activity**<jupyter_code>#Inspecting 'Last Activity' column
lead['Last Activity'].value_counts(normalize=True) 
#Combining together to form 'Others' Categories. 
lead['Last Activity']=lead['Last Activity'].replace(['Form Submitted on Website'      ,
'Unreachable'                    ,
'Unsubscribed'                   ,
'Had a Phone Conversation'       ,
'Approached upfront'             ,
'View in browser link Clicked'   ,
'Email Received'                 ,
'Email Marked Spam'              ,
'Visited Booth in Tradeshow'     ,
'Resubscribed to emails'],'Others') 
#Inspecting 'Last Activity' column
lead['Last Activity'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>**Specialization**<jupyter_code>#Inspecting the specialization column
lead['Specialization'].value_counts(normalize=True)
#Combining categories < 2%  into 'Others'
lead['Specialization']=lead['Specialization'].replace(['Rural and Agribusiness','E-Business','Services Excellence','Hospitality Management','E-COMMERCE','Retail Management'],'Others')      
#Inspecting the column again to recheck the same.
lead['Specialization'].value_counts(normalize=True)
#We also handle the missing values for this Specialisation column, we add another column known as "Not Specified" for all missing values.
lead.Specialization.fillna('Not Specified', inplace=True)
#Inspecting the column again to recheck the same.
lead['Specialization'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>**Current Occupation**<jupyter_code>#Inspecting value counts for Current Occupation
lead['What is your current occupation'].value_counts(normalize=True)
#Combining categories 
lead['What is your current occupation']=lead['What is your current occupation'].replace(['Other','Housewife','Businessman'],'Others')
#Rechecking to find if the categories are correctly combined.
lead['What is your current occupation'].value_counts(normalize=True)<jupyter_output><empty_output><jupyter_text>Imputing columns with low null value percentages- **Generally categorical columns are imputed with 'mode'.**- **And, Numerical columns are imputed with mean or median.**<jupyter_code>#Percentage of missing values in each column.
nullseries = round(lead.isnull().sum() * 100 / len(lead),2)
print(nullseries)<jupyter_output>Prospect ID                                0.00
Lead Number                                0.00
Lead Origin                                0.00
Lead Source                                0.39
Converted                                  0.00
TotalVisits                                1.48
Total Time Spent on Website                0.00
Page Views Per Visit                       1.48
Last Activity                              1.11
Specialization                             0.00
What is your current occupation           29.11
A free copy of Mastering The Interview     0.00
dtype: float64<jupyter_text>**Imputing categorical columns**<jupyter_code>#Imputing categorical columns with Mode
lead['Lead Source'].value_counts(normalize=True) 
#Imputing missing values in 'Lead Source' with mode. 
lead['Lead Source'].fillna('Google',inplace=True)
#Imputing categorical columns with Mode
lead['Last Activity'].value_counts(normalize=True) 
#Imputing missing values in 'Last Activity' with mode. 
lead['Last Activity'].fillna('Email Opened',inplace=True)
#Imputing categorical columns with Mode
lead['What is your current occupation'].value_counts(normalize=True) 
#Replacing categorical columns with Unemployed in Current Occupation 
lead['What is your current occupation'].fillna('Unemployed', inplace = True)<jupyter_output><empty_output><jupyter_text>**Imputing numerical columns**<jupyter_code>#Since TotalVisits is a numerical column, we first inspect the data.
lead['TotalVisits'].describe()<jupyter_output><empty_output><jupyter_text>**Observation : We can see that both mean and median = 3 and we can impute it with either, but we choose to impute it with median.**<jupyter_code>#Imputing missing values in 'TotalVisits' with mean/median which is 3. 
lead['TotalVisits'].fillna(lead['TotalVisits'].median(),inplace=True)
#Checking for distribution
lead['TotalVisits'].describe()
#Page Per Views is also a numerical column, hence we check the distribution first.
lead['Page Views Per Visit'].describe()<jupyter_output><empty_output><jupyter_text>**Observation : We can see that both mean and median ~ 2 and we can impute it with either, but we choose to impute it with median.**<jupyter_code>#Imputing missing values with mean/median which is 2. 
lead['Page Views Per Visit'].fillna(lead['Page Views Per Visit'].median(),inplace=True)
#Page Per Views is also a numerical column, hence we check the distribution first.
lead['Page Views Per Visit'].describe()<jupyter_output><empty_output><jupyter_text>Cross-Checking for missing values again.<jupyter_code>#Percentage of missing values in each column.
nullseries = round(lead.isnull().sum() * 100 / len(lead),2)
print(nullseries)<jupyter_output>Prospect ID                               0.0
Lead Number                               0.0
Lead Origin                               0.0
Lead Source                               0.0
Converted                                 0.0
TotalVisits                               0.0
Total Time Spent on Website               0.0
Page Views Per Visit                      0.0
Last Activity                             0.0
Specialization                            0.0
What is your current occupation           0.0
A free copy of Mastering The Interview    0.0
dtype: float64<jupyter_text>**Observation : There are no more missing values, we have cleaned the data thoroughly.**<jupyter_code>#Row-wise null count
print(lead.isnull().sum(axis=1))<jupyter_output>0       0
1       0
2       0
3       0
4       0
       ..
9235    0
9236    0
9237    0
9238    0
9239    0
Length: 9240, dtype: int64<jupyter_text>**Duplicates check**<jupyter_code>#Checking for any duplicates in the data.
lead.loc[lead.duplicated()]<jupyter_output><empty_output><jupyter_text>**Observation : There are no duplicate rows in the dataset.**  EDA (Exploratory Data Analysis)<jupyter_code>#Displaying the data types on different columns
lead.dtypes<jupyter_output><empty_output><jupyter_text>**EDA - Numerical Values**<jupyter_code>Required_columns= ['TotalVisits','Total Time Spent on Website','Page Views Per Visit']
sns.set_palette("dark")
plt.figure(figsize = (20, 20))

for col in enumerate(Required_columns):
    plt.subplot(4, 2, col[0]+1)
    plt.title(col[1])
    sns.distplot(lead[col[1]], hist =False)
    plt.tight_layout()
    plt.xticks(rotation = 90)<jupyter_output><empty_output><jupyter_text>**Observation :** **Uni-variate Analysis - Numerical values:**- The max probability for TotalVisits is found to be around 15-20. It increases initially but decreases further.- The max probability for PageViewsPerVisit is found to be around to be 3-5- The probability of time spent is found to be high for time between 0-300 seconds and decreases further.<jupyter_code>Required_columns= ['TotalVisits','Total Time Spent on Website','Page Views Per Visit']
sns.set_palette("bright")
plt.figure(figsize = (20, 20))

for col in enumerate(Required_columns):
    plt.subplot(4, 2, col[0]+1)
    plt.title(col[1])
    ax = sns.boxplot(x='Converted', y=col[1], data = lead)
    ax.set_yscale('log')
    plt.tight_layout()
    plt.xticks(rotation = 90)<jupyter_output><empty_output><jupyter_text>**Observation :****Bi-variate Analysis - Numerical values:**- The mean is found to be higher in case of Converted people rather than non-converted people. - The average page views for both converted and non converted is found to be the same. - The average total visits for both converted and non converted people is found to be the same. **EDA - Categorical values**<jupyter_code>#Copying from the lead dataframe for easier eda
leadCopy = lead.copy()
#Displaying the dataframe
leadCopy.head()
leadCopy['Category'] = leadCopy.Converted.astype('category')
Required_columns= ['Lead Origin','Lead Source', 'Last Activity', 'Specialization', 'What is your current occupation', 'A free copy of Mastering The Interview']
sns.set_palette("bright")

plt.figure(figsize = (15, 15))

for col in enumerate(Required_columns):
    plt.subplot(3, 2, col[0]+1)
    plt.title(col[1])
    ax = sns.countplot(data = leadCopy, x=col[1], hue='Converted') 
    plt.xticks(rotation = 90)
    plt.tight_layout()<jupyter_output><empty_output><jupyter_text>**Univariate analysis for Categorical data**- The percentage of Converted people is found to be greater for Landing Page Submission. We can also see that if Lead source is Add Form, the ratio of lead conversion is very high(almost not converted is very less).- Google is found to be the important source for Lead Conversion - We need to target people via Emails and SMS as it is found that the probability of response in case Converted leads is found to be higher. - We cannot infer much about conversion rate from specialisation as people who do not select any specialisation can also be converted to a lead.But the ratio of non converted leads is higher than converted ones if they didn't choose specialisation.- It is clearly visible from the graph that we need to target the Unemployed and Working Professional to get a higher conversion rate. The ratio of conversion rate is higher than not converted people for working professionals. - People usually do not subscribe for a free copy of mastering the interview.  Outliers Handling<jupyter_code>#Inspectig the data types of all columns
lead.dtypes
#There are 4 numerical columns - Converted , TotalVisits, Total Time Spent on Website, 'Page Views Per Visit
#Plotting them to check for outliers.

numerical_columns=['Converted','TotalVisits','Total Time Spent on Website','Page Views Per Visit']

plt.figure(figsize = (11, 11))
sns.set_palette('bright')
for col in enumerate(numerical_columns):
    plt.subplot(2,2, col[0]+1)
    sns.boxplot(x = col[1], data = lead)
    plt.xticks(rotation = 90)
    plt.tight_layout()
#TotalVisits and Page Views Per Visit have outliers
lead['TotalVisits'].describe()
#diff1 will provide the difference between outliers and 99th percentile. 
#If the value is too big, we can conclude that there are outliers.
diff1 = lead.TotalVisits.max() - lead.TotalVisits.quantile(0.99)
diff1
lead['Page Views Per Visit'].describe()
#diff2 will provide the difference between outliers and 99th percentile. 
#If the value is too big, we can conclude that there are outliers.
diff2 = lead['Page Views Per Visit'].max() - lead['Page Views Per Visit'].quantile(0.99)
diff2<jupyter_output><empty_output><jupyter_text>**Capping Outliers**- We Perform Soft Capping.- We check only for outliers in the upper range that are > 99th percentile.<jupyter_code>req_cols=['TotalVisits','Page Views Per Visit'] 
for col in req_cols:
    percentiles = lead[col].quantile(0.99)  ## capping higher range outliers
    lead[col][lead[col] >= percentiles] = percentiles
#Plotting the columns again to visualize after capping
numerical_columns=['Converted','TotalVisits','Total Time Spent on Website','Page Views Per Visit']

plt.figure(figsize = (11, 11))
sns.set_palette('bright')
for col in enumerate(numerical_columns):
    plt.subplot(2,2, col[0]+1)
    sns.boxplot(x = col[1], data = lead)
    plt.xticks(rotation = 90)
    plt.tight_layout()<jupyter_output><empty_output><jupyter_text>Data Preparation Following steps will be done as part of data preparation:1. Converting binary variable (Yes/No) "A free copy of Mastering The Interview" to 1/02. Create dummies for categorical columns3. Perform train-test split4. Perform Scaling **Converting Binary Variables**- Any kind of Data Preparation while performing Model Building involves converting Yes/No columns to binary values.<jupyter_code>varlist =['A free copy of Mastering The Interview']
def binary_map(x):
  return x.map({'Yes' : 1, "No": 0})

lead[varlist] = lead[varlist].apply(binary_map)
#Inspecting the data for the same. 
lead.head()<jupyter_output><empty_output><jupyter_text>**Preliminary step before creating dummies.**- There are some columns with long names and this might be difficult for our analysis, hence we rename these columns.<jupyter_code>#Checking for all columns
lead.columns
#Rename the columns
lead.rename(columns={'Total Time Spent on Website' : 'Time Spent', 'What is your current occupation':'Current Occupation', 'A free copy of Mastering The Interview':'Free Copy'}, inplace=True )
#Check if it has correctly been renamed 
lead.columns<jupyter_output><empty_output><jupyter_text>**Create Dummies for Categorical columns**<jupyter_code>#Checking for categorical columns in the dataframe.
lead.dtypes
#Creating a dummy variable for some of the categorical variables and dropping the first one.
dummy1 = pd.get_dummies(lead[['Lead Origin','Lead Source','Last Activity','Specialization','Current Occupation']], drop_first=True)
#Adding the results to the master dataframe
lead = pd.concat([lead, dummy1], axis=1)
#Checking if the dummy-variables are added into the dataframe correctly.
lead.head()
#Dropping the original columns
lead.drop(['Lead Origin','Lead Source','Last Activity','Specialization','Current Occupation'],axis=1,inplace=True)
#Inspecting the dataframe again 
lead.head()<jupyter_output><empty_output><jupyter_text>**Checking the datatypes of all columns**<jupyter_code>#Checking Datatypes
lead.dtypes<jupyter_output><empty_output><jupyter_text>**Observation :** - **We can see that mostly all columns are already converted into Integer/Numerical datatype.**- **Hence, we can continue with our evaluation.**  Test-Train Data Split<jupyter_code>#Importing the required libraries to divide the data into test data and training data.
from sklearn.model_selection import train_test_split
#Adding all feature columns to X Variable
X = lead.drop(['Converted','Prospect ID','Lead Number'], axis=1)
X.head()
#Adding our target column to Y Variable
y = lead['Converted']
y.head()
# Splitting the data into train and test (70%-30%)
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.7, test_size=0.3, random_state=100)<jupyter_output><empty_output><jupyter_text>**Scaling all the required variables**<jupyter_code>#Checking the dataframe again
X.head()
#We can see that columns except TotalVisits, Time Spent and Page Views Per Visit all the other columns are present in the range of 0 - 1, so we need to scale these variables as well.
scaler = StandardScaler()
X_train[['TotalVisits','Time Spent','Page Views Per Visit']] = scaler.fit_transform(X_train[['TotalVisits','Time Spent','Page Views Per Visit']])
X_train.head()
#Checking the mean and standard deviation
X_train.describe()<jupyter_output><empty_output><jupyter_text>**Observation : Standard Deviation is set to 1 and hence it is scaled correctly.** **Conversion Rate before modelling**<jupyter_code>#Checking the Converted Rate before modelling
Converted_Rate = (sum(lead['Converted'])/len(lead['Converted'].index))*100
Converted_Rate<jupyter_output><empty_output><jupyter_text>**Observation : We can see that conversion rate before modelling is found to be around 38.53%**  Model Building **Model - I**<jupyter_code># Logistic regression model
logm1 = sm.GLM(y_train,(sm.add_constant(X_train)), family = sm.families.Binomial())
logm1.fit()
print(logm1.fit().summary())
#Total number of columns
print(len(X_train.columns))<jupyter_output>35<jupyter_text>**Observation : We currently have around 35 columns for our evaluation.** **Model - II : Feature Selection using RFE**<jupyter_code>#Importing Logistic Regression module from SkLearn
from sklearn.linear_model import LogisticRegression
logreg = LogisticRegression()
#We currently have 35 features with respect to our discussion, and since 35 is a huge number we will reduce it to 20 columns using RFE.
from sklearn.feature_selection import RFE
rfe = RFE(logreg, 20) # running RFE with 20 variables as output
rfe = rfe.fit(X_train, y_train)
#Printing all the columns that are selected using RFE
rfe.support_
#List of all columns on priority ranking
list(zip(X_train.columns, rfe.support_, rfe.ranking_))
#This list will hold all the data that is being selected through RFE.
col = X_train.columns[rfe.support_]
col
#Columns that have been rejected because of RFE.
X_train.columns[~rfe.support_]<jupyter_output><empty_output><jupyter_text>**Assessing the model that has already been generated through RFE using Statsmodel - Model 2**<jupyter_code>X_train_sm = sm.add_constant(X_train[col])
logm2 = sm.GLM(y_train,X_train_sm, family = sm.families.Binomial())
res = logm2.fit()
print(res.summary())
# Check for the VIF values of the feature variables. 
from statsmodels.stats.outliers_influence import variance_inflation_factor
# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train[col].columns
vif['VIF'] = [variance_inflation_factor(X_train[col].values, i) for i in range(X_train[col].shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif<jupyter_output><empty_output><jupyter_text>**Observations:-**- **We see there are many variables with a high P-value and VIF value.**- **We need to continue model building process to improve the model.**- **Factors that are needed for consideration : VIF < 5% and P-Value < 5%** **Inferences from the above  :-**- **Lead Source_Others - has a P-value of 56.3%.** - **The above model has 20 features, and we need to reduce it further.**- **Current Occupation_Unemployed has a VIF score of ~18.** **Model- III [Dropping Lead_Source Others]**<jupyter_code>col = col.drop('Lead Source_Others')
col
print(len(col))
# Let's re-run the model using the selected variables
X_train_sm = sm.add_constant(X_train[col])
logm3 = sm.GLM(y_train,X_train_sm, family = sm.families.Binomial())
res = logm3.fit()
print(res.summary())
# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train[col].columns
vif['VIF'] = [variance_inflation_factor(X_train[col].values, i) for i in range(X_train[col].shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif<jupyter_output><empty_output><jupyter_text>**Inferences from the above model :-**- **Lead Origin_Others - has a P-value of 40.0%.**- **The above model has 19 features, and we need to reduce it further.**- **Current Occupation_Unemployed has a VIF score of ~18.** **For Model 4 - We need to remove Lead Origin_Others** **Model - IV [Drop Lead Origin_Others]**<jupyter_code>col = col.drop('Lead Origin_Others')
col
print(len(col))
#Let's re-run the model using the selected variables
X_train_sm = sm.add_constant(X_train[col])
logm4 = sm.GLM(y_train,X_train_sm, family = sm.families.Binomial())
res = logm4.fit()
print(res.summary())
#Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train[col].columns
vif['VIF'] = [variance_inflation_factor(X_train[col].values, i) for i in range(X_train[col].shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif<jupyter_output><empty_output><jupyter_text>**Inferences from the above model :-**- **Current Occupation_Student - has a P-value of 32.47%.**- **The above model has 18features, and we need to reduce it further.**- **Current Occupation_Unemployed has a VIF score of ~18.** **For Model - 5, we need to remove Current Occupation_Student and then proceed further.** **Model - V [Drop Current_Occupation_Student]**<jupyter_code>#Dropping "Current Occupation_Student" for further evaluation
col = col.drop('Current Occupation_Student')
#Print all the features and the total number of features that we consider for further model building.
print(col)
print(len(col))
#Let's re-run the model using the selected variables
X_train_sm = sm.add_constant(X_train[col])
logm5 = sm.GLM(y_train,X_train_sm, family = sm.families.Binomial())
res = logm5.fit()
print(res.summary())
# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train[col].columns
vif['VIF'] = [variance_inflation_factor(X_train[col].values, i) for i in range(X_train[col].shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif
# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train[col].columns
vif['VIF'] = [variance_inflation_factor(X_train[col].values, i) for i in range(X_train[col].shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif<jupyter_output><empty_output><jupyter_text>**Inferences from the above model :-**- **Specialization International_Business - has a P-value of 13.5%.** - **The above model has 17 features, and we need to reduce it further.**- **Current Occupation_Unemployed has a VIF score of ~13.** **For further model building [Model - 6], we need to remove Specialization International_Business and proceed.** **Model - VI [Drop Specialization_International Business]**<jupyter_code>#Dropping "Specialization International_Business" for model building improvement
col = col.drop('Specialization_International Business', 1)
col
print(len(col))
#Let's re-run the model using the selected variables
X_train_sm = sm.add_constant(X_train[col])
logm6 = sm.GLM(y_train,X_train_sm, family = sm.families.Binomial())
res = logm6.fit()
print(res.summary())<jupyter_output>Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:              Converted   No. Observations:                 6468
Model:                            GLM   Df Residuals:                     6451
Model Family:                Binomial   Df Model:                           16
Link Function:                  logit   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -2709.9
Date:                Wed, 26 Jan 2022   Deviance:                       5419.9
Time:                        02:56:23   Pearson chi2:                 8.10e+03
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
===========================================================================================================
                       [...]<jupyter_text>**Inferences from the above model :-**- **Last Activity_Email Link Clicked - has a P-value of 11.5%.** - **The above model has 16 features, and we need to reduce it further.**- **Current Occupation_Unemployed has a VIF score of ~13.** **For further model building [Model - 7], we need to remove Last Activity_Email Link Clicked and proceed.** **Model - VII [Drop Last Activity_Email Link Clicked ]**<jupyter_code>#Dropping "Last Activity Email Link Clicked" to improve the model.
col = col.drop('Last Activity_Email Link Clicked', 1)
#Printing all the required features considered for model building
print(col)
print(len(col))
# Let's re-run the model using the selected variables
X_train_sm = sm.add_constant(X_train[col])
logm7 = sm.GLM(y_train,X_train_sm, family = sm.families.Binomial())
res = logm7.fit()
print(res.summary())<jupyter_output>Generalized Linear Model Regression Results                  
==============================================================================
Dep. Variable:              Converted   No. Observations:                 6468
Model:                            GLM   Df Residuals:                     6452
Model Family:                Binomial   Df Model:                           15
Link Function:                  logit   Scale:                          1.0000
Method:                          IRLS   Log-Likelihood:                -2711.1
Date:                Wed, 26 Jan 2022   Deviance:                       5422.3
Time:                        02:56:23   Pearson chi2:                 8.11e+03
No. Iterations:                     6                                         
Covariance Type:            nonrobust                                         
===========================================================================================================
                       [...]<jupyter_text>**Inferences from the above model :-**- **All P-Values are less than 5%, now lets check for VIF.**- **The above model has 15 features, and we need to reduce it further.**<jupyter_code>#Check for the VIF values of the feature variables. 
from statsmodels.stats.outliers_influence import variance_inflation_factor
#Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train[col].columns
vif['VIF'] = [variance_inflation_factor(X_train[col].values, i) for i in range(X_train[col].shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif<jupyter_output><empty_output><jupyter_text>**Current occupation_Unemployed has a high VIF of 12.20 and we need to reduce this further.** **Model - VIII [Drop Current Occupation_Unemployed ]**<jupyter_code>#Dropping "Current Occupation_Employed" because of high VIF
col = col.drop('Current Occupation_Unemployed', 1)
#Displaying all the features and the number of features for our model building. 
print(col)
print(len(col))
# Let's re-run the model using the selected variables
X_train_sm = sm.add_constant(X_train[col])
logm8 = sm.GLM(y_train,X_train_sm, family = sm.families.Binomial())
res = logm8.fit()
print(res.summary())
# Create a dataframe that will contain the names of all the feature variables and their respective VIFs
vif = pd.DataFrame()
vif['Features'] = X_train[col].columns
vif['VIF'] = [variance_inflation_factor(X_train[col].values, i) for i in range(X_train[col].shape[1])]
vif['VIF'] = round(vif['VIF'], 2)
vif = vif.sort_values(by = "VIF", ascending = False)
vif<jupyter_output><empty_output><jupyter_text>**After multiple re-runs and improving the model upto 8 times, we can say that, all the selected features have a P-value < 5%. and VIF value < 5** **Thus, 8th model is our final model.** **Prediction of Y - values, as per the final model.**<jupyter_code>#Getting the predicted values on the train set
#X_train_sm will provide the predictor variable. 
#res will contain the Logistic Regression model result
#y_train_Pred will provide us the Predicted Y Values for the given feature variables.

y_train_pred = res.predict(X_train_sm)
y_train_pred[:10]
#Reshaping the predicted probabilities so that it returns an array of Conversion_Probabilities.
y_train_pred = y_train_pred.values.reshape(-1)<jupyter_output><empty_output><jupyter_text>**Creation of DataFrame - with the predicted Y values.**<jupyter_code>#Creating a data frame with the actual Converted score and Probability of conversion
#The dataframe is stored in the variable y_train_pred_final.

y_train_pred_final = pd.DataFrame({'Converted':y_train.values, 'Conversion_Prob':y_train_pred})
y_train_pred_final['Cust_Id'] = y_train.index
y_train_pred_final.head()
#Mapping the prediction conversion Rate according to the probability cut off.
y_train_pred_final['predicted'] = y_train_pred_final.Conversion_Prob.map(lambda x: 1 if x > 0.5 else 0)

# Let's see the head
y_train_pred_final.head()<jupyter_output><empty_output><jupyter_text>**Calculation of Metrics for the final model**<jupyter_code>#Importing the required libraries for metric calculation
from sklearn import metrics

# Confusion matrix - Final Model 
confusion = metrics.confusion_matrix(y_train_pred_final.Converted, y_train_pred_final.predicted )
print(confusion)
# Let's check the overall accuracy.
print(metrics.accuracy_score(y_train_pred_final.Converted, y_train_pred_final.predicted))
#Calculation of Final Model characteristics.
TP = confusion[1,1] # true positive 
TN = confusion[0,0] # true negatives
FP = confusion[0,1] # false positives
FN = confusion[1,0] # false negatives
#Let's see the sensitivity of our logistic regression model
TP / float(TP+FN)
#Let's calculate specificity
TN / float(TN+FP)
#Calculate false postive rate - predicting churn when customer does not have churned
print(FP/ float(TN+FP))
#Negative predictive value
print (TN / float(TN+ FN))<jupyter_output>0.8152198678514468<jupyter_text>**ROC Curve and its parameters**<jupyter_code>#Function to plot and draw ROC curve.
def draw_roc( actual, probs ):
    fpr, tpr, thresholds = metrics.roc_curve( actual, probs,
                                              drop_intermediate = False )
    auc_score = metrics.roc_auc_score( actual, probs )
    plt.figure(figsize=(5, 5))
    plt.plot( fpr, tpr, label='ROC curve (area = %0.2f)' % auc_score )
    plt.plot([0, 1], [0, 1], 'k--')
    plt.xlim([0.0, 1.0])
    plt.ylim([0.0, 1.05])
    plt.xlabel('False Positive Rate or [1 - True Negative Rate]')
    plt.ylabel('True Positive Rate')
    plt.title('Receiver operating characteristic example')
    plt.legend(loc="lower right")
    plt.show()

    return None
#Calculation and FPR and TPR and thresholds for trade off Sensitivity and Specificity.
fpr, tpr, thresholds = metrics.roc_curve( y_train_pred_final.Converted, y_train_pred_final.Conversion_Prob, drop_intermediate = False )
#Invoking the function to plot ROC - Curve. 
draw_roc(y_train_pred_final.Converted, y_train_pred_final.Conversion_Prob)<jupyter_output><empty_output><jupyter_text>**The ROC curve is plotted to find the trade off between Sensitivity and Specificity.**<jupyter_code>#Let's create columns with different probability cutoffs 
numbers = [float(x)/10 for x in range(10)]
for i in numbers:
    y_train_pred_final[i]= y_train_pred_final.Conversion_Prob.map(lambda x: 1 if x > i else 0)
y_train_pred_final.head()
#Now let's calculate accuracy sensitivity and specificity for various probability cutoffs.
cutoff_df = pd.DataFrame( columns = ['prob','accuracy','sensi','speci'])
from sklearn.metrics import confusion_matrix

# TP = confusion[1,1] # true positive 
# TN = confusion[0,0] # true negatives
# FP = confusion[0,1] # false positives
# FN = confusion[1,0] # false negatives

num = [0.0,0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9]
for i in num:
    cm1 = metrics.confusion_matrix(y_train_pred_final.Converted, y_train_pred_final[i] )
    total1=sum(sum(cm1))
    accuracy = (cm1[0,0]+cm1[1,1])/total1
    
    speci = cm1[0,0]/(cm1[0,0]+cm1[0,1])
    sensi = cm1[1,1]/(cm1[1,0]+cm1[1,1])
    cutoff_df.loc[i] =[ i ,accuracy,sensi,speci]
print(cutoff_df)<jupyter_output>prob  accuracy     sensi     speci
0.0   0.0  0.381262  1.000000  0.000000
0.1   0.1  0.599567  0.972425  0.369815
0.2   0.2  0.724799  0.925385  0.601199
0.3   0.3  0.798392  0.838200  0.773863
0.4   0.4  0.818027  0.777778  0.842829
0.5   0.5  0.809060  0.671127  0.894053
0.6   0.6  0.797619  0.596918  0.921289
0.7   0.7  0.780767  0.513382  0.945527
0.8   0.8  0.756957  0.414031  0.968266
0.9   0.9  0.708565  0.252636  0.989505<jupyter_text>**Intersection between Accuracy - Sensitivity and Specificity**<jupyter_code>#Let's plot accuracy sensitivity and specificity for various probabilities.
cutoff_df.plot.line(x='prob', y=['accuracy','sensi','speci'])
plt.xticks(np.arange(0, 1, step=0.05), rotation = 90)
plt.figure(figsize=(11,11))
plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>**From the curve above, 0.35 is the optimum point to take it as a cutoff probability.**  **Calculation of predicted values on the score : 0.35**<jupyter_code>y_train_pred_final['final_predicted'] = y_train_pred_final.Conversion_Prob.map( lambda x: 1 if x > 0.35 else 0)
y_train_pred_final.head()<jupyter_output><empty_output><jupyter_text>**Metrics calculation for after choosing final probability cut off.**<jupyter_code>#Let's check the overall accuracy.
metrics.accuracy_score(y_train_pred_final.Converted, y_train_pred_final.final_predicted)
#Confusion matrix after probability cutoff : 0.35 
confusion2 = metrics.confusion_matrix(y_train_pred_final.Converted, y_train_pred_final.final_predicted )
confusion2
#Confusion Matrix - Calculating all parameters.
TP = confusion2[1,1] # true positive 
TN = confusion2[0,0] # true negatives
FP = confusion2[0,1] # false positives
FN = confusion2[1,0] # false negatives
#Let's see the sensitivity of our logistic regression model
TP / float(TP+FN)
#Let us calculate specificity
TN / float(TN+FP)
#Calculate false postive rate - predicting conversion rate when customer will be a hot lead or not
print(FP/ float(TN+FP))
#Positive predictive value 
print (TP / float(TP+FP))
#Negative predictive value
print (TN / float(TN+ FN))<jupyter_output>0.8714666666666666<jupyter_text>**ACCURACY - SPECIFICITY - SENSITIVITY** - **ACCURACY - 81.19%**- **SENSITIVITY - 80.45%**- **SPECIFICITY - 81.7%**  Model Evaluation - ON TEST SET<jupyter_code>#Perform all the scaling on the available features.
X_test[['TotalVisits','Time Spent','Page Views Per Visit']] = scaler.transform(X_test[['TotalVisits','Time Spent','Page Views Per Visit']])
#Selecting all the features, which are selected as per final model
#col has all the features available in the final model.

X_test = X_test[col]
X_test.head()<jupyter_output><empty_output><jupyter_text>Building the model - ON TEST SET<jupyter_code>#Adding a constant for X_Test.
X_test_sm = sm.add_constant(X_test)
#y_test_pred - X_test_sm has all the features after model building
#res = > Result of model building

y_test_pred = res.predict(X_test_sm)
#Displaying all the predicted values
y_test_pred[:10]<jupyter_output><empty_output><jupyter_text>**Creating a dataframe with the predicted values**<jupyter_code>#Converting y_pred to a dataframe which is an array
y_test_pred_1 = pd.DataFrame(y_test_pred)
y_test_pred_1.head()
#Converting y_test to dataframe
y_test_df = pd.DataFrame(y_test)
#Putting CustID to index
y_test_df['CustID'] = y_test_df.index
#Removing index for both dataframes to append them side by side 
y_test_pred_1.reset_index(drop=True, inplace=True)
y_test_df.reset_index(drop=True, inplace=True)
#Appending y_test_df and y_pred_1
y_pred_final = pd.concat([y_test_df, y_test_pred_1],axis=1)
y_pred_final.head()
#Renaming the column 
y_pred_final= y_pred_final.rename(columns={ 0 : 'Conversion_Prob'})
#Predicting the test model on cut off probabilty
y_pred_final['final_predicted'] = y_pred_final.Conversion_Prob.map(lambda x: 1 if x > 0.35 else 0)
#Displaying the predicted value dataframe.
y_pred_final.head()<jupyter_output><empty_output><jupyter_text>**Calculation of Model Evaluation parameters**<jupyter_code>#Let's check the overall accuracy.
metrics.accuracy_score(y_pred_final.Converted, y_pred_final.final_predicted)
confusionTest= metrics.confusion_matrix(y_pred_final.Converted, y_pred_final.final_predicted)
confusionTest
TP = confusionTest[1,1] # true positive 
TN = confusionTest[0,0] # true negatives
FP = confusionTest[0,1] # false positives
FN = confusionTest[1,0] # false negatives
#Let's see the sensitivity of our logistic regression model
sensitivity = round(TP / float(TP+FN),2) *100 
print(sensitivity)
#Let's calculate specificity
TN / float(TN+FP)<jupyter_output><empty_output><jupyter_text>**ACCURACY - SPECIFICITY - SENSITIVITY** - **ACCURACY - 80.08%**- **SENSITIVITY - 80.0%**- **SPECIFICITY - 80.3%**  OBSERVATION : **We got 0.5% to 1% difference on train and test data's performance metrics.This implies that our final model didn't overfit training data and is performing well.** **From our model, We can conclude following points :**    **1. The customer/leads who fills the form are the potential leads.****2. We must majorly focus on working professionals.****3. We must majorly focus on leads whose last activity is SMS sent or Email opened.** **4. It’s always good to focus on customers, who have spent significant time on our website.****5. It’s better to focus least on customers to whom the sent mail is bounced back.****6. If the lead source is referral, he/she may not be the potential lead.****7. If the lead didn’t fill specialization, he/she may not know what to study and are not right people to target. So, it’s better to focus less on such cases.**  LEAD SCORE - ASSIGNMENT TO ALL CUSTOMERS<jupyter_code>#Preparing the dataframe - to scale the variables and prepare the complete data.
X[['TotalVisits','Time Spent','Page Views Per Visit']] = scaler.transform(X[['TotalVisits','Time Spent','Page Views Per Visit']])
#Selecting the features in our final model on the scaled dataframe.
X = X[col]
X.head()
#Adding a constant to the dataframe
X_sm = sm.add_constant(X)
#Predicting the y-values on X dataframe (Leads.csv)
y_ans_pred = res.predict(X_sm)
#Printing the probabilities
y_ans_pred[:10]
#Converting y_pred to a dataframe which is an array
y_ans_pred_df = pd.DataFrame(y_ans_pred)
#Converting y to dataframe
y_df = pd.DataFrame(y)
#Putting Lead_Number to index
y_df['Lead_Number'] = lead['Lead Number']
#Removing index for both dataframes to append them side by side 
y_ans_pred_df.reset_index(drop=True, inplace=True)
y_df.reset_index(drop=True, inplace=True)
#Appending y_test_df and y_df
y_ans_final = pd.concat([y_df, y_ans_pred_df],axis=1)
#Renaming the column 
y_ans_final= y_ans_final.rename(columns={ 0 : 'Conversion_Prob'})
#Displaying the Answer_Dataframe
y_ans_final.head()<jupyter_output><empty_output><jupyter_text>**Finding the converted value using the probability cut-off = 0.35**<jupyter_code>#Finding the Predicted values using the probability cut off
y_ans_final['final_predicted'] = y_ans_final.Conversion_Prob.map(lambda x: 1 if x > 0.35 else 0)
#Appending the score to the dataframe.
y_ans_final['Score'] = round(y_ans_final['Conversion_Prob'] * 100, 2)
#Displaying the predicted conversion on the final values
y_ans_final.head()<jupyter_output><empty_output><jupyter_text>**Metric Calculation - Complete Dataframe**<jupyter_code># Let's check the overall accuracy.
metrics.accuracy_score(y_ans_final.Converted, y_ans_final.final_predicted)
#Confusion Matrix
confusionAns= metrics.confusion_matrix(y_ans_final.Converted, y_ans_final.final_predicted)
confusionAns
#Confusion Matrix - Parameters
TP = confusionAns[1,1] # true positive 
TN = confusionAns[0,0] # true negatives
FP = confusionAns[0,1] # false positives
FN = confusionAns[1,0] # false negatives
# Let's see the sensitivity of our logistic regression model
sensitivity = round(TP / float(TP+FN),2) *100 
print(sensitivity)
# Let us calculate specificity
TN / float(TN+FP)<jupyter_output><empty_output><jupyter_text>Sorting The Dataframe - To Find The Top Leads<jupyter_code>#Sorting values to find the top leads. 
ResultDf = y_ans_final.sort_values(by='Score', ascending=False)
#Displaying the top leads
ResultDf.head()
FilteredResult = ResultDf.loc[ResultDf['Score'] >= 35]
print(ResultDf.shape)
print(FilteredResult.shape)
FilteredResult.head(10)
#Checking the Converted Rate
Converted_Rate_AfterModelling= (sum(FilteredResult['Converted'])/len(FilteredResult['Converted'].index))*100
Converted_Rate_AfterModelling<jupyter_output><empty_output><jupyter_text>**Observation - We provide a list of only 4000 who have the chances of getting converted into a potential lead so it is easier for the company to limit their target. When we check for conversion rate, it is found to be around 73%.**  Sorting The Dataframe - To Find The Hot Leads With Higher Conversion Rate.<jupyter_code>HotLeads = ResultDf.loc[ResultDf['Score'] >= 95]
HotLeads.shape
#Checking the Converted Rate
Converted_Rate_ForHotLeads= (sum(HotLeads['Converted'])/len(HotLeads['Converted'].index))*100
Converted_Rate_ForHotLeads<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup