<jupyter_start><jupyter_text>Capstone Project: Create a Customer Segmentation Report for Arvato Financial ServicesIn this project, you will analyze demographics data for customers of a mail-order sales company in Germany, comparing it against demographics information for the general population. You'll use unsupervised learning techniques to perform customer segmentation, identifying the parts of the population that best describe the core customer base of the company. Then, you'll apply what you've learned on a third dataset with demographics information for targets of a marketing campaign for the company, and use a model to predict which individuals are most likely to convert into becoming customers for the company. The data that you will use has been provided by our partners at Bertelsmann Arvato Analytics, and represents a real-life data science task.If you completed the first term of this program, you will be familiar with the first part of this project, from the unsupervised learning project. The versions of those two datasets used in this project will include many more features and has not been pre-cleaned. You are also free to choose whatever approach you'd like to analyzing the data rather than follow pre-determined steps. In your work on this project, make sure that you carefully document your steps and decisions, since your main deliverable for this project will be a blog post reporting your findings.  Importing Libraries<jupyter_code># import libraries here; add more as necessary
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import pickle

#imports to help me plot my venn diagrams
import matplotlib_venn as venn2
from matplotlib_venn import venn2
from pylab import rcParams

# import the util.py file where I define my functions
from utils import *

# sklearn
from sklearn.preprocessing import StandardScaler, RobustScaler, MinMaxScaler, OneHotEncoder
from sklearn.impute import SimpleImputer
from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.metrics import confusion_matrix,precision_recall_fscore_support
from sklearn.utils.multiclass import unique_labels
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split, StratifiedKFold, cross_val_score, RandomizedSearchCV, GridSearchCV
from sklearn.metrics import roc_auc_score
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier
from sklearn.ensemble import GradientBoostingClassifier as gbm
from sklearn.linear_model import LogisticRegression
from sklearn.neural_network import MLPClassifier
from sklearn.model_selection import StratifiedKFold, learning_curve

import xgboost as xgb
import lightgbm as lgb

import skopt
from skopt import BayesSearchCV
class BayesSearchCV(BayesSearchCV):
    def _run_search(self, x): raise BaseException('Use newer skopt')

# magic word for producing visualizations in notebook
%matplotlib inline<jupyter_output><empty_output><jupyter_text>Part 0: Get to Know the DataThere are four data files associated with this project:- `Udacity_AZDIAS_052018.csv`: Demographics data for the general population of Germany; 891 211 persons (rows) x 366 features (columns).- `Udacity_CUSTOMERS_052018.csv`: Demographics data for customers of a mail-order company; 191 652 persons (rows) x 369 features (columns).- `Udacity_MAILOUT_052018_TRAIN.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 982 persons (rows) x 367 (columns).- `Udacity_MAILOUT_052018_TEST.csv`: Demographics data for individuals who were targets of a marketing campaign; 42 833 persons (rows) x 366 (columns).Each row of the demographics files represents a single person, but also includes information outside of individuals, including information about their household, building, and neighborhood. Use the information from the first two files to figure out how customers ("CUSTOMERS") are similar to or differ from the general population at large ("AZDIAS"), then use your analysis to make predictions on the other two files ("MAILOUT"), predicting which recipients are most likely to become a customer for the mail-order company.The "CUSTOMERS" file contains three extra columns ('CUSTOMER_GROUP', 'ONLINE_PURCHASE', and 'PRODUCT_GROUP'), which provide broad information about the customers depicted in the file. The original "MAILOUT" file included one additional column, "RESPONSE", which indicated whether or not each recipient became a customer of the company. For the "TRAIN" subset, this column has been retained, but in the "TEST" subset it has been removed; it is against that withheld column that your final predictions will be assessed in the Kaggle competition.Otherwise, all of the remaining columns are the same between the three data files. For more information about the columns depicted in the files, you can refer to two Excel spreadsheets provided in the workspace. [One of them](./DIAS Information Levels - Attributes 2017.xlsx) is a top-level list of attributes and descriptions, organized by informational category. [The other](./DIAS Attributes - Values 2017.xlsx) is a detailed mapping of data values for each feature in alphabetical order.In the below cell, we've provided some initial code to load in the first two datasets. Note for all of the `.csv` data files in this project that they're semicolon (`;`) delimited, so an additional argument in the [`read_csv()`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_csv.html) call has been included to read in the data properly. Also, considering the size of the datasets, it may take some time for them to load completely.You'll notice when the data is loaded in that a warning message will immediately pop up. Before you really start digging into the modeling and analysis, you're going to need to perform some cleaning. Take some time to browse the structure of the data and look over the informational spreadsheets to understand the data values. Make some decisions on which features to keep, which features to drop, and if any revisions need to be made on data formats. It'll be a good idea to create a function with pre-processing steps, since you'll need to clean all of the datasets before you work with them.<jupyter_code># load in the data
'''
There are 2 warnings when we read in the datasets:
DtypeWarning: Columns (19,20) have mixed types. Specify dtype option on import or set low_memory=False.
interactivity=interactivity, compiler=compiler, result=result)

This warning happens when pandas attempts to guess datatypes on particular columns, I will address this on 
the pre-processing steps
'''
azdias = pd.read_csv(r"C:\Users\sousa\Desktop\github\Arvato\data\azdias.csv")
customers = pd.read_csv(r"C:\Users\sousa\Desktop\github\Arvato\data\customers.csv")
attributes = pd.read_csv(r"C:\Users\sousa\Desktop\github\Arvato\data\features.csv")
dias_xls = pd.read_excel(r"C:\Users\sousa\Desktop\github\Arvato\data\DIAS Attributes - Values 2017.xlsx", header = 1)<jupyter_output><empty_output><jupyter_text>I want to use the dias_xls file to help me find the values that correspond to missing or unknow so I will perform a few fixes.Namely, havind row 1 be the header for the dataframe and removing the extra unnammed column.<jupyter_code>dias_xls.drop(columns=['Unnamed: 0'], inplace=True)
dias_xls['Attribute'] = dias_xls['Attribute'].ffill()
dias_xls.head()
# I will now check what is the problem with the columns 19 and 20
# getting the name of these columns
print(azdias.iloc[:,19:21].columns)
print(customers.iloc[:,19:21].columns)<jupyter_output>Index(['CAMEO_DEUG_2015', 'CAMEO_INTL_2015'], dtype='object')
Index(['CAMEO_DEUG_2015', 'CAMEO_INTL_2015'], dtype='object')<jupyter_text>It seems like the mixed type issue comes from that  X that appears in these columns.There are ints, floats and strings all in the mix, I also want to do a quick fix on CAMEO_DEU_2015<jupyter_code>attributes.head()
azdias = special_feature_handler(azdias)
customers = special_feature_handler(customers)<jupyter_output><empty_output><jupyter_text>Checking if values were fixed Change this cell to code if you want to perform the checksazdias.CAMEO_DEUG_2015.unique()customers.CAMEO_INTL_2015.unique() Considering the appearance of these mixed type data entries I created a function to check the dtype of the different attributesThis might be useful in case some attributes have too many category values, which might fragment the data clustering too much.<jupyter_code>#doing a quick check of categorical features and see if some are too granular to be maintained
cat_check = categorical_checker(azdias, attributes)<jupyter_output>AGER_TYP                      5
ANREDE_KZ                     2
CAMEO_DEU_2015               45
CAMEO_DEUG_2015               9
CJT_GESAMTTYP                 6
D19_BANKEN_DATUM             10
D19_BANKEN_OFFLINE_DATUM     10
D19_BANKEN_ONLINE_DATUM      10
D19_GESAMT_DATUM             10
D19_GESAMT_OFFLINE_DATUM     10
D19_GESAMT_ONLINE_DATUM      10
D19_KONSUMTYP                 7
D19_TELKO_DATUM              10
D19_TELKO_OFFLINE_DATUM      10
D19_TELKO_ONLINE_DATUM       10
D19_VERSAND_DATUM            10
D19_VERSAND_OFFLINE_DATUM    10
D19_VERSAND_ONLINE_DATUM     10
D19_VERSI_DATUM              10
D19_VERSI_OFFLINE_DATUM      10
D19_VERSI_ONLINE_DATUM       10
FINANZTYP                     6
GEBAEUDETYP                   7
GFK_URLAUBERTYP              12
GREEN_AVANTGARDE              2
KBA05_BAUMAX                  6
KK_KUNDENTYP                  6
LP_FAMILIE_FEIN              12
LP_FAMILIE_GROB               6
LP_STATUS_FEIN               10
LP_STATUS_GROB                5
NATIONAL[...]<jupyter_text>Based on the categorical info it might be a good idea do drop CAMEO_DEU_2015 column, it is far too fragmented with 45 different category values, this is an idea to revisit after testing the models There is an extra column called Unnamed that seems like an index duplication, I will drop it We also have 3 columns that are different between azdias and customers:'CUSTOMER_GROUP', 'ONLINE_PURCHASE', and 'PRODUCT_GROUP'I will drop those to harmonize the 2 datasets<jupyter_code>customers = customers.drop(['CUSTOMER_GROUP', 'ONLINE_PURCHASE', 'PRODUCT_GROUP'], inplace=False, axis=1)<jupyter_output><empty_output><jupyter_text>I will now check overal shapes of the datasets  Azdias Shape<jupyter_code># checking how the azdias dataframe looks like
print('Printing dataframe shape')
print(azdias.shape)
print('________________________________________________________')

azdias.head()<jupyter_output>Printing dataframe shape
(891221, 366)
________________________________________________________<jupyter_text>Customers Shape<jupyter_code># checking how the customer dataframe looks like
print('Printing dataframe shape')
print(customers.shape)
print('________________________________________________________')

customers.head()<jupyter_output>Printing dataframe shape
(191652, 366)
________________________________________________________<jupyter_text>Attributes shape<jupyter_code># Check the summary csv file
print(attributes.shape)
attributes.head()<jupyter_output>(332, 5)<jupyter_text>On the dataframe shapes: For now it is noted that the 2 initial working dataframes are harmonized in terms of  number of columns: azdias: (891221, 366) customers: (191652, 366) attributes: (332, 5)<jupyter_code>#saving the unique attribute names to lists
attributes_list = attributes.attribute.unique().tolist()
azdias_list = list(azdias.columns)
customers_list = list(customers.columns)
#establishing uniqueness of the attributes accross the datasets in work
common_to_all = (set(attributes_list) & set(azdias_list) & set(customers_list))
unique_to_azdias = (set(azdias_list) - set(attributes_list) - set(customers_list))
unique_to_customers = (set(customers_list) - set(attributes_list) - set(azdias_list))
unique_to_attributes = (set(attributes_list) - set(customers_list) - set(azdias_list))
unique_to_attributes_vs_azdias = (set(attributes_list) - set(azdias_list))
unique_to_azdias_vs_attributes = (set(attributes_list) - set(azdias_list))
common_azdias_attributes = (set(azdias_list) & set(attributes_list))

print("No of items common to all 3 daframes: " + str(len(common_to_all)))
print("No of items exclusive to azdias: " + str(len(unique_to_azdias)))
print("No of items exclusive to customers: " + str(len(unique_to_customers)))
print("No of items exclusive to attributes: " + str(len(unique_to_attributes)))
print("No of items overlapping between azdias and attributes: " + str(len(common_azdias_attributes)))
print("No of items exclusive to attributes vs azdias: " + str(len(unique_to_attributes_vs_azdias)))
print("No of items exclusive to azdias vs attributes: " + str(len(unique_to_azdias_vs_attributes)))
rcParams['figure.figsize'] = 8, 8

ax = plt.axes()
ax.set_facecolor('lightgrey')
v = venn2([len(azdias_list), len(attributes_list), len(common_azdias_attributes)], 
      set_labels=('Azdias', 'Attributes'),
         set_colors = ['cyan', 'grey']);

plt.title("Attribute presence on Azdias vs DIAS Attributes ")
plt.show()<jupyter_output><empty_output><jupyter_text>From this little exploration we got quite a little bit of information:- There are 3 extra features in the customers dataset, it corresponds to the columns 'CUSTOMER_GROUP', 'ONLINE_PURCHASE', and 'PRODUCT_GROUP'- All the datasets share 327 features between them- The attributes file has 5 columns corresponding to feature information that does not exist in the other datasets  PreprocessingNow that I have a birds-eye view of the data I will proceed with cleaning and handling missing calues, re-encode features (since the first portion of this project will involve unsupervised learning), perform some feature enginnering and scaling.Assessing missing data and replacing it with nan.I will also decide on which strategy to use before scaling features:-remove nulls -> scalling -> put nulls back or-remove nulls -> scalling -> impute<jupyter_code>azdias_pre_cleanup = azdias.copy()
customers_pre_cleanup = customers.copy()
unknowns_to_NANs(azdias, dias_xls)
unknowns_to_NANs(customers, dias_xls)<jupyter_output><empty_output><jupyter_text>Since the next step involves dropping columns missing data over a threshold it is important to check if there is a column match between azdias and customers before and after the cleanup processThere is a chance that some columns are missing too much data in one dataframe and being dropped while they are abundant in the other, causing a discrepancy in the shape between the 2 dataframesIt is always hard to define a threshold on how much missing data is too much, my first approach will consider over 30% too muchBased on model performance this is an idea to revisit and adjust<jupyter_code>balance_checker(azdias, customers)<jupyter_output>Feature balance between dfs?:  True<jupyter_text>Prior to cleanup customers and azdias match<jupyter_code>percent_missing_azdias_df = percentage_of_missing(azdias)
percent_missing_azdias_pc_df = percentage_of_missing(azdias_pre_cleanup)

percent_missing_customers_df = percentage_of_missing(customers)
percent_missing_customers_pc_df = percentage_of_missing(customers_pre_cleanup)
print('Identified missing data in Azdias: ')
print('Pre-cleanup: ' + str(azdias_pre_cleanup.isnull().sum().sum()) + 
      ' Post_cleanup: ' + str(azdias.isnull().sum().sum()))

print('Identified missing data in Customers: ')
print('Pre-cleanup: ' + str(customers_pre_cleanup.isnull().sum().sum()) +
      ' Post_cleanup: ' + str(customers.isnull().sum().sum()))
print('Azdias columns not missing values(percentage):')
print('Pre-cleanup: ', (percent_missing_azdias_df['percent_missing'] == 0.0).sum())
print('Post-cleanup: ', (percent_missing_azdias_pc_df['percent_missing'] == 0.0).sum())

print('Customers columns not missing values(percentage):')
print('Pre-cleanup: ', (percent_missing_customers_df['percent_missing'] == 0.0).sum())
print('Post-cleanup: ', (percent_missing_customers_pc_df['percent_missing'] == 0.0).sum())<jupyter_output>Azdias columns not missing values(percentage):
Pre-cleanup:  87
Post-cleanup:  93
Customers columns not missing values(percentage):
Pre-cleanup:  87
Post-cleanup:  93<jupyter_text>Deciding on what data to maintain based on the percentage missing<jupyter_code># missing more or less than 30% of the data
azdias_missing_over_30 = split_on_percentage(percent_missing_azdias_df, 30, '>')
azdias_missing_less_30 = split_on_percentage(percent_missing_azdias_df, 30, '<=')

customers_missing_over_30 = split_on_percentage(percent_missing_customers_df, 30, '>')
customers_missing_less_30 = split_on_percentage(percent_missing_customers_df, 30, '<=')
#plotting select features and their missing data percentages
figure, axes = plt.subplots(4, 1, figsize = (15,15), squeeze = False)

azdias_missing_over_30.sort_values(by = 'percent_missing', ascending = False).plot(kind = 'bar', x = 'column_name', y = 'percent_missing',
                                                                                ax = axes[0][0], color = 'cyan', title = 'Azdias percentage of missing values over 30%' )

#due to the sheer amount of data points to be plotted this does not make an appealing vis so I will restrict
#the number of plotted points to 40
azdias_missing_less_30.sort_values(by = 'percent_missing', ascending = False)[:40].plot(kind = 'bar', x = 'column_name', y = 'percent_missing',
                                                                                ax = axes[1][0], color = 'cyan', title = 'Azdias percentage of missing values less 30%' )

customers_missing_over_30.sort_values(by = 'percent_missing', ascending = False).plot(kind = 'bar', x = 'column_name', y = 'percent_missing',
                                                                                ax = axes[2][0], color = 'grey', title = 'Customers percentage of missing values over 30%' )

#due to the sheer amount of data points to be plotted this does not make an appealing vis so I will restrict
#the number of plotted points to 40
customers_missing_less_30.sort_values(by = 'percent_missing', ascending = False)[:40].plot(kind = 'bar', x = 'column_name', y = 'percent_missing',
                                                                                ax = axes[3][0],color = 'grey', title = 'Customers percentage of missing values less 30%' )

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>The vast majority of the columns with missing values have a percent of missing under 30% Based on this information I will remove columns with more than 30% missing values<jupyter_code>#extracting column names with more than 30% values missing so we can drop them from azdias df
azdias_col_delete = columns_to_delete(azdias_missing_over_30)

#extracting column names with more than 30% values missing so we can drop them from customers df
customers_col_delete = columns_to_delete(customers_missing_over_30)
#dropping the columns identified in the previous lists

azdias = azdias.drop(azdias_col_delete, axis = 1)
customers = customers.drop(customers_col_delete, axis = 1)
#since I just dropped several columns I will do another balance check
balance_checker(azdias, customers)<jupyter_output>Feature balance between dfs?:  False
Your first argument df differs from the second on the following columns: 
{'REGIOTYP', 'KKK'}
Your second argument df differs from the first on the following columns: 
set()<jupyter_text>Now that we dropped columns missing more than 30% of their data let's check if we should also drop rows based on a particular threshold<jupyter_code>#plotting distribution of null values
row_hist(azdias, customers, 30)<jupyter_output><empty_output><jupyter_text>Based on this visualization we deduct 2 things - most of the rows have less than 100 missing values  - both customer and azdias have probably overlapping rows in which they are missing the same info  deleting rows based on the information acquired in the previous histogram azdias = row_dropper(azdias, 100)customers = row_dropper(customers, 100) plotting null values distribution after cleanuprow_hist(azdias, customers, 30) balance_checker(azdias, customers)  It seems like dropping the rows makes the clustering perform worse so for now I will skip it  Feature Encoding Like I previously checked using the categorical_checker there are many features in need of re-encoding for the unsupervised learning portion - numerical features will be kept as is- ordinal features will be kept as is- categorical features and mixed type features will have to be re-encoded (label or dummies)<jupyter_code>#checking for mixed type features
attributes[attributes.type == 'mixed']
#retrieve a list of categorical features for future encoding
cats = attributes[attributes.type == 'categorical']
list(cats['attribute'])<jupyter_output><empty_output><jupyter_text>At this point I already dealt with the CAMEO_INTL_2015 column by converting XX to nanPRAEGENDE_JUGENDJAHRE has 3 dimentions: generation decade, if people are mainstream or avant-garde and if they are from east or west, I will create new features out of this particular columnLP_LEBENSPHASE_GROB seems to encode the same information as the CAMEO column and it is divided between gross(grob) and fine (fein)<jupyter_code>balance_checker(azdias, customers)<jupyter_output>Feature balance between dfs?:  False
Your first argument df differs from the second on the following columns: 
{'REGIOTYP', 'KKK'}
Your second argument df differs from the first on the following columns: 
set()<jupyter_text>Feature engineering Based on the previous exploration there are a few features that are good candidates for novel feature creation<jupyter_code>azdias_eng = azdias.copy()
customers_eng = customers.copy()
azdias_eng = feat_eng(azdias_eng)
customers_eng = feat_eng(customers_eng)<jupyter_output>Creating PRAEGENDE_JUGENDJAHRE_DECADE feature
Creating PRAEGENDE_JUGENDJAHRE_MOVEMENT feature
Creating WOHNLAGE_QUALITY feature
Creating WOHNLAGE_AREA feature
Creating Wealth and Family feature
Creating LP_LEBENSPHASE_FEIN_life_stage and LP_LEBENSPHASE_FEIN_fine_scale feature
Creating PRAEGENDE_JUGENDJAHRE_DECADE feature
Creating PRAEGENDE_JUGENDJAHRE_MOVEMENT feature
Creating WOHNLAGE_QUALITY feature
Creating WOHNLAGE_AREA feature
Creating Wealth and Family feature
Creating LP_LEBENSPHASE_FEIN_life_stage and LP_LEBENSPHASE_FEIN_fine_scale feature<jupyter_text>balance_checker(azdias, customers)  Now that I am done with creating new features and dealing with the most obvious columns I might need to encode the remaining categorical features Considering this post: https://stats.stackexchange.com/questions/224051/one-hot-vs-dummy-encoding-in-scikit-learn there are advantages and drawbacks with chosing one-hot-encoding vs dummy encoding. There are also concerns regarding using dummies all together https://towardsdatascience.com/one-hot-encoding-is-making-your-tree-based-ensembles-worse-heres-why-d64b282b5769 so I will keep this in mind while moving forward  Feature scaling Before moving on to dimentionality reduction I need to apply feature scaling, this way principal component vectors won't be affected by the variation that naturally occurs in the data Before applying the scaler there should be no missing values in the data azdias_eng.to_csv(r"C:\Users\sousa\Desktop\github\Arvato\data\azdias_eng.csv")customers_eng.to_csv(r"C:\Users\sousa\Desktop\github\Arvato\data\customers_eng.csv")<jupyter_code>#dataframes using StandardScaler
azdias_SS = feature_scaling(azdias_eng, 'StandardScaler')
customers_SS = feature_scaling(customers_eng, 'StandardScaler')
#dataframes using MinMaxScaler
azdias_MMS = feature_scaling(azdias_eng, 'MinMaxScaler')
customers_MMS = feature_scaling(customers_eng, 'MinMaxScaler')<jupyter_output><empty_output><jupyter_text>Dimensionality Reduction Finally I will use PCA (linear technique) to select only the components that seem to be more impactfull<jupyter_code>n_components_azdias = len(azdias_SS.columns.values)
n_components_customers = len(customers_SS.columns.values)

azdias_SS_pca = pca_model(azdias_SS, n_components_azdias)
customers_SS_pca = pca_model(customers_SS, n_components_customers)

azdias_MMS_pca = pca_model(azdias_MMS, n_components_azdias)
customers_MMS_pca = pca_model(customers_MMS, n_components_customers)<jupyter_output><empty_output><jupyter_text>Change to code if intending to save file progress for other notebooksazdias_SS.to_pickle(r"C:\Users\sousa\Desktop\github\Arvato\data\azdias_SS.pickle")azdias_MMS.to_pickle(r"C:\Users\sousa\Desktop\github\Arvato\data\azdias_MMS.pickle")<jupyter_code>scree_plots(azdias_SS_pca, azdias_MMS_pca, ' azdias')
scree_plots(customers_SS_pca, customers_MMS_pca, ' customers')<jupyter_output><empty_output><jupyter_text>Each principal component is a directional vector pointing to the highest variance. The greater the distance from 0 the more the vector points to a feature.Lets check some interesting features in up to the third dimension.<jupyter_code>display_interesting_features(azdias_SS, azdias_SS_pca, 0)
display_interesting_features(azdias_SS, azdias_SS_pca, 1)
display_interesting_features(azdias_SS, azdias_SS_pca, 3)<jupyter_output>Lowest: 
	PRAEGENDE_JUGENDJAHRE_DECADE -0.196
	FINANZ_UNAUFFAELLIGER -0.191
	FINANZ_SPARER        -0.191
Highest: 
	LP_LEBENSPHASE_FEIN_life_stage 0.147
	ALTERSKATEGORIE_GROB 0.160
	FINANZ_VORSORGER     0.168<jupyter_text>The highest the weight of an attribute the more relevant it is considered to be, lets take a look on the most important features for a few dimensions considering that positive weights might relate to a positive relationship and negative weights a negative one.:- dimension 1(0) using standard scaler:    These are some of features related to positive weights:        - MOBI_RASTER: refers to the individual's mobility    - KBA13_ANTG1: lower share of car owners    - PLZ8_ANTG1 : lower number of 1-2 family houses        And these are some of the feaures related to negative weights:        - KBA13_ANTG4: refers to posession of higher number of cars    - PLZ-ANTG3: number of 6-10 family houses in the PLZ8    - CAMEO_DEU_2015: detailed classification os cultural and living status     So overall the first dimension refers to the social status and living conditions of the individuals present in the dataset.Interesting to see that even though different features were selected for dimensionality reduction but they overall have the same meaning.  Based on these plots:- using standard scaler with 150 principal components 90% of the original variance can be represented- using minmax scaler with 150 components we represent 90% of the original variance  Moving on I will pick the standard scaler PCA and I will re-fit with with a number of components that explains over 80% of the explained variance  Part 1: Customer Segmentation ReportThe main bulk of your analysis will come in this part of the project. Here, you should use unsupervised learning techniques to describe the relationship between the demographics of the company's existing customers and the general population of Germany. By the end of this part, you should be able to describe parts of the general population that are more likely to be part of the mail-order company's main customer base, and which parts of the general population are less so.  After a lot of data Pre-Processing we are finally getting to the analysis, I will start by attempting KMeans to find relevant clusters  Now that I have reduced the number of components to use, it is important to select the number of clusters to aim at for kmeans<jupyter_code>pca = PCA(150)
azdias_SS_pca = pca.fit_transform(azdias_SS)
customers_SS_PCA = pca.fit_transform(customers_SS)<jupyter_output><empty_output><jupyter_text>The elbow method (https://bl.ocks.org/rpgove/0060ff3b656618e9136b) is a way to validate the optimal number of clusters to use for a particular dataset.It can take some time training the dataset, optimising for the optimal n of clusters means that less resources are used. Based on the elbow plot 9 clusters should be enough to proceed with the kmeans training<jupyter_code># refitting using just 9 clusters
kmeans = KMeans(9)
kmodel = kmeans.fit(azdias_SS_pca)
#and now we can compare the customer data to the general demographics
azdias_clusters = kmodel.predict(pca.transform(azdias_SS))
customers_clusters = kmodel.predict(pca.transform(customers_SS))
#getting clusters for the LNRs for the customers
cluster_map = pd.DataFrame()
cluster_map['LNR'] = azdias_eng.index.values
cluster_map['cluster'] = kmodel.labels_<jupyter_output><empty_output><jupyter_text>Experimenting with visualization of the clusters preparing cluster visualizationfrom collections import Counterazdias_labels = kmeans.labels_customers_labels = kmeans.labels_model_feat = list(azdias_eng.columns)cust_feat = list(customers_eng.columns)model_feat_df = pd.DataFrame()model_feat_df['model_feat'] = model_featFind model features not in customer featuresmodel_feat_notin_cust = [feat for feat in model_feat if feat not in cust_feat]len(model_feat_notin_cust)customers_pca = pca.transform(customers_eng)customers_labels = kmeans.predict(customers_pca)counts_customer = Counter(customers_labels)n_customers = customers_pca.shape[0]customer_freqs = {label: 100*(freq / n_customers) for label, freq in counts_customer.items()}counts_population = Counter(azdias_labels)n_population = azdias_SS_pca.shape[0]population_freqs = {label: 100*(freq / n_population) for label, freq in counts_population.items()}customer_clusters = pd.DataFrame.from_dict(customer_freqs, orient='index', columns=['% of data'])customer_clusters['Cluster'] = customer_clusters.indexcustomer_clusters['DataSet'] = 'Customers Data'population_clusters = pd.DataFrame.from_dict(population_freqs, orient='index', columns=['% of data'])population_clusters['Cluster'] = population_clusters.indexpopulation_clusters['DataSet'] = 'General Population'all_clusters = pd.concat([customer_clusters, population_clusters])all_clusters<jupyter_code>sns.catplot(x='Cluster', y='% of data', hue='DataSet', data=all_clusters, kind='bar')
plt.show()
#transform the customers using pca
customers_pca = pca.transform(customers_SS)

#predict clustering using the kmeans
predict_customers = kmodel.predict(customers_pca)

#cluster and center prediction and info
clust_preds = kmodel.predict(azdias_SS_pca)
centers = kmodel.cluster_centers_

# Compare the proportion of data in each cluster for the customer data to the
# proportion of data in each cluster for the general population.
general_pop = []
customers_pop = []
x = [i+1 for i in range(9)]
for i in range(9):
    general_pop.append((clust_preds == i).sum()/len(clust_preds))
    customers_pop.append((predict_customers == i).sum()/len(predict_customers))


df_general = pd.DataFrame({'cluster' : x, 'general_proportion' : general_pop, 'customers_proportion':customers_pop})

df_general.plot(x='cluster', y = ['general_proportion', 'customers_proportion'], kind='bar', figsize=(9,6))
plt.ylabel('proportion of persons in each cluster')
plt.show()<jupyter_output><empty_output><jupyter_text>Part 2: Supervised Learning ModelNow that you've found which parts of the population are more likely to be customers of the mail-order company, it's time to build a prediction model. Each of the rows in the "MAILOUT" data files represents an individual that was targeted for a mailout campaign. Ideally, we should be able to use the demographic information from each individual to decide whether or not it will be worth it to include that person in the campaign.The "MAILOUT" data has been split into two approximately equal parts, each with almost 43 000 data rows. In this part, you can verify your model with the "TRAIN" partition, which includes a column, "RESPONSE", that states whether or not a person became a customer of the company following the campaign. In the next part, you'll need to create predictions on the "TEST" partition, where the "RESPONSE" column has been withheld.<jupyter_code>mailout_train = pd.read_csv(r"C:\Users\sousa\Desktop\github\Arvato\data\mailout_train.csv")
mailout_test = pd.read_csv(r"C:\Users\sousa\Desktop\github\Arvato\data\mailout_test.csv")
#running all the cleaning and feature transformation functions
#fixing the mixed type columns
mailout_train = special_feature_handler(mailout_train)
mailout_test = special_feature_handler(mailout_test)

#dealing with missing and unknowns
unknowns_to_NANs(mailout_train, dias_xls)
unknowns_to_NANs(mailout_test, dias_xls)

#getting percentages of missing
percent_missing_train = percentage_of_missing(mailout_train)
percent_missing_test = percentage_of_missing(mailout_test)

#getting missing over 30%
train_missing_over_30 = split_on_percentage(percent_missing_train, 30, '>')
test_missing_over_30 = split_on_percentage(percent_missing_test, 30, '>')

#getting columns to delete
train_col_delete = columns_to_delete(train_missing_over_30)
test_col_delete = columns_to_delete(test_missing_over_30)

#dropping cols
mailout_train = mailout_train.drop(train_col_delete, axis = 1)
mailout_test = mailout_test.drop(test_col_delete, axis = 1)

#feature engineering
mailout_train = feat_eng(mailout_train)
mailout_test = feat_eng(mailout_test)<jupyter_output>Creating PRAEGENDE_JUGENDJAHRE_DECADE feature
Creating PRAEGENDE_JUGENDJAHRE_MOVEMENT feature
Creating WOHNLAGE_QUALITY feature
Creating WOHNLAGE_AREA feature
Creating Wealth and Family feature
Creating LP_LEBENSPHASE_FEIN_life_stage and LP_LEBENSPHASE_FEIN_fine_scale feature
Creating PRAEGENDE_JUGENDJAHRE_DECADE feature
Creating PRAEGENDE_JUGENDJAHRE_MOVEMENT feature
Creating WOHNLAGE_QUALITY feature
Creating WOHNLAGE_AREA feature
Creating Wealth and Family feature
Creating LP_LEBENSPHASE_FEIN_life_stage and LP_LEBENSPHASE_FEIN_fine_scale feature<jupyter_text>mailout_train.to_csv('mailout_train_clean.csv')mailout_test.to_csv('mailout_test_clean.csv') since I just dropped several columns I will do another balance check<jupyter_code>balance_checker(azdias_eng, mailout_train)
balance_checker(mailout_train, mailout_test)<jupyter_output>Feature balance between dfs?:  False
Your first argument df differs from the second on the following columns: 
{'RESPONSE'}
Your second argument df differs from the first on the following columns: 
set()<jupyter_text>Before moving on to the model I want to experiment a bit with the identified clusters and how the responses align with the clusters<jupyter_code>#merging mailout and cluster map on the LNR column
clusters_mailout = pd.merge(mailout_train, cluster_map, on = 'LNR')
response_and_cluster = clusters_mailout[['LNR', 'RESPONSE', 'cluster']]
ax = sns.countplot(x="cluster", hue="RESPONSE", data=response_and_cluster)
plot1 = response_and_cluster.groupby(['cluster', 'RESPONSE'])['RESPONSE'].count().unstack('RESPONSE')
plot1[[0, 1]].plot(kind = 'bar', stacked = True)
df_pos_response = response_and_cluster[response_and_cluster['RESPONSE'] == 1]
df_neg_response = response_and_cluster[response_and_cluster['RESPONSE'] == 0]
figure, axes = plt.subplots(2, 1, figsize = (15,15), squeeze = False)

pos = df_pos_response['cluster'].value_counts().plot(kind='bar',figsize=(15,10), color = 'C1', ax = axes[0][0], title = 'Clusters with positive responses')
neg = df_neg_response['cluster'].value_counts().plot(kind='bar',figsize=(15,10), ax = axes[1][0], title = 'Clusters with negative responses')

plt.tight_layout()
plt.show()<jupyter_output><empty_output><jupyter_text>Change to code if intending to save file progress for other notebooksmailout_train.to_pickle(r"C:\Users\sousa\Desktop\github\Arvato\data\mailout_train.pickle")mailout_test.to_pickle(r"C:\Users\sousa\Desktop\github\Arvato\data\mailout_test.pickle")  Getting to the models for response prediction<jupyter_code>#getting the target
target = mailout_train['RESPONSE']
mailout_train_clean = mailout_train.drop(['RESPONSE'], inplace=False, axis=1)
#dropping LNR
mailout_test_clean = mailout_test.copy()
mailout_train_clean.drop(['LNR'], inplace = True, axis = 1)
mailout_test_clean.drop(['LNR'], inplace = True, axis = 1)
balance_checker(mailout_train_clean, mailout_test_clean)
#checking the label distribution

sns.countplot(target).set_title("Label distribution")<jupyter_output><empty_output><jupyter_text>Based on this plot these datasets are quite imbalanced, there are quite a few more responses corresponding to 0 than to 1.Considering this accuracy will not be an appropriate metric, ROC-AUC is a better option (1 being a perfect score, and 0.5 just random chance)Lets try model selection<jupyter_code>SEED = 28
# 5 stratified folds
skf = StratifiedKFold(n_splits=5, random_state=SEED, shuffle=True)
skf.get_n_splits(mailout_train_clean, target)<jupyter_output><empty_output><jupyter_text>The best performing model seem to be XGB so from here on I will continue the analysis with this model.There also seems to be no difference beetween standard scaler and Min Max Scaler. since the second is a bit faster I will stick with it.<jupyter_code>scaler = MinMaxScaler()
scaler.fit(mailout_train_clean.astype('float'))

mailout_train_scaled = scaler.transform(mailout_train_clean)
mailout_test_scaled = scaler.transform(mailout_test_clean)

# map back to dfs
mailout_train_scaled = pd.DataFrame(data=mailout_train_scaled, columns=mailout_train_clean.columns)
mailout_test_scaled = pd.DataFrame(data=mailout_test_scaled, columns=mailout_test_clean.columns)
balance_checker(mailout_train, mailout_test)<jupyter_output><empty_output><jupyter_text>I experimented with multiple models to determine which one was the best performing. The best performer was xgboost so I decided to use the Bayesian Optimization algorithm to fine tune the hyperparameters that produce the best results.<jupyter_code>#submission 1 parameters
bayes_xgb = xgb.XGBClassifier(base_score=0.5, 
                              booster='gbtree', 
                              colsample_bylevel=1,
                              colsample_bytree=0.6872852588903648, 
                              eval_metric='auc', 
                              gamma=1.0,
                              learning_rate=0.014017007043823785, 
                              max_delta_step=0, 
                              max_depth=7,
                              min_child_weight=1, 
                              missing=None, 
                              n_estimators=236, 
                              n_jobs=-1,
                              nthread=None, 
                              objective='binary:logistic', 
                              random_state=0,
                              reg_alpha=1e-09, 
                              reg_lambda=1, 
                              scale_pos_weight=1, 
                              seed=None,
                              silent=1, 
                              subsample=0.5)<jupyter_output><empty_output><jupyter_text>submission 2 parametersbayes_xgb = xgb.XGBClassifier(base_score=0.5,                               booster='gbtree',                               colsample_bylevel=1,                              colsample_bytree=1,                               eval_metric='auc',                               gamma=0,                              learning_rate=0.04788137748021131,                               max_delta_step=0,                               max_depth=1,                              min_child_weight=10,                               missing=None,                               n_estimators=429,                               n_jobs=-1,                              nthread=None,                               objective='binary:logistic',                               random_state=0,                              reg_alpha=0.17651703245342792,                               reg_lambda=1,                               scale_pos_weight=1,                               seed=None,                              silent=1,                               subsample=0.5) submission 3 parametersbayes_xgb = xgb.XGBClassifier(base_score=0.5,                               booster='gbtree',                               colsample_bylevel=1,                              colsample_bytree=0.9889952021544406,                               eval_metric='auc',                               gamma=0.7810813412544743,                              learning_rate=0.044734814719961276,                               max_delta_step=0,                               max_depth=1,                              min_child_weight=10,                               missing=None,                               n_estimators=454,                               n_jobs=-1,                              nthread=None,                               objective='binary:logistic',                               random_state=0,                              reg_alpha=1.0,                               reg_lambda=1,                               scale_pos_weight=1,                               seed=None,                              silent=1,                               subsample=0.5) submission 4 parametersbayes_xgb = xgb.XGBClassifier(base_score=0.5,                               booster='gbtree',                               colsample_bylevel=1,                              colsample_bytree=0.8,                               eval_metric='auc',                               gamma=0,                              learning_rate=0.1,                               max_delta_step=0,                               max_depth=5,                              min_child_weight=1,                               missing=None,                               n_estimators=454,                               n_jobs=2,                              nthread=None,                               objective='binary:logistic',                               random_state=0,                              reg_alpha=1.0,                               reg_lambda=1,                               scale_pos_weight=1,                               seed=None,                              silent=1,                               subsample=0.8) submission 5 parametersbayes_xgb = xgb.XGBClassifier(base_score=0.5,                               booster='gbtree',                               colsample_bylevel=1,                              colsample_bytree=0.6872852588903648,                               eval_metric='auc',                               gamma=1,                              learning_rate=0.01,                               max_delta_step=0,                               max_depth=3,                              min_child_weight=4,                               missing=None,                               n_estimators=454,                               n_jobs=2,                              nthread=None,                               objective='binary:logistic',                               random_state=0,                              reg_alpha=1e-09,                               reg_lambda=1,                               scale_pos_weight=1,                               seed=None,                              silent=1,                               subsample=0.8)                              {'colsample_bylevel': 0.6207294509491699, 'colsample_bytree': 0.9623594386716922, 'gamma': 1.0, 'learning_rate': 0.001, 'max_delta_step': 1, 'max_depth': 6, 'min_child_weight': 1, 'n_estimators': 500, 'reg_alpha': 1e-09, 'reg_lambda': 0.0002004101751465717, 'scale_pos_weight': 56, 'subsample': 0.6092942995200367} submission 6 parametersbayes_xgb = xgb.XGBClassifier(base_score=0.5,                               booster='gbtree',                               colsample_bylevel=0.6207294509491699,                              colsample_bytree=0.9623594386716922,                               eval_metric='auc',                               gamma=1,                              learning_rate=0.001,                               max_delta_step=1,                               max_depth=6,                              min_child_weight=1,                               missing=None,                               n_estimators=500,                               n_jobs=2,                              nthread=None,                               objective='binary:logistic',                               random_state=0,                              reg_alpha=1e-09,                               reg_lambda=0.0002004101751465717,                               scale_pos_weight=56,                               seed=None,                              silent=1,                               subsample=0.6092942995200367) bayes_xgb = xgb.XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,              colsample_bynode=1, colsample_bytree=0.5578105865941786,              eval_metric='auc', gamma=0.07148993933953135,              learning_rate=0.002858444321957567, max_delta_step=0, max_depth=4,              min_child_weight=1, missing=None, n_estimators=500, n_jobs=-1,              nthread=None, objective='binary:logistic', random_state=0,              reg_alpha=0.004635659074237927, reg_lambda=1, scale_pos_weight=41,              seed=None, silent=1, subsample=0.522846932344412, verbosity=1)<jupyter_code>bayes_xgb.fit(mailout_train_scaled, target)
plot_feature_importances(model=bayes_xgb, model_type="XGBClassifier", features=mailout_train_scaled.columns)<jupyter_output><empty_output><jupyter_text>submission 1 lgbmbayes_lgbm = lgb.LGBMClassifier(application='binary',                                 boosting_type='gbdt',                                 class_weight=None,                                colsample_bytree=1.0,                                 importance_type='split',                                learning_rate=0.09531171453332088,                                 max_bin=100,                                 max_depth=2,                                metric='auc',                                 min_child_samples=14,                                 min_child_weight=0.001,                                min_data_in_leaf=265,                                 min_split_gain=0.0,                                 n_estimators=24,                                n_jobs=-1,                                 num_leaves=450,                                 objective=None,                                 random_state=None,                                reg_alpha=1e-09,                                 reg_lambda=1.0,                                scale_pos_weight=69.87642288579819,                                 silent=True,                                 subsample=1.0,                                subsample_for_bin=200000,                                 subsample_freq=0,                                 verbose=0)<jupyter_code>#submission 2 lgbm
bayes_lgbm = lgb.LGBMClassifier(application='binary', 
                                boosting_type='gbdt', 
                                class_weight=None,
                                colsample_bytree=1.0, 
                                importance_type='split',
                                learning_rate=0.0882410095694084, 
                                max_bin=826, 
                                max_depth=2,
                                metric='auc', 
                                min_child_samples=0, 
                                min_child_weight=0.001,
                                min_data_in_leaf=32, 
                                min_split_gain=0.0, 
                                n_estimators=24,
                                n_jobs=-1, 
                                num_leaves=94, 
                                objective=None, 
                                random_state=None,
                                reg_alpha=1.0, 
                                reg_lambda=1e-09,
                                scale_pos_weight=90.0, 
                                silent=True, 
                                subsample=1.0,
                                subsample_for_bin=200000, 
                                subsample_freq=0, 
                                verbose=0)
bayes_lgbm.fit(mailout_train_scaled, target)
plot_feature_importances(model=bayes_lgbm, model_type="LGBMClassifier", features=mailout_train_scaled.columns)
plot_comparison_feature(column = 'D19_SOZIALES', df=mailout_train)<jupyter_output><empty_output><jupyter_text>Part 3: Kaggle CompetitionNow that you've created a model to predict which individuals are most likely to respond to a mailout campaign, it's time to test that model in competition through Kaggle. If you click on the link [here](http://www.kaggle.com/t/21e6d45d4c574c7fa2d868f0e8c83140), you'll be taken to the competition page where, if you have a Kaggle account, you can enter. If you're one of the top performers, you may have the chance to be contacted by a hiring manager from Arvato or Bertelsmann for an interview!Your entry to the competition should be a CSV file with two columns. The first column should be a copy of "LNR", which acts as an ID number for each individual in the "TEST" partition. The second column, "RESPONSE", should be some measure of how likely each individual became a customer – this might not be a straightforward probability. As you should have found in Part 2, there is a large output class imbalance, where most individuals did not respond to the mailout. Thus, predicting individual classes and using accuracy does not seem to be an appropriate performance evaluation method. Instead, the competition will be using AUC to evaluate performance. The exact values of the "RESPONSE" column do not matter as much: only that the higher values try to capture as many of the actual customers as possible, early in the ROC curve sweep.<jupyter_code>#fit and predict xgb
bayes_xgb.fit(mailout_train_scaled, target)
predictions = bayes_xgb.predict_proba(mailout_test_scaled)[:,1]
#fit and predict lgbm
bayes_lgbm.fit(mailout_train_scaled, target)
predictions = bayes_lgbm.predict_proba(mailout_test_scaled)[:,1]
# create submission file
lnr = pd.DataFrame(mailout_test['LNR'].astype('int32'))
predictions = pd.DataFrame(predictions)
predictions = predictions.rename(columns={0: "RESPONSE"})
dfs = [lnr, predictions]
submission = pd.concat(dfs, sort=False, axis=1)
submission.set_index('LNR', inplace = True)
submission.head()
submission.shape
submission.to_csv('kaggle_sub10.csv')<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup