<jupyter_start><jupyter_text>Project The aim of the project was to answer questions related to Stackoverflow's survey. A blog where the questions are discussed can be found here:https://medium.com/@alicja.roelofsen/what-makes-a-developer-f1d8f82ebc19  Gather<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import AdaBoostClassifier
from sklearn.metrics import accuracy_score, fbeta_score
import seaborn as sns
%matplotlib inline
df_original = pd.read_csv('./survey_results_public.csv')<jupyter_output><empty_output><jupyter_text>Pre-assessmentPlease note that large part of the assesment has been done on the downloaded survay schema, in excel, where it was easier to scroll through the large number od columns.<jupyter_code>df_original.head()<jupyter_output><empty_output><jupyter_text>Business understandingThe aim of the following analysis is to answer the following questions:1. Is there correlation between the education of responders (developers, analysts) and the level of education of their parents?2. Do people who choose a technical career path (developers, anlysts) have certain personal traits or preferences in common?3. Based on person's characteristics, can you predict if they enjoyed the survey?  Assessment of columns needed for Question 1<jupyter_code>df_original[['HighestEducationParents', 'FormalEducation']].head()
df_original[['HighestEducationParents', 'FormalEducation']].info()
df_original['HighestEducationParents'].value_counts()
df_original['FormalEducation'].value_counts()<jupyter_output><empty_output><jupyter_text>ConclusionColumn HighestEducationParents has many null values, will need to account for that.Cathegories in the two columns differ, and are too granular, will need to clean this.  Assessment of columns needed for Question 2<jupyter_code>df_original[['ProblemSolving', 'BuildingThings', 'LearningNewTech', 
                                'UnderstandComputers', 'SeriousWork', 'KinshipDevelopers', 
                                'ChallengeMyself', 'CompetePeers', 'ChangeWorld']].head()
df_original[['ProblemSolving', 'BuildingThings', 'LearningNewTech', 
                                'UnderstandComputers', 'SeriousWork', 'KinshipDevelopers', 
                                'ChallengeMyself', 'CompetePeers', 'ChangeWorld']].info()
df_original['ProblemSolving'].value_counts()<jupyter_output><empty_output><jupyter_text>ConclusionAll the columns have many null values, will need to account for that. No other cleaning necessary.  Assessment of columns needed for Question 3<jupyter_code>df_original[['QuestionsInteresting', 'Professional', 'ProgramHobby', 'Country', 'EmploymentStatus', 'FormalEducation', 
               'CareerSatisfaction', 'StackOverflowSatisfaction', 'StackOverflowCommunity', 
               'StackOverflowHelpful', 'StackOverflowBetter', 'Gender', 'Race']].head()
df_original[['QuestionsInteresting', 'Professional', 'ProgramHobby', 'Country', 'EmploymentStatus', 'FormalEducation', 
               'CareerSatisfaction', 'StackOverflowSatisfaction', 'StackOverflowCommunity', 
               'StackOverflowHelpful', 'StackOverflowBetter', 'Gender', 'Race']].info()
df_original['QuestionsInteresting'].value_counts()
df_original['StackOverflowSatisfaction'].value_counts()<jupyter_output><empty_output><jupyter_text>ConclusionThe QuestionsInteresting column we will try to predict has many nulls, those rows will have to be removed. Other columns, such as StackOverflowSatisfaction can have values imputed - although bias might be introduced, in the model of interest (checking what features might have impact on the response), and for the exploratory purposes, we can accept this trade-off.  Question 1Is there correlation between the education of responders (developers, analysts) and the level of education of their parents?  Clean data to produce analysis<jupyter_code># choose the columns of interest
df1 = df_original.copy()
df1 = df1[['HighestEducationParents', 'FormalEducation']] 

# clean data - keep only valid answers
df1 = df1[(df1['HighestEducationParents'] != "I prefer not to answer") & (df1['HighestEducationParents'] != "I don't know/not sure") & (df1['FormalEducation'] != "I prefer not to answer")]
# drop NaNs - we only want to see valid answers
df1 = df1.dropna(subset=['HighestEducationParents', 'FormalEducation'], how='any')

df1.head(5)
# reduce the cathegorical columns
# clean (there are differences between choices, e.g. "A doctoral degree" vs "Doctoral degree"), so unify 
profession_map = {'A bachelor\'s degree': 'University (Bachelor, other)', 
                  'Bachelor\'s degree': 'University (Bachelor, other)', 
                  'A doctoral degree': 'University (Master, Doctoral)', 
                  'Doctoral degree': 'University (Master, Doctoral)', 
                  'A master\'s degree': 'University (Master, Doctoral)',
                  'Master\'s degree': 'University (Master, Doctoral)',
                  'High school': 'School',
                  'Secondary school': 'School',
                  'Some college/university study, no bachelor\'s degree': 'University (Bachelor, other)',
                  'Some college/university study without earning a bachelor\'s degree': 'University (Bachelor, other)',
                  'A professional degree': 'School',
                  'Professional degree': 'School',
                  'Primary/elementary school': 'School',
                  'No education': 'No education',
                  'I never completed any formal education':'No education'
                  }
def map_categories(df, column, category_map):
    '''
    Function to map the cathegorical values of a column accoring to provided map
    INPUT:
    df - the pandas dataframe which hold columns to modify 
    colum - a single column to map the values
    cathegory_map - a dictionary with the current categories as keys, and new categories as values
    
    OUTPUT:
    new_df - new dataframe with the selected column remapped
    
    '''
    df_new = df.copy()
    df_new[column] = df_new[column].map(category_map)
    return df_new
    
# map both columns
df1 = map_categories(df1, 'HighestEducationParents', profession_map)
df1 = map_categories(df1, 'FormalEducation', profession_map)
# add column for aggregations
df1['count'] = 1
# inspect output
df1.head()
# groupby to get aggregated counts
df1 = pd.DataFrame(df1.groupby(['HighestEducationParents', 'FormalEducation']).count())
# flatten
df1 = df1.reset_index()
df1.head()
# restructure the table to get correlation
df1 = df1.pivot(index='FormalEducation', columns='HighestEducationParents', values='count')
# add row sum to compute percentages
df1['sum']=df1.sum(axis=1)
# get percentages instead of couts
df1 = df1.div(df1.iloc[:,-1], axis=0 )
df1 = df1.drop('sum', axis=1)
# inspect result
df1
# plot correlation
sns.heatmap(df1, annot=True, fmt=".2f");<jupyter_output><empty_output><jupyter_text>Question 1 - ConclusionsThe heatmap shows the correlation between the highest level of education of either of the parents and the education of the responder. - One can clearly see the trend on the diagonal, meaning that childen follow the footsteps of the parents.- Examples:     - 50% of respondents with one of the parents having Master or PhD have similar — Master or PhD — degree.    - The percentage for Bachelor or other University degree is 55%    - For School education 43%.  Question 2Do people who choose a technical career path (developers, anlysts) have certain personal traits or preferences in common?<jupyter_code># choose columns of interest
list_of_characteristics_cols = ['ProblemSolving', 'BuildingThings', 'LearningNewTech', 
                                'UnderstandComputers', 'SeriousWork', 'KinshipDevelopers', 
                                'ChallengeMyself', 'CompetePeers', 'ChangeWorld']
# extract the columns of interest and inspect
df3 = df_original.copy()
df3 = df3[list_of_characteristics_cols]
df3.head()
# drop NaNs - we are interested only in valid entries; the number of responses is high enough that we can do that
df3 = df3.dropna(axis=0)
# testing single plot
sns.countplot(x="ProblemSolving",  data=df3).set(title='xxx');
# plot all categorical features of interest
fig, axes = plt.subplots(3, 3, figsize = (25,15))

# create subplots and add bar charts
cc = 0
for i in range(3):
    for j in range(3):
        sns.countplot(ax=axes[i, j], x=list_of_characteristics_cols[cc],  data=df3).set(title=list_of_characteristics_cols[cc]);
        cc = cc+1

# improve the layout to prevent overlapping labels
fig.tight_layout(pad=3.0)<jupyter_output><empty_output><jupyter_text>Question 2 - ConclusionThe charts above show that most of Stackoverflow’s responders like problem solving, to build things, to challenge themselves and are interested in changing the world.Noticably, the distribution on understanding computers suggests that many responders are not interested in having such understanding.  Question 3Based on person's characteristics, can you predict if they enjoyed the survey?<jupyter_code># Column to predict
survey_judgement_cols = ['QuestionsInteresting']
# Feature columns
person_cols = ['Professional', 'ProgramHobby', 'Country', 'EmploymentStatus', 'FormalEducation', 
               'CareerSatisfaction', 'StackOverflowSatisfaction', 'StackOverflowCommunity', 
               'StackOverflowHelpful', 'StackOverflowBetter', 'Gender', 'Race']
# copy original df
df = df_original.copy()
# extract all columns of interest
df = df[person_cols+survey_judgement_cols]
# inspect result
df.head()
# drop rows with NaNs in response column 
# we only want to account for valid entries when buildig the model 
df = df.dropna(subset=['QuestionsInteresting'], axis=0)
# extract response column
y = df['QuestionsInteresting']
df = df.drop(['QuestionsInteresting'], axis=1)
# impute the numeric columns with the mean, as this can lead to better predictive power in the ML model
num_vars = df.select_dtypes(include=['float', 'int']).columns
for col in num_vars:
    df[col].fillna((df[col].mean()), inplace=True)
# dummy the categorical variables
# to interpret the results more easily and have different influences of each level on the response
# note that the number of columns and cathegories is small enough to avoid the issues related to dummying the categorical variables 

cat_vars = df.select_dtypes(include=['object']).copy().columns
for var in  cat_vars:
   # for each cat add dummy var, drop original column
   df = pd.concat([df.drop(var, axis=1), pd.get_dummies(df[var], prefix=var, prefix_sep='_', drop_first=True)], axis=1)
# The features used for prediction
X = df
# inspect
df.head()
# split into training and test sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)
# Train the supervised model on the training set using .fit(X_train, y_train)
model = AdaBoostClassifier(base_estimator = DecisionTreeClassifier(max_depth=5, min_samples_leaf = 10), n_estimators = 4)
model.fit(X_train, y_train)
# Extract the feature importances using .feature_importances_ 
importances = model.feature_importances_ 
# Predict
y_test_preds = model.predict(X_test)
y_train_preds = model.predict(X_train)
print("Accuracy on training data: {:.4f}".format(accuracy_score(y_train, y_train_preds)))
print("Accuracy on testing data: {:.4f}".format(accuracy_score(y_test, y_test_preds)))
# display the five most important features
# which ones
indices = np.argsort(importances)[::-1]
# name of feature
columns = X_train.columns.values[indices[:5]]
# value of the weight
values = importances[indices][:5]
# Creat the plot
fig = plt.figure(figsize = (15,5))
# title
plt.title("Normalized Weights for First Five Most Predictive Features", fontsize = 16)
# plot the weights
plt.bar(np.arange(5), values, width = 0.6, align="center", color = '#00A000')
# add column names
plt.xticks(np.arange(5), columns)
plt.xlim((-0.5, 4.5))
plt.ylabel("Weight", fontsize = 12)
plt.xlabel("Feature", fontsize = 12)

plt.tight_layout()
plt.show()  
# Inspect the values (weights) of how important the features were, to verify the plot
print(values)<jupyter_output>[ 0.16400518  0.15359409  0.10808175  0.05187123  0.04196224]
StarcoderdataJupyterStructuredCleanDedup