<jupyter_start><jupyter_text>Gradient boost guided exampleHaving walked through gradient boost by hand, now let's try it with SKlearn.  We'll still use the European Social Survey Data, but now with a categorical outcome: Whether or not someone lives with a partner.<jupyter_code>df = pd.read_csv((
    "https://raw.githubusercontent.com/Thinkful-Ed/data-201-resources/"
    "master/ESS_practice_data/ESSdata_Thinkful.csv")).dropna()

# Definine outcome and predictors.
# Set our outcome to 0 and 1.
y = df['partner'] - 1
X = df.loc[:, ~df.columns.isin(['partner', 'cntry', 'idno'])]

# Make the categorical variable 'country' into dummies.
X = pd.concat([X, pd.get_dummies(df['cntry'])], axis=1)

# Create training and test sets.
offset = int(X.shape[0] * 0.9)

# Put 90% of the data in the training set.
X_train, y_train = X[:offset], y[:offset]

# And put 10% in the test set.
X_test, y_test = X[offset:], y[offset:]<jupyter_output><empty_output><jupyter_text>Since we're now working with a binary outcome, we've switched to a classifier.  Now our loss function can't be the residuals.  Our options are "deviance", or "exponential".  Deviance is used for logistic regression, and we'll try that here.<jupyter_code># We'll make 500 iterations, use 2-deep trees, and set our loss function.
params = {'n_estimators': 500,
          'max_depth': 2,
          'loss': 'deviance'}

# Initialize and fit the model.
clf = ensemble.GradientBoostingClassifier(**params)
clf.fit(X_train, y_train)

predict_train = clf.predict(X_train)
predict_test = clf.predict(X_test)

# Accuracy tables.
table_train = pd.crosstab(y_train, predict_train, margins=True)
table_test = pd.crosstab(y_test, predict_test, margins=True)

train_tI_errors = table_train.loc[0.0,1.0] / table_train.loc['All','All']
train_tII_errors = table_train.loc[1.0,0.0] / table_train.loc['All','All']

test_tI_errors = table_test.loc[0.0,1.0]/table_test.loc['All','All']
test_tII_errors = table_test.loc[1.0,0.0]/table_test.loc['All','All']

print((
    'Training set accuracy:\n'
    'Percent Type I errors: {}\n'
    'Percent Type II errors: {}\n\n'
    'Test set accuracy:\n'
    'Percent Type I errors: {}\n'
    'Percent Type II errors: {}'
).format(train_tI_errors, train_tII_errors, test_tI_errors, test_tII_errors))<jupyter_output>Training set accuracy:
Percent Type I errors: 0.04650845608292417
Percent Type II errors: 0.17607746863066012

Test set accuracy:
Percent Type I errors: 0.06257668711656442
Percent Type II errors: 0.18527607361963191<jupyter_text>Unlike decision trees, gradient boost solutions are not terribly easy to interpret on the surface.  But they aren't quite a black box.  We can get a measure of how important various features are by counting how many times a feature is used over the course of many decision trees.<jupyter_code>feature_importance = clf.feature_importances_

# Make importances relative to max importance.
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
pos = np.arange(sorted_idx.shape[0]) + .5
plt.subplot(1, 2, 2)
plt.barh(pos, feature_importance[sorted_idx], align='center')
plt.yticks(pos, X.columns[sorted_idx])
plt.xlabel('Relative Importance')
plt.title('Variable Importance')
plt.show()<jupyter_output><empty_output><jupyter_text>It appears that age and happiness are the most important features in predicting whether or not someone lives with a partner.<jupyter_code>df.describe()<jupyter_output><empty_output><jupyter_text>DRILL: Improve this gradient boost modelWhile this model is already doing alright, we've seen from the Type I and Type II error rates that there is definitely room for improvement.  Your task is to see how low you can get the error rates to go in the test set, based on your model in the training set.  Strategies you might use include:* Creating new features* Applying more overfitting-prevention strategies like subsampling* More iterations* Trying a different loss function* Changing the structure of the weak learner: Allowing more leaves in the tree, or other modificationsHave fun!<jupyter_code>import itertools
n_estimators = [100, 250, 500, 750, 1000]
max_depth = [2, 3, 5, 8, 10]
loss = ['deviance', 'exponential']
subsample = [0.3, 0.5, 0.8, 0.9, 1.0]

lists = [n_estimators, max_depth, loss, subsample]

param_dict = pd.DataFrame(list(itertools.product(*lists)), 
                          columns=['n_estimators', 'max_depth', 'loss', 'subsample']).to_dict(orient='records')
for params in param_dict:
    clf = ensemble.GradientBoostingClassifier(**params)
    clf.fit(X_train, y_train)

    predict_test = clf.predict(X_test)

    table_test = pd.crosstab(y_test, predict_test, margins=True)

    test_tI_errors = table_test.loc[0.0,1.0]/table_test.loc['All','All']
    test_tII_errors = table_test.loc[1.0,0.0]/table_test.loc['All','All']

    print((
        '---Parameters--- {}\n\n'
        'Test set accuracy:\n'
        'Percent Type I errors: {}\n'
        'Percent Type II errors: {}'
    ).format(params, test_tI_errors, test_tII_errors))
    params['typeI'] = test_tI_errors
    params['typeII'] = test_tII_errors
param_df = pd.DataFrame(param_dict, columns=['n_estimators', 'max_depth', 'loss', 'subsample', 'typeI', 'typeII'])
param_df.head()
param_df['total_error'] = param_df['typeI'] +  param_df['typeII']
min_fp = param_df.sort_values('typeI')
min_fn = param_df.sort_values('typeII')
min_ttlerr = param_df.sort_values('total_error')
min_fp.head()
min_fn.head()
min_ttlerr.head(30)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup