<jupyter_start><jupyter_text>Randomized Search with Scikit-OptimizeIn this notebook, we will perform **Randomized Search** to select the best **hyperparameters** for a Gradient Boosting Classifier, using the open source Python package [Scikit-Optimize](https://scikit-optimize.github.io/stable/index.html).The randomized search is performed with the class [dummy_minimize](https://scikit-optimize.github.io/stable/modules/generated/skopt.dummy_minimize.htmlskopt.dummy_minimize). ProcedureTo tune the hyper-parameters of our model we need to:- define a model- decide which parameters to optimize- define the objective function we want to minimize. NOTEScikit-Optimize will always **minimize** the objective function, so if we want to maximize a function, for example the roc-auc, we need to **negate** the metric. Thus, instead of maximizing the roc-auc, we minimize the -roc-auc.<jupyter_code>import numpy as np
import pandas as pd

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score, train_test_split

from skopt import dummy_minimize # for the randomized search
from skopt.plots import plot_convergence
from skopt.space import Real, Integer, Categorical
from skopt.utils import use_named_args
# load dataset

breast_cancer_X, breast_cancer_y = load_breast_cancer(return_X_y=True)
X = pd.DataFrame(breast_cancer_X)
y = pd.Series(breast_cancer_y).map({0:1, 1:0})

X.head()
# the target:
# percentage of benign (0) and malign tumors (1)

y.value_counts() / len(y)
# split dataset into a train and test set

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0)

X_train.shape, X_test.shape<jupyter_output><empty_output><jupyter_text>Define the Hyperparameter SpaceScikit-optimize provides an utility function to create the range of values to examine for each hyperparameters. More details in [skopt.Space](https://scikit-optimize.github.io/stable/modules/generated/skopt.Space.html)<jupyter_code># With Integer, we create a space of integers, sampled uniformly
# between the minimum and maximum indicated values

Integer(10, 120, name="n_estimators")
# With Real, we create a space of real values, sampled uniformly
# between the minimum and maximum indicated values

Real(0, 1, name="min_samples_split")
# With Categorical, we create a space of categories

Categorical(['deviance', 'exponential'], name="loss")
# determine the hyperparameter space

param_grid = [
    Integer(10, 120, name="n_estimators"),
    Real(0, 0.999, name="min_samples_split"),
    Integer(1, 5, name="max_depth"),
    Categorical(['deviance', 'exponential'], name="loss"),
]

# Scikit-optimize parameter grid is a list
type(param_grid)<jupyter_output><empty_output><jupyter_text>Define the model<jupyter_code># set up the gradient boosting classifier

gbm = GradientBoostingClassifier(random_state=0)<jupyter_output><empty_output><jupyter_text>Define the objective functionThis is the hyperparameter response space, the function we want to minimize.<jupyter_code># We design a function to maximize the accuracy, of a GBM,
# with cross-validation

# the decorator allows our objective function to receive the parameters as
# keyword arguments. This is a requirement for scikit-optimize.

@use_named_args(param_grid)
def objective(**params):
    
    # model with new parameters
    gbm.set_params(**params)

    # optimization function (hyperparam response function)
    value = np.mean(
        cross_val_score(
            gbm, 
            X_train,
            y_train,
            cv=3,
            n_jobs=-4,
            scoring='accuracy')
    )

    # negate because we need to minimize
    return -value<jupyter_output><empty_output><jupyter_text>Randomized Search<jupyter_code># dummy_minimize performs the randomized search

search = dummy_minimize(
    objective,  # the objective function to minimize
    param_grid,  # the hyperparameter space
    n_calls=50,  # the number of subsequent evaluations of f(x)
    random_state=0,
)
# function value at the minimum.
# note that it is the negative of the accuracy

"Best score=%.4f" % search.fun
print("""Best parameters:
=========================
- n_estimators=%d
- min_samples_split=%.6f
- max_depth=%d
- loss=%s""" % (search.x[0], 
                search.x[1],
                search.x[2],
                search.x[3]))<jupyter_output>Best parameters:
=========================
- n_estimators=112
- min_samples_split=0.669967
- max_depth=3
- loss=deviance<jupyter_text>Evaluate convergence of the search[plot_convergence](https://scikit-optimize.github.io/stable/modules/generated/skopt.plots.plot_convergence.htmlskopt.plots.plot_convergence)<jupyter_code>plot_convergence(search)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup