<jupyter_start><jupyter_text>Tutorial of base functions  For classification you will need to use `CatBoostClassifier`.  Binary classification There are several `loss_function`s to use:* `Logloss`,* `CrossEntropy`,* or a custom objective function.<jupyter_code>from catboost import CatBoostClassifier
# Initialize data
cat_features = [0,1,2] # indices for cat_features
train_data = [["a","b",1,4,5,6],["a","b",4,5,6,7],["c","d",30,40,50,60]] # cat_features should be string or int
train_labels = [1,1,-1]
test_data = [["a","b",2,4,6,8],["a","d",1,4,50,60]]
# Initialize CatBoostClassifier
model = CatBoostClassifier(iterations=2, learning_rate=1, depth=2, loss_function='Logloss')
# Fit model
model.fit(train_data, train_labels, cat_features)
# Get predicted classes
preds_class = model.predict(test_data)
# Get predicted probabilities for each class
preds_proba = model.predict_proba(test_data)
# Get predicted RawFormulaVal
preds_raw = model.predict(test_data, prediction_type='RawFormulaVal')
print(preds_class)<jupyter_output>[ 1.  1.]<jupyter_text>For Multiclassification use `loss_function='MultiClass'`.<jupyter_code>from sklearn.datasets import load_iris
iris = load_iris()
from sklearn.model_selection import train_test_split
x_train, x_test, y_train, y_test = train_test_split(iris.data, iris.target, test_size=0.3, random_state=42)
# Initialize CatBoostClassifier
model = CatBoostClassifier(
    iterations=100,
    learning_rate=1,
    depth=6,
    loss_function='MultiClass',
    classes_count=3,
    logging_level='Silent'
)
# Fit model
model.fit(x_train, y_train)
from sklearn.metrics import accuracy_score, log_loss
# Get predicted classes
preds_class = model.predict(x_test)
print("accuracy = {}".format(accuracy_score(y_test, preds_class)))
# Get predicted probabilities for each class
preds_proba = model.predict_proba(x_test)
print("log_loss = {}".format(log_loss(y_test, preds_proba)))<jupyter_output>accuracy = 1.0
log_loss = 0.00454027925764<jupyter_text>Also you can use pre-training results (baseline). The results (only `raw_values`, not probability or class) can be set as baseline for the new model.The shape of the baseline depends on the machine learning problem being solved:* Classification: a two-dimensional array with `shape = (length of data, number of classes)`* Regression: a one-dimensional array.<jupyter_code># Get baseline (only with prediction_type='RawFormulaVal')
baseline = model.predict(x_train, prediction_type='RawFormulaVal')
# Fit new model
model.fit(x_train, y_train, baseline=baseline)<jupyter_output><empty_output><jupyter_text>Using object weights The weight for each object in the input data can be set in the form of a one-dimensional array like data (length = data length).The weight is used for calculating the final values of the trees. By default, it is set to 1 for all objects.<jupyter_code>import numpy as np
weight = np.random.random(len(x_train))
model.fit(x_train, y_train, sample_weight=weight)<jupyter_output><empty_output><jupyter_text>Using best model If this parameter is set, the number of trees that are saved in the resulting model is defined as follows:Build the number of trees defined by the training parameters.Identify the iteration with the optimal loss function value.No trees are saved after this iteration.The `eval_set` parameter is obligatory for the `fit` method if the best model mode is on.`eval_set` should have the same values for the following training dataset parameters:* `cat_features`* `baseline` (if set).<jupyter_code>eval_set = (x_test, y_test)
model.fit(x_train, y_train, use_best_model=True, eval_set=eval_set)<jupyter_output><empty_output><jupyter_text>Using `staged_predict` The values of the model can be output for each i-th tree of the model by taking into consideration only the trees in the range [1;i].This feature is implemented via the `staged_predict` method (available in the `CatBoostClassifier` and `CatBoostRegressor` classes).<jupyter_code>staged_predictions = model.staged_predict(x_test)
# It is equivalent to use predict() with `ntree_end` in loop, but faster
staged_predictions = []
for i in range(1, model.tree_count_ + 1):
    staged_predictions.append(model.predict(x_test, ntree_end=i))<jupyter_output><empty_output><jupyter_text>Custom objective function A custom objective function can be used by specifying a python object as the value for the `loss_function` parameter. In this case the objective is always maximized.Depending on the machine learning problem the python object should have one of the following functions defined:* `calc_ders_range`* `calc_ders_multi(approxes, target, weight)` (for multiclassification)<jupyter_code>import math
class LoglossObjective(object):
    def calc_ders_range(self, approxes, targets, weights):
        # approxes, targets, weights are indexed containers of floats
        # (containers which have only __len__ and __getitem__ defined).
        # weights parameter can be None.
        #
        # To understand what these parameters mean, assume that there is
        # a subset of your dataset that is currently being processed.
        # approxes contains current predictions for this subset,
        # targets contains target values you provided with the dataset.
        #
        # This function should return a list of pairs (der1, der2), where
        # der1 is the first derivative of the loss function with respect
        # to the predicted value, and der2 is the second derivative.
        #
        # In our case, logloss is defined by the following formula:
        # target * log(sigmoid(approx)) + (1 - target) * (1 - sigmoid(approx))
        # where sigmoid(x) = 1 / (1 + e^(-x)).
        assert len(approxes) == len(targets)
        if weights is not None:
            assert len(weights) == len(approxes)

        result = []
        for index in xrange(len(targets)):
            e = math.exp(approxes[index])
            p = e / (1 + e)
            der1 = (1 - p) if targets[index] > 0.0 else -p
            der2 = -p * (1 - p)

            if weights is not None:
                der1 *= weights[index]
                der2 *= weights[index]

            result.append((der1, der2))

        return result
model = CatBoostClassifier(
    random_seed=0, 
    loss_function=LoglossObjective(), 
    eval_metric="Logloss", 
    logging_level='Silent'
)
# Fit model
model.fit(x_train, y_train)
# Only prediction_type='RawFormulaVal' is allowed with custom `loss_function`
preds_raw = model.predict(x_test, prediction_type='RawFormulaVal')<jupyter_output><empty_output><jupyter_text>Custom metric function for overfitting detector and best model selection To set a custom metric function for overfitting detector and best model selection, create an object that implements the following interface:<jupyter_code>class CustomMetric(object):
    def get_final_error(self, error, weight):
        return 0.0

    def is_max_optimal(self):
        return True

    def evaluate(self, approxes, target, weight):
        # approxes - list of list-like objects (one object per approx dimension)
        # target - list-like object
        # weight - list-like object, can be None
        return 0.0, 0.0<jupyter_output><empty_output><jupyter_text>The following is an example of an implementation of the Logloss function:<jupyter_code>import math
class LoglossMetric(object):
    def get_final_error(self, error, weight):
        return error / (weight + 1e-38)

    def is_max_optimal(self):
        return True

    def evaluate(self, approxes, target, weight):
        # approxes is a list of indexed containers
        # (containers with only __len__ and __getitem__ defined),
        # one container per approx dimension.
        # Each container contains floats.
        # weight is a one dimensional indexed container.
        # target is float.
        
        # weight parameter can be None.
        # Returns pair (error, weights sum)
        
        assert len(approxes) == 1
        assert len(target) == len(approxes[0])

        approx = approxes[0]

        error_sum = 0.0
        weight_sum = 0.0

        for i in xrange(len(approx)):
            w = 1.0 if weight is None else weight[i]
            weight_sum += w
            error_sum += w * (target[i] * approx[i] - math.log(1 + math.exp(approx[i])))

        return error_sum, weight_sum

model = CatBoostClassifier(iterations=5, random_seed=0, eval_metric=LoglossMetric())
model.fit(x_train, y_train, use_best_model=True, eval_set=(x_test, y_test))<jupyter_output>0:	learn: -0.6413782	test: -0.6411148	best: -0.6411148 (0)	total: 10.3ms	remaining: 41.3ms
1:	learn: -0.5916443	test: -0.5978797	best: -0.5978797 (1)	total: 21.9ms	remaining: 32.9ms
2:	learn: -0.5449263	test: -0.5490158	best: -0.5490158 (2)	total: 31.3ms	remaining: 20.9ms
3:	learn: -0.5022478	test: -0.5051050	best: -0.5051050 (3)	total: 40.6ms	remaining: 10.2ms
4:	learn: -0.4643401	test: -0.4653301	best: -0.4653301 (4)	total: 50.9ms	remaining: 0us

bestTest = -0.4653300749
bestIteration = 4

Shrink model to first 5 iterations.<jupyter_text>Pass the created object to the `eval_metric` parameter:<jupyter_code>model = CatBoostClassifier(eval_metric=CustomMetric(), logging_level='Silent')
model.fit(x_train, y_train, use_best_model=True, eval_set=(x_test, y_test))<jupyter_output><empty_output><jupyter_text>Use cross validation:<jupyter_code># From catboost
from catboost import cv, Pool
pool = Pool(x_train, y_train)
params = {'iterations': 100, 'depth': 2, 'loss_function': 'MultiClass', 'classes_count': 3, 'logging_level': 'Silent'}
scores = cv(pool, params)<jupyter_output><empty_output><jupyter_text>`cv` returns the following structure:<jupyter_code>scores.head()<jupyter_output><empty_output><jupyter_text>Each of the arrays contains loss in each iteration (like `staged_predict`):<jupyter_code># From sklearn
from sklearn.model_selection import cross_val_score

model = CatBoostClassifier(iterations=400, loss_function='MultiClass', custom_loss='Accuracy', logging_level='Silent')
scores = cross_val_score(model, x_train, y_train, scoring='accuracy', fit_params=None)<jupyter_output>/home/nikitxskv/.local/lib/python2.7/site-packages/sklearn/base.py:114: DeprecationWarning: Estimator CatBoostClassifier modifies parameters in __init__. This behavior is deprecated as of 0.18 and support for this behavior will be removed in 0.20.
  % type(estimator).__name__, DeprecationWarning)
/home/nikitxskv/.local/lib/python2.7/site-packages/sklearn/base.py:114: DeprecationWarning: Estimator CatBoostClassifier modifies parameters in __init__. This behavior is deprecated as of 0.18 and support for this behavior will be removed in 0.20.
  % type(estimator).__name__, DeprecationWarning)
/home/nikitxskv/.local/lib/python2.7/site-packages/sklearn/base.py:114: DeprecationWarning: Estimator CatBoostClassifier modifies parameters in __init__. This behavior is deprecated as of 0.18 and support for this behavior will be removed in 0.20.
  % type(estimator).__name__, DeprecationWarning)<jupyter_text>Also catboost support `GridSearchCV` from `sklearn`:<jupyter_code>from sklearn.model_selection import GridSearchCV

# Initialize
model = CatBoostClassifier(loss_function='MultiClass', logging_level='Silent')
# Init params dict
params = {'iterations': [10, 20, 30], 'learning_rate': [0.01, 0.03, 0.1], 'depth': [4, 6, 8]}
# Get GridSearchCV
model = GridSearchCV(model, params)
model.fit(x_train, y_train)
print(sorted(model.cv_results_.keys()))<jupyter_output>['mean_fit_time', 'mean_score_time', 'mean_test_score', 'mean_train_score', 'param_depth', 'param_iterations', 'param_learning_rate', 'params', 'rank_test_score', 'split0_test_score', 'split0_train_score', 'split1_test_score', 'split1_train_score', 'split2_test_score', 'split2_train_score', 'std_fit_time', 'std_score_time', 'std_test_score', 'std_train_score']<jupyter_text>Save model to file:<jupyter_code>model = CatBoostClassifier(iterations=400, loss_function='MultiClass', custom_loss='Accuracy', logging_level='Silent')
model.fit(x_train, y_train)
model.save_model('model')<jupyter_output><empty_output><jupyter_text>Load model:<jupyter_code>model.load_model('model')<jupyter_output><empty_output><jupyter_text>Copy model:<jupyter_code>model_copy = model.copy()<jupyter_output><empty_output><jupyter_text>Get `feature_importance`:<jupyter_code># for get attr feature_importance_ you should add parameter `calc_feature_importance=True` to model
model = CatBoostClassifier(
    iterations=400,
    loss_function='MultiClass',
    calc_feature_importance=True,
    logging_level='Silent'
)
model.fit(x_train, y_train)
print(model.feature_importances_)<jupyter_output>[4.031255278950752, 6.348744477463416, 43.90142650200039, 45.71857374158544]<jupyter_text>Get model params:<jupyter_code># params used for fitting the model
model.get_params()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup