<jupyter_start><jupyter_text>Installing PyCaret ![](https://pycaret.org/wp-content/uploads/2020/03/Divi93_43.png)[PyCaret](https://pycaret.readthedocs.io/en/latest/) is an open-source, low-code machine learning library in Python that automates machine learning workflows. It is an end-to-end machine learning and model management tool that speeds up the experiment cycle exponentially.In comparison with the other open-source machine learning libraries, PyCaret is an alternate low-code library that can be used to replace hundreds of lines of code with few words only. This makes experiments exponentially fast and efficient. PyCaret is essentially a Python wrapper around several machine learning libraries and frameworks such as scikit-learn, XGBoost, LightGBM, CatBoost, spaCy, Optuna, Hyperopt, Ray, and many more.<jupyter_code>!pip install pycaret[full]
!pip install interpret-community<jupyter_output><empty_output><jupyter_text>Getting the Data<jupyter_code>import pandas as pd
import numpy as np
from numpy import sum as arraysum
dataset = pd.read_csv('https://raw.githubusercontent.com/elvanselvano/purwadhika-final-project/main/dataset/dev_preprocessed.csv')
dataset.drop(['index'], axis=1, inplace=True)
dataset.head()
dataset.info()
dataset.describe()<jupyter_output><empty_output><jupyter_text>In order to evaluate the predictions on completely unseen data, a sample of 1015 records has been withheld from the original dataset to be used for predictions. This should not be confused with a train/test split as this particular split is performed to simulate a real life scenario. Another way to think about this is that these 1015 records were not available at the time when the machine learning experiment was performed.<jupyter_code>data_unseen = pd.read_csv('https://raw.githubusercontent.com/elvanselvano/purwadhika-final-project/main/dataset/test_preprocessed.csv')
data_unseen.drop(['index'], axis=1, inplace=True)
data_unseen.head()
data_unseen.info()
data_unseen.describe()<jupyter_output><empty_output><jupyter_text>The training data are the transactions that happened from 2016 to 2018 which contains 7231 records and the testing data are from 2018 (after the latest transaction from training data) to 2018 which contains 2310 records. We split the testing data to the recent years to simulate real-world scenario.<jupyter_code>print(f"Training Data: {dataset['SALEYEAR'].min()} to {dataset['SALEYEAR'].max()}")
print(f"Testing Data: {data_unseen['SALEYEAR'].min()} to {data_unseen['SALEYEAR'].max()}")<jupyter_output>Training Data: 2016 to 2018
Testing Data: 2018 to 2018<jupyter_text>Setting up Environment in PyCaretThe `setup()` function initializes the environment in pycaret and creates the transformation pipeline to prepare the data for modeling and deployment. Besides performing some basic processing tasks by default, PyCaret also offers wide array of pre-processing features which structurally elevates an average machine learning experiment to an advanced solution.<jupyter_code>import pycaret.regression as reg
regression_setup = reg.setup(data=dataset, target='PRICE', session_id=42, use_gpu=True, train_size=0.8, normalize=True, transformation=True,
                         categorical_features=['GRADE', 'HEAT', 'STYLE', 'AC', 'STRUCT', 'EXTWALL', 'ROOF', 'INTWALL', 'WARD', 'RMDL', 'CNDTN'],
                         numeric_features=['BATHRM', 'HF_BATHRM', 'BEDRM', 'SALE_NUM', 'SQFT_ROOMS', 'AYB_SALEYEAR_DIFF', 'EYB_SALEYEAR_DIFF', 'GBA', 'KITCHENS', 'FIREPLACES', 'LANDAREA', 'SALEYEAR'])<jupyter_output><empty_output><jupyter_text>Baseline Model When we evaluate a model's performance, it is important to create a baseline model so we can see whether our model improved from the baseline and how much did it improve. Fundamentally, a baseline is a model that is both simple to set up and has a reasonable chance of providing decent results. Experimenting with them is usually quick and low cost, since implementations are widely available in popular packages.Source: https://blog.insightdatascience.com/always-start-with-a-stupid-model-no-exceptions-3a22314b9aaa<jupyter_code>from sklearn.dummy import DummyRegressor
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score

X = dataset.drop('PRICE', axis=1)
y = dataset['PRICE']

def evaluate_baseline(model):
     scores = cross_val_score(model, X, y, cv=5, scoring='neg_mean_absolute_error')
     scores = -scores.mean()
     return scores

dummy_reg_mean = DummyRegressor(strategy="mean")
dummy_reg_median = DummyRegressor(strategy='median')
dummy_reg_quantile = DummyRegressor(strategy='quantile', quantile=0.44)
dummy_reg_constant = DummyRegressor(strategy='constant', constant=450000)

print('Dummy Regressor (mean):', evaluate_baseline(dummy_reg_mean))
print('Dummy Regressor (median): ', evaluate_baseline(dummy_reg_median))
print('Dummy Regressor (quantile):', evaluate_baseline(dummy_reg_quantile))
print('Dummy Regressor (constant):', evaluate_baseline(dummy_reg_constant))<jupyter_output>Dummy Regressor (mean): 231931.97336832358
Dummy Regressor (median):  242516.16876075938
Dummy Regressor (quantile): 244499.10204320285
Dummy Regressor (constant): 252708.41737758572<jupyter_text>Comparing All Models When comparing models, we will create all models using 5 fold cross validation. We will focus on the Mean Absolute Error (MAE) because it is robuts to outliers which exist in our dataset.<jupyter_code>models = reg.compare_models(fold=5, sort='MAE')<jupyter_output><empty_output><jupyter_text>Create a Model![](https://avatars.mds.yandex.net/get-yablogs/51163/file_1500371746775/orig)[CatBoost](https://catboost.ai/) is an algorithm for gradient boosting on decision trees. It is developed by Yandex researchers and engineers, and is used for search, recommendation systems, personal assistant, self-driving cars, weather prediction and many other tasks at Yandex and in other companies, including CERN, Cloudflare, Careem taxi. It is in open-source and can be used by anyone.Based on the above benchmark, we can see that CatBoost is the top performing model with Mean Absolute Error of 7.791458e+04 during the 5-fold cross validation. Therefore, we will create a catboost model and tune its parameters in the next step.<jupyter_code>catboost = reg.create_model('catboost', random_state=42)<jupyter_output><empty_output><jupyter_text>Tuning the Model In order to tune the model, we use the library [Hyperopt](http://hyperopt.github.io/hyperopt/), an open source hyperparameter optimization framework to automate hyperparameter search. It efficiently search large spaces and prune unpromising trials for faster results. Since it doesn't produce better model, we will proceed with the original model.<jupyter_code>tuned_catboost = reg.tune_model(catboost, search_library = "tune-sklearn", search_algorithm="hyperopt",  optimize="MSE", n_iter=50)<jupyter_output><empty_output><jupyter_text>Prediction Error Plot<jupyter_code>reg.plot_model(catboost, plot = 'error')<jupyter_output><empty_output><jupyter_text>Learning Curve<jupyter_code>reg.plot_model(catboost, plot = 'learning')<jupyter_output><empty_output><jupyter_text>Validation Curve<jupyter_code>reg.plot_model(catboost, plot = 'vc')<jupyter_output><empty_output><jupyter_text>Feature Importance Using SHAP<jupyter_code>reg.interpret_model(catboost)<jupyter_output><empty_output><jupyter_text>Finalize and save the model After evaluate the results on development data, we will finalize the model by train it using the whole development data and then evaluate it on the testing data to simulate real-world scenario.<jupyter_code>final_model = reg.finalize_model(catboost)
reg.save_model(final_model, model_name='catboost_model')<jupyter_output>Transformation Pipeline and Model Successfully Saved<jupyter_text>Predictions on unseen data During prediction, we will use prediction interval. Prediction interval is a quantification of the uncertainty on a prediction which provides a probabilistic upper and lower bounds on the estimate of an outcome variable.In this case, we will use 95% prediction interval which means: *Given a prediction of ‘y’ given ‘x’, there is a 95% likelihood that the range ‘a’ to ‘b’ covers the true outcome*![](https://machinelearningmastery.com/wp-content/uploads/2018/04/Relationship-between-prediction-actual-value-and-prediction-interval.png)Source: https://machinelearningmastery.com/prediction-intervals-for-machine-learning/<jupyter_code>saved_catboost = reg.load_model('catboost_model')<jupyter_output>Transformation Pipeline and Model Successfully Loaded<jupyter_text>A prediction interval is different from a confidence interval. A confidence interval quantifies the uncertainty on an estimated population variable, such as the mean or standard deviation. Whereas a prediction interval quantifies the uncertainty on a single observation estimated from the population.<jupyter_code>data_unseen_features = data_unseen.drop('PRICE', axis=1)
data_unseen_target = data_unseen['PRICE']
predictions = reg.predict_model(saved_catboost, data=data_unseen_features)
label = predictions['Label']
y = data_unseen_target<jupyter_output><empty_output><jupyter_text>We can estimate the standard deviation in the prediction direction using the following formula:`SE = sqrt(1 / (N - 2) * e(i)^2 for i to N)`<jupyter_code>sum_errs = arraysum((y - label)**2)
stdev = np.sqrt(1/(len(y)-2) * sum_errs)
print(sum_errs, stdev)<jupyter_output>12896151072290.123 112830.19360365305<jupyter_text>We will use the significance level of 50%, which is 0.67449 standard deviations.<jupyter_code>interval = 0.67449 * stdev
print(interval)<jupyter_output>76102.83728372795<jupyter_text>Once the interval is calculated, we can summarize the bounds on the prediction to the user.<jupyter_code>lower, upper = label - interval, label + interval<jupyter_output><empty_output><jupyter_text>Evaluate Predictions<jupyter_code>predictions<jupyter_output><empty_output><jupyter_text>First, we will evaluate the number of records that are in our prediction interval. We can see that out of 1015 records, there are 637 records that are in the interval.<jupyter_code>data_unseen.sample(10)
data_unseen.shape
data_unseen[(data_unseen['PRICE'] > lower) & (data_unseen['PRICE'] < upper)].shape
from sklearn.metrics import max_error
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import r2_score
from sklearn.metrics import mean_squared_error

print("Max Error: " + str(max_error(data_unseen_target, predictions["Label"])))
print("Mean Absolute Error: " + str(mean_absolute_error(data_unseen_target, predictions["Label"])))
print("R2 Score: " + str(r2_score(data_unseen_target, predictions["Label"])))
print("Root Mean Squared Error: " + str(mean_squared_error(data_unseen_target, predictions["Label"], squared=False)))<jupyter_output>Max Error: 653394.3540491124
Mean Absolute Error: 79837.02015288273
R2 Score: 0.7870558902427096
Root Mean Squared Error: 112718.97603734043
StarcoderdataJupyterStructuredCleanDedup