<jupyter_start><jupyter_text>Correlation analysis<jupyter_code>train_df.corr()['price'].sort_values()<jupyter_output><empty_output><jupyter_text>We can find the `sqft_living` `grade` and `sqft_above` have strong correlations with price.<jupyter_code>train_df.drop(['id', 'price'], axis=1, inplace=True)
test_df.drop(['id', 'price'], axis=1, inplace=True)
train_df.shape
train_df.dtypes<jupyter_output><empty_output><jupyter_text>Feature transformationdate is string type so we convert it to numerica by 1. Keeping 6 characters (Year Month)2. Use LabelEncoder to convert categorical to numerical<jupyter_code>def convert_to_date(date_string: str):
    """
    Only keep year and month 
    """
    # date = date_string[:8]
    # return pd.to_datetime(date, format='%Y%m%d', errors='ignore')
    return date_string[:6]
train_df.date = train_df.date.apply(convert_to_date)
test_df.date = test_df.date.apply(convert_to_date)
from sklearn.preprocessing import LabelEncoder

label_encoder = LabelEncoder()

train_df.date = label_encoder.fit_transform(train_df.date)
test_df.date = label_encoder.fit_transform(test_df.date)<jupyter_output><empty_output><jupyter_text>Normlize columnsBecause value range of some columns are very big. This could dominate the train process.We normalize them by MinMaxScaler<jupyter_code>from sklearn import preprocessing

# Create x, where x the 'scores' column's values as floats
x = train_df.values.astype(float)

# Create a minimum and maximum processor object
min_max_scaler = preprocessing.MinMaxScaler()

# Create an object to transform the data to fit minmax processor
x_scaled = min_max_scaler.fit_transform(x)
x_scaled.shape
# Run the normalizer on the dataframe
train_df_norm = pd.DataFrame(x_scaled, columns=train_df.columns)
train_df_norm.head()<jupyter_output><empty_output><jupyter_text>Distribution of priceLet's see the distribution of house price<jupyter_code>y_train.hist(xlabelsize=30, ylabelsize=30, bins=120,figsize=(28,15))
y_train.describe()
(y_train>640000).sum()<jupyter_output><empty_output><jupyter_text>75% of the house prices are in the range between 0 and 640000From the table, we can see the descriptive statistics of training data   Skewness[skewness](https://whatis.techtarget.com/definition/skewness)The skewness should be about zero for normal distribution. A skenewss  value greater than zero means that there is more weight in the left tail of the distribution<jupyter_code>plt.figure()
qq = stats.probplot(y_train, plot=plt)
plt.show()

print("Skewness: {:.3f}".format(y_train.skew()))<jupyter_output><empty_output><jupyter_text>Our data has a positive skewness. There is more weight in the left tail of the price distributionNext, we take a log of price column and see what happens!<jupyter_code>y_train = np.log1p(y_train)
y_test = np.log1p(y_test)

y_train.hist(xlabelsize=30, ylabelsize=30, bins=120,figsize=(28,15))


print("Skewness: {:.3f}".format(y_train.skew()))<jupyter_output>Skewness: 0.419<jupyter_text>The distribution is more like a normal distribution than before!  Q-Q Plot<jupyter_code>plt.figure()
qq = stats.probplot(y_train, plot=plt)
plt.show()<jupyter_output><empty_output><jupyter_text>By taking a log of price column, it is close to normal distribution<jupyter_code>train_df.isnull().sum()<jupyter_output><empty_output><jupyter_text>GBM<jupyter_code>import numpy as np
import matplotlib.pyplot as plt

from sklearn import ensemble
from sklearn import datasets
from sklearn.utils import shuffle
from sklearn.metrics import mean_squared_error
X_train, y_train = train_df.values, y_train.values
X_test, y_test = test_df.values, y_test.values


params = {'n_estimators': 500, 'max_depth': 5, 'min_samples_split': 2,
          'learning_rate': 0.01, 'loss': 'ls'}

gb_reg = ensemble.GradientBoostingRegressor(**params)

gb_reg.fit(X_train, y_train)
mse = mean_squared_error(y_test, gb_reg.predict(X_test))
print("MSE: %.4f" % mse)

# #############################################################################
# Plot training deviance

# compute test set deviance
test_score = np.zeros((params['n_estimators'],), dtype=np.float64)

for i, y_pred in enumerate(gb_reg.staged_predict(X_test)):
    test_score[i] = gb_reg.loss_(y_test, y_pred)

plt.figure(figsize=(12, 6))
plt.subplot(1, 2, 1)
plt.title('Deviance')
plt.plot(np.arange(params['n_estimators']) + 1, gb_reg.train_score_, 'b-',
         label='Training Set Deviance')
plt.plot(np.arange(params['n_estimators']) + 1, test_score, 'r-',
         label='Test Set Deviance')
plt.legend(loc='upper right')
plt.xlabel('Boosting Iterations')
plt.ylabel('Deviance')

# #############################################################################
# Plot feature importance
feature_importance = gb_reg.feature_importances_
# make importances relative to max importance
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
pos = np.arange(sorted_idx.shape[0]) + .5
plt.subplot(1, 2, 2)
plt.barh(pos, feature_importance[sorted_idx], align='center')
plt.yticks(pos, train_df.columns[sorted_idx])
plt.xlabel('Relative Importance')
plt.title('Variable Importance')
plt.show()<jupyter_output>MSE: 0.0242<jupyter_text>Random Forest<jupyter_code>from sklearn.ensemble import RandomForestRegressor




params = {'n_estimators': 500, 'max_depth': 8, 'min_samples_split': 2}

rf_reg = RandomForestRegressor(**params)

rf_reg.fit(X_train, y_train)
mse = mean_squared_error(y_test, rf_reg.predict(X_test))
print("MSE: %.4f" % mse)


# #############################################################################
# Plot feature importance
feature_importance = rf_reg.feature_importances_
# make importances relative to max importance
feature_importance = 100.0 * (feature_importance / feature_importance.max())
sorted_idx = np.argsort(feature_importance)
pos = np.arange(sorted_idx.shape[0]) + .5
plt.subplot(1, 2, 2)
plt.barh(pos, feature_importance[sorted_idx], align='center')
plt.yticks(pos, train_df.columns[sorted_idx])
plt.xlabel('Relative Importance')
plt.title('Variable Importance')
plt.show()<jupyter_output>MSE: 0.0308<jupyter_text>XGBoost<jupyter_code>import xgboost as xgb
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)
best_xgb_model = xgboost.XGBRegressor(colsample_bytree=0.4,
                 gamma=0,                 
                 learning_rate=0.07,
                 max_depth=3,
                 min_child_weight=1.5,
                 n_estimators=10000,                                                                    
                 reg_alpha=0.75,
                 reg_lambda=0.45,
                 subsample=0.6,
                 seed=42)
best_xgb_model.fit(train_x,train_y)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup