<jupyter_start><jupyter_text>Regression Modeling _by <NAME>_  Contents: - [Import and Set up Data](Import-and-Set-up-Data) - [Null Model ](Null-Model) - [Model 1: Linear Regression ](Model-1:-Linear-Regression) - [Model 2: Linear Regression with RFE](Model-2:-Linear-Regression-with-RFE) - [Model 3: RFE and Ridge](Model-3:-RFE-and-Ridge) - [Model 4: Lasso with RFE](Model-4:-Lasso-with-RFE) - [Model 5: ElasticNet with RFE](Model-5:-ElasticNet-with-RFE) - [Model 6: Linear Regression with Limited Features](Model-6:-Linear-Regression-with-Limited-Features) - [Model 7: Elastic Net with Limited Features](Model-7:-Elastic-Net-with-Limited-Features) - [Model 8: Lasso with Limited Features ](Model-8:-Lasso-with-Limited-Features) - [Model 9: Ridge with Limited Features](Model-9:-Ridge-with-Limited-Features) - [Model 10: KNN Regressor](Model-10:-KNN-Regressor) - [Model 11: Simple Neural Networks](Model-11:-Simple-Neural-Networks) - [Model 12: Best Model: Neural Networks adding Layers and Early Stopping](Model-12:-Best-Model:-Neural-Networks-adding-Layers-and-Early-Stopping) - [Model 13: Decision Tree Regressor](Model-13:-Decision-Tree-Regressor)<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import pickle
from sklearn.impute import KNNImputer
from sklearn.neighbors import KNeighborsRegressor
from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score
from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet, RidgeCV, LassoCV, ElasticNetCV
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error 
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, PolynomialFeatures 
from sklearn.feature_selection import RFE
from sklearn.pipeline import make_pipeline
from sklearn.compose import make_column_transformer, make_column_selector
import tensorflow as tf
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, BatchNormalization
from tensorflow.keras.regularizers import l1, l2, l1_l2
from tensorflow.keras.callbacks import EarlyStopping
from sklearn.tree import DecisionTreeRegressor
import seaborn as sns
from sklearn.tree import plot_tree
from sklearn.dummy import DummyRegressor<jupyter_output><empty_output><jupyter_text>Import and Set up Data[(back to top)](Regression-Modeling)<jupyter_code>df = pd.read_csv('../data/cleaned_aqi_and_temp_data_2017-2019.csv')
df.info()
df["weekday"] = df["weekday"].astype("category")

df["month"] = df["month"].astype("category")

df['date'] = pd.to_datetime(df['date'])

df.drop('pct_change_aqi', axis = 1, inplace = True)

df.set_index('date', inplace = True)

df.head()
X = df.drop('cumulative_aqi', axis =1 )
y = df['cumulative_aqi']
X_train, X_test, y_train, y_test = train_test_split(X, y)
def errors(y_true, y_pred):
    rmse = mean_squared_error(y_true, y_pred, squared = False)
    r2 = r2_score(y_true, y_pred)
    mae = mean_absolute_error(y_true, y_pred)
    return (f'root mean sqaured error is {rmse}, r squared is {r2}, and mean absolute error is {mae}')<jupyter_output><empty_output><jupyter_text>Null Model [(back to top)](Regression-Modeling)<jupyter_code>dr = DummyRegressor()
dummy = dr.fit(X_train, y_train)
dummy.score(X_test, y_test)
null_preds = dummy.predict(X_test)
errors(y_test, null_preds)<jupyter_output><empty_output><jupyter_text>Model 1: Linear Regression [(back to top)](Regression-Modeling)<jupyter_code>pipe1 = make_pipeline(KNNImputer(), StandardScaler(), LinearRegression())
pipe1.fit(X_train, y_train)
cross_val_score(pipe1, X_train, y_train)
predict1 = pipe1.predict(X_test)
errors(y_test, predict1)
coef = pipe1.named_steps['linearregression'].coef_

name = X_train.columns

importance = pd.DataFrame(zip(name, coef), columns=['name', 'coef']).sort_values(by='coef', ascending=False)
importance<jupyter_output><empty_output><jupyter_text>Model 2: Linear Regression with RFE[(back to top)](Regression-Modeling)<jupyter_code>pipe2 = make_pipeline(KNNImputer(),StandardScaler(), RFE(estimator=LinearRegression()), LinearRegression())
params = {'rfe__n_features_to_select': [15, 10, 8, 5]}
gs2 = GridSearchCV(pipe2, params, n_jobs=1)
gs2.fit(X_train, y_train)
print(gs2.score(X_train, y_train))
print(gs2.score(X_test, y_test))
predict2 = gs2.predict(X_test)
errors(y_test, predict2)
gs2.best_params_
coef = gs2.best_estimator_['linearregression'].coef_
name = X_train.columns
importance = pd.DataFrame(zip(name, coef), columns=['name', 'coef']).sort_values(by='coef', ascending=False)
importance<jupyter_output><empty_output><jupyter_text>Model 3: RFE and Ridge [(back to top)](Regression-Modeling)<jupyter_code>pipe3 = make_pipeline(KNNImputer(),StandardScaler(), RFE(estimator=LinearRegression()), Ridge())
params = {'rfe__n_features_to_select': [17, 15, 10, 8, 5],
          'ridge__alpha': [.01, .1 ,1 ,10 ,100, 1000]}
gs3 = GridSearchCV(pipe3, params, n_jobs=1)
gs3.fit(X_train, y_train)
print(gs3.score(X_train, y_train))
print(gs3.score(X_test, y_test))
predict3 = gs3.predict(X_test)
print(errors(y_test, predict3))
gs3.best_params_
coef = gs3.best_estimator_['ridge'].coef_
name = X_train.columns
importance = pd.DataFrame(zip(name, coef), columns=['name', 'coef']).sort_values(by='coef', ascending=False)
importance<jupyter_output><empty_output><jupyter_text>Model 4: Lasso with RFE[(back to top)](Regression-Modeling)<jupyter_code>pipe4 = make_pipeline(KNNImputer(),StandardScaler(), RFE(estimator=LinearRegression()), Lasso())
params = {'rfe__n_features_to_select': [17, 15, 10, 8, 5],
          'lasso__alpha': [.01, .1 ,1 ,10 ,100, 1000]}
gs4 = GridSearchCV(pipe4, params, n_jobs=1)
gs4.fit(X_train, y_train)
print(gs4.score(X_train, y_train))
print(gs4.score(X_test, y_test))
predict4 = gs4.predict(X_test)
print(errors(y_test, predict4))
gs4.best_params_
coef = gs4.best_estimator_['lasso'].coef_
name = X_train.columns
importance = pd.DataFrame(zip(name, coef), columns=['name', 'coef']).sort_values(by='coef', ascending=False)
importance<jupyter_output><empty_output><jupyter_text>Model 5: ElasticNet with RFE[(back to top)](Regression-Modeling)<jupyter_code>pipe5 = make_pipeline(KNNImputer(),StandardScaler(), RFE(estimator=LinearRegression()), ElasticNet())
params = {'rfe__n_features_to_select': [17, 15, 10, 8, 5],
          'elasticnet__alpha': [.01, .1 ,1 ,10 ,100, 1000],
          'elasticnet__l1_ratio': [0, .2, .4 , .6, .8, 1]
         }
gs5 = GridSearchCV(pipe5, params, n_jobs=1)
gs5.fit(X_train, y_train)
print(gs5.score(X_train, y_train))
print(gs5.score(X_test, y_test))
predict5 = gs5.predict(X_test)
print(errors(y_test, predict5))
gs5.best_params_
coef = gs5.best_estimator_['elasticnet'].coef_

name = X_train.columns
importance = pd.DataFrame(zip(name, coef), columns=['name', 'coef']).sort_values(by='coef', ascending=False)
importance<jupyter_output><empty_output><jupyter_text>Model 6: Linear Regression with Limited Features[(back to top)](Regression-Modeling) to reduce mulitcolinearity will take out readings for each pollutant AQI, and low and high temp.<jupyter_code>X = df.drop(columns = ['cumulative_aqi', 'temp_high', 'temp_low', 'co_aqi_val', 'no2_aqi_val', 'pm10_aqi_val', 'pm2.5_aqi_val', 'so2_aqi_val', 'ozone_aqi_val'])
y = df['cumulative_aqi']
X
X_train, X_test, y_train, y_test = train_test_split(X, y)
pipe6 = make_pipeline(KNNImputer(), StandardScaler(), LinearRegression())
pipe6.fit(X_train, y_train)
cross_val_score(pipe6, X_train, y_train)
predict6 = pipe6.predict(X_test)
errors(y_test, predict6)
coef = pipe6.named_steps['linearregression'].coef_

name = X_train.columns

importance = pd.DataFrame(zip(name, coef), columns=['name', 'coef']).sort_values(by='coef', ascending=False)
importance<jupyter_output><empty_output><jupyter_text>Model 7: Elastic Net with Limited Features [(back to top)](Regression-Modeling)<jupyter_code>pipe7 = make_pipeline(KNNImputer(),StandardScaler(), ElasticNet())
params = {'elasticnet__alpha': [.01, .1 ,1 ,10 ,100, 1000],
          'elasticnet__l1_ratio': [0, .2, .4 , .6, .8, 1]
         }
gs7 = GridSearchCV(pipe7, params, n_jobs=1)
gs7.fit(X_train, y_train)
print(gs7.score(X_train, y_train))
print(gs7.score(X_test, y_test))
predict7 = gs7.predict(X_test)
print(errors(y_test, predict7))
gs7.best_params_
gs5.best_estimator_['elasticnet'].coef_<jupyter_output><empty_output><jupyter_text>Model 8: Lasso with Limited Features [(back to top)](Regression-Modeling)<jupyter_code>pipe8 = make_pipeline(KNNImputer(),StandardScaler(), Lasso())
params = {'lasso__alpha': [.01, .1 ,1 ,10 ,100, 1000]}
gs8 = GridSearchCV(pipe8, params, n_jobs=1)
gs8.fit(X_train, y_train)
print(gs8.score(X_train, y_train))
print(gs8.score(X_test, y_test))
predict8 = gs8.predict(X_test)
print(errors(y_test, predict8))
gs8.best_params_
coef = gs8.best_estimator_['lasso'].coef_
name = X_train.columns
importance = pd.DataFrame(zip(name, coef), columns=['name', 'coef']).sort_values(by='coef', ascending=False)
importance<jupyter_output><empty_output><jupyter_text>Model 9: Ridge with Limited Features[(back to top)](Regression-Modeling)<jupyter_code>pipe9 = make_pipeline(KNNImputer(),StandardScaler(), Ridge())
params = {'ridge__alpha': [.01, .1 ,1 ,10 ,100, 1000]}
gs9 = GridSearchCV(pipe9, params, n_jobs=1)
gs9.fit(X_train, y_train)
print(gs9.score(X_train, y_train))
print(gs9.score(X_test, y_test))
predict9 = gs9.predict(X_test)
print(errors(y_test, predict9))
gs9.best_params_
coef = gs9.best_estimator_['ridge'].coef_
name = X_train.columns
importance = pd.DataFrame(zip(name, coef), columns=['name', 'coef']).sort_values(by='coef', ascending=False)
importance<jupyter_output><empty_output><jupyter_text>Model 10: KNN Regressor[(back to top)](Regression-Modeling)<jupyter_code>pipe10 = make_pipeline(KNNImputer(),StandardScaler(), KNeighborsRegressor())
params = {'kneighborsregressor__n_neighbors': [3,5,10]}
gs10 = GridSearchCV(pipe10, params, n_jobs=1)
gs10.fit(X_train, y_train)
print(gs10.score(X_train, y_train))
print(gs10.score(X_test, y_test))
predict10 = gs10.predict(X_test)
print(errors(y_test, predict10))
gs10.best_params_<jupyter_output>0.8575264756941775
0.7709339014101684
root mean sqaured error is 7.445794384388985, r squared is 0.7709339014101684, and mean absolute error is 5.00948905109489<jupyter_text>Model 11: Simple Neural Networks[(back to top)](Regression-Modeling)<jupyter_code>ss = StandardScaler()

X_train_scaled = ss.fit_transform(X_train)
X_test_scaled = ss.transform(X_test)

knn = KNNImputer()

X_train_knn = knn.fit_transform(X_train_scaled)

X_test_knn = knn.transform(X_test_scaled)
model11 = Sequential()
model11.add(Dense(24, activation = 'relu'))
model11.add(Dense(1,))
model11.compile(optimizer = 'adam', loss='mse', metrics = ['mse'])
model11.compile(optimizer = 'adam', loss='mse', metrics = ['mse'])
history11 = model11.fit(
    X_train_knn,
    y_train,
    batch_size = 100,
    validation_data=(X_test_knn, y_test),
    epochs= 100,
)
plt.subplot(1, 2, 1)
plt.title('Loss')
plt.plot(history11.history['loss'], label = 'Training Loss')
plt.plot(history11.history['val_loss'], label = 'Validation Loss')
plt.legend();

plt.subplot(1, 2, 2)
plt.title('MSE')
plt.plot(history11.history['val_mse'], label = 'Validation MSE')
plt.plot(history11.history['mse'], label = 'Training MSE')
plt.legend();
preds11 = model11.predict(X_test_knn)
errors(y_test, preds11)<jupyter_output><empty_output><jupyter_text>Model 12: Best Model: Neural Networks adding Layers and Early Stopping[(back to top)](Regression-Modeling)<jupyter_code>model12 = Sequential()
model12.add(Dense(50, activation = 'relu'))
model12.add(Dense(200, activation = 'relu'))
model12.add(Dense(50, activation = 'relu'))
model12.add(Dense(50, activation = 'relu'))
model12.add(Dense(1,))

model12.compile(optimizer = 'adam', loss='mse', metrics = ['mse'])
early_stop = EarlyStopping(patience = 5) 
history12 = model12.fit(
    X_train_knn,
    y_train,
    batch_size = 100,
    callbacks = [early_stop],
    validation_data=(X_test_knn, y_test),
    epochs= 100
)
plt.subplot(1, 2, 1)
plt.title('Loss')
plt.plot(history12.history['loss'], label = 'Training Loss')
plt.plot(history12.history['val_loss'], label = 'Validation Loss')
plt.legend();

plt.subplot(1, 2, 2)
plt.title('MSE')
plt.plot(history12.history['val_mse'], label = 'Validation MSE')
plt.plot(history12.history['mse'], label = 'Training MSE')
plt.legend();<jupyter_output><empty_output><jupyter_text>from the loss and MSE graphs we can see that the data is not over fit with training and validation<jupyter_code>pred12 = model12.predict(X_test_knn)
errors(y_test, pred12)<jupyter_output><empty_output><jupyter_text>Overall, this was the best performing model with the lowest mean squared error and r2<jupyter_code>plt.subplots(figsize = (10,5))
plt.title('AQI on Test Data vs Predicted Neural Networks AQI', fontsize = 17)
plt.xlabel('Predicted AQI', fontsize = 15)
plt.ylabel('Cumulative AQI', fontsize = 15)
sns.scatterplot(x = y_test, y = pred12[:, 0]);<jupyter_output><empty_output><jupyter_text>Predicted and actual AQI values are aligning well<jupyter_code>plt.subplots(figsize = (10,5))
plt.title('AQI on Test Data vs Predicted Neural Networks AQI', fontsize = 17)
plt.xlabel('Date', fontsize = 15)
plt.ylabel('Cumulative AQI', fontsize = 15)
sns.scatterplot(x = y_test.index, y = y_test, label = 'Test AQI');
sns.scatterplot(x = y_test.index, y = pred12[:, 0], label = 'Predicted AQI');<jupyter_output><empty_output><jupyter_text>Test AQI values in blue align very well with the predicted aqi values displayed in orange from our model  Model 13: Decision Tree Regressor[(back to top)](Regression-Modeling)<jupyter_code>pipe13 = make_pipeline(KNNImputer(),StandardScaler(), DecisionTreeRegressor())

params13 = {
'decisiontreeregressor__max_depth': [2,5,9]}

gs_tree13 = GridSearchCV(pipe13, params13, n_jobs =-1)

gs_tree13.fit(X_train, y_train)
print(gs_tree13.score(X_train, y_train))
print(gs_tree13.score(X_test, y_test))
gs_tree13.best_params_
pred13 = gs_tree13.predict(X_test)
errors(y_test, pred13)<jupyter_output><empty_output><jupyter_text>Decision tree second best model with second best r2 and msme<jupyter_code>importance = gs_tree13.best_estimator_['decisiontreeregressor'].feature_importances_
name = X_train.columns
importance = pd.DataFrame(zip(name, importance), columns=['name', 'importance']).sort_values(by='importance', ascending=False)
importance<jupyter_output><empty_output><jupyter_text>Decision tree important features shows that pm 2.5 mean and ozone max demonstrated as the most important features in determining air quality.<jupyter_code>plt.subplots(figsize = (10,5))
plt.title('AQI on Test Data vs Predicted Decision Tree Regressor AQI', fontsize = 17)
plt.xlabel('Date', fontsize = 15)
plt.ylabel('Cumulative AQI', fontsize = 15)
sns.scatterplot(x = y_test.index, y = y_test);
sns.scatterplot(x = y_test.index, y = pred13);<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup