<jupyter_start><jupyter_text>Module 10 - Regression Algorithms - Decomposing Linear Models Into Their Coefficients and Model Composition In the last few notebooks, I covered the baseline Linear Regression (OLS), Ridge, and LASSO. Linear Regression finds the line of best fit through the least squares loss function, however, is subject to overfitting the training data sets.To smooth out the models, we can use Ridge which tries to "even out" all the coefficients, or use LASSO which tries to reduce model complexity.In the Log-Log models, since all continuous inputs have been standardized, we can compare relative importance out of them, but, also see how they combine into the model.We can also plot the indiviudal coefficient lines and see how they build the model together.If we define a linear space x of intervals, we can calculate what each coefficient would be at that value of x. Then we can graph them to see how the composition works.For model interpretation this is pretty useless, however, it's interesting to see how the models differ in preventing overfitting. *(Placing $x=1$ on all log coefficients probably doesn't correlate into an actual car.)*I'm going to start by importing everything from the notebook *mod10-regression-ridge-lasso-log-log-models.ipynb*.  Part A: Setting Up Log-Log Models for Linear Regression, Ridge, and LASSO<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"

from sklearn import set_config
set_config(display='diagram')
loc = "https://raw.githubusercontent.com/mhall-simon/python/main/data/car-mpg/auto-mpg.data"

df = pd.read_csv(loc, sep="\s+", header=None, na_values="?")

cols = {0:"mpg", 1:"cylinders", 2:"displacement", 3:"horsepower", 4:"weight", 5:"accel", 6:"year", 7:"origin", 8:"model"}
df = df.rename(columns=cols)

df = df.replace(np.nan, df.horsepower.mean())

df = df.drop("model", axis=1)

X = df.iloc[:,1:]
y = df.iloc[:,0]
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=107)
categorical_columns = ['cylinders','origin','year']
numerical_columns = ['displacement','horsepower','weight','accel']

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.preprocessing import StandardScaler

numerical_transformer = Pipeline(steps=[('poly', PolynomialFeatures(degree=3, include_bias=False, interaction_only=True)),
                                        ('scaler', StandardScaler())])
from sklearn.preprocessing import OneHotEncoder

categorical_transformer = OneHotEncoder(handle_unknown='ignore')
from sklearn.compose import ColumnTransformer

preprocessor = ColumnTransformer(
    transformers=[
        ('num', numerical_transformer, numerical_columns),
        ('cat', categorical_transformer, categorical_columns)])
from sklearn.linear_model import LinearRegression
from sklearn.compose import TransformedTargetRegressor
from sklearn.pipeline import make_pipeline

linear = make_pipeline(
    preprocessor,
    TransformedTargetRegressor(
        regressor=LinearRegression(),
        func = np.log,
        inverse_func = np.exp
    )
)

linear.fit(X_train, y_train)
from sklearn.linear_model import Ridge

ridge = make_pipeline(
    preprocessor,
    TransformedTargetRegressor(
        regressor=Ridge(alpha=0.2442053094548655),
        func = np.log,
        inverse_func = np.exp
    )
)

ridge.fit(X_train, y_train)
from sklearn.linear_model import Lasso

lasso = make_pipeline(
    preprocessor,
    TransformedTargetRegressor(
        regressor=Lasso(alpha=0.0001325711365590111, max_iter=100000),
        func = np.log,
        inverse_func = np.exp
    )
)

lasso.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Part B: Extracting Continuous Coefficients Out of Each Model<jupyter_code>numerical_names = linear.named_steps['columntransformer'].named_transformers_['num'].steps[0][1].get_feature_names(input_features=numerical_columns)

categorical_names = linear.named_steps['columntransformer'].named_transformers_['cat'].get_feature_names(input_features=categorical_columns)

feature_names = np.concatenate([numerical_names, categorical_names])
feature_names
coefs = pd.DataFrame(
    linear.named_steps['transformedtargetregressor'].regressor_.coef_,
    columns=['linear'], index=feature_names
)

coefs.head()<jupyter_output><empty_output><jupyter_text>Now, let's add columns for Ridge and LASSO:<jupyter_code>coefs['ridge'] = ridge.named_steps['transformedtargetregressor'].regressor_.coef_
coefs['lasso'] = lasso.named_steps['transformedtargetregressor'].regressor_.coef_
coefs.iloc[0:14,:]
lin = np.linspace(-5,5,100)<jupyter_output><empty_output><jupyter_text>Let's plot our coefficients for the linear regression:The gray lines are the individual components, and the red line is the entire weighted regression line.<jupyter_code># Plot for Linear Regression:
summed = coefs.iloc[0:14,0].sum()

for i in range(14):
    plt.plot(lin, lin*coefs.iloc[i,0], color='grey')

plt.plot(lin, lin*summed, color='red')
plt.ylim([-10,10])
plt.title("Coefficients in Linear Regression Model")
plt.show();<jupyter_output><empty_output><jupyter_text>Key things to take away from this graph is that there's a lot of strongly positive and negative coefficients, that are offsetting each other to get to the red line.A big concern for this is that the patterns found in the training set may have been overfit compared to the true population estimates, making the model less reliable for predictions.Ridge helps us out by providing smoothing to the model:<jupyter_code># Plot for Ridge
summed = coefs.iloc[0:14,1].sum()

for i in range(14):
    plt.plot(lin, lin*coefs.iloc[i,1], color='grey')

plt.plot(lin, lin*summed, color='red')
plt.ylim([-10,10])
plt.show();<jupyter_output><empty_output><jupyter_text>The red predictive line is nearly the same as the one above, however, there's a lot less variation with the coefficients in the model.This smoothing is why Ridge models are preferred, and can prevent overfitting trends found in the training data.Let's compare it to LASSO:<jupyter_code># Plot for LASSO
summed = coefs.iloc[0:14,2].sum()

for i in range(14):
    plt.plot(lin, lin*coefs.iloc[i,2], color='grey')

plt.plot(lin, lin*summed, color='red')
plt.ylim([-10,10])
plt.show();<jupyter_output><empty_output><jupyter_text>If we plot the three lines, we see that they're very similar, however, the Ridge and LASSO models significantly reduced complexity and are less likely to "overfit" testing data.<jupyter_code>summed = coefs.iloc[0:14,0].sum()
plt.plot(lin, lin*summed, label="Linear Regression")

summed = coefs.iloc[0:14,1].sum()
plt.plot(lin, lin*summed, label="Ridge")

summed = coefs.iloc[0:14,2].sum()
plt.plot(lin, lin*summed, label="LASSO")

plt.legend()
plt.show();<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup