<jupyter_start><jupyter_text>Linear Regression  Model Representation and Hypothesis FunctionGiven the following assumptions:- Training set contains $m$ observations with $n$ features- Training set of features $X = [(x_1^{(1)}, x_2^{(1)}, \ldots, x_n^{(1)}), \ldots , (x_1^{(m)}, x_2^{(m)}, \ldots, x_n^{(m)}) ]$- Training set of labels $y = [y^{(1)}, \ldots , y^{(m)}]$- Parameter vector $\theta = [\theta_0, \ldots , \theta_n]$Goal is to create a hypothesis function $h_{\theta}(x)$ with parameters (coefficients) $\theta$ in order to accurately calculate the label for a new, unseen $x$. The form of the hypothesis function is:$$h_{\theta}(x) = \theta_0 + \theta_1x_1 + \ldots + \theta_nx_n$$Assuming $x_0 = 1$ for all observations, this is equivalent to:$$h_{\theta}(x) = \displaystyle \sum_{j=0}^{n} \theta_jx_j^{(i)}$$You can add a new first column to $X$ applying the assumption $x_0^{(i)} = 1$ for all observations. This creates a design matrix of dimension $m \times (n+1)$ that's compatible to perform matrix multiplication with the $(n+1) \times 1$ vector $\theta$. The vectorized representation of the hypothesis function (that results in an $m \times 1$ vector with $h_{\theta}(x^{(i)})$ for all observations, $i$ to $m$) is:$$h_{\theta}(x) = X\theta$$  Feature ScalingAdjusting the model's numeric features so they're all on the same scale is important for linear regression, particularly if you use either the gradient descent algorithm to minimize the cost function, or you use regularization.A simple technique is to normalize each feature so it has a mean of zero and standard deviation of 1:$$x = \frac {x - \bar{x}}{s}$$Where $\bar{x}$ is the sample mean of a feature and $s$ is the sample standard deviation.<jupyter_code>def feature_normalize(X):
    '''
    Normalizes the features in X where mean value of each
        feature is 0 and the standard deviation is 1
    X: mxn feature matrix
    Output: returns mxn normalized version of X,
        the 1xn row vector of means for each feature, and
        the 1xn row vector of standard deviations
    '''
    mean = X.mean()
    s = X.std()
    X_norm = (X - mean) / s
    return (X_norm, mean, s)<jupyter_output><empty_output><jupyter_text>Linear Regression Cost FunctionGiven a training set of $m$ features $X$, $m$ labels $Y$, and $n$ parameters $\theta$, want to minimize the following cost function:$$J(\theta_0, \ldots, \theta_n) = J(\theta) = \frac{1}{2m} \displaystyle \sum_{i=1}^{m} (h_{\theta}(x^{(i)}) - y^{(i)})^2$$Vectorized formula:$$J(\theta) = \frac{1}{2m} (X\theta - y)^{T} (X\theta - y)$$<jupyter_code>def calc_cost(X, y, theta):
    '''
    Calculates the cost function J(theta) with given theta parameters
        Note: m = # of training observations, n = # of features
    X: mx(n+1) design matrix of features
    y: mx1 vector of labels
    theta: (n+1)x1 vector of parameters
    Output: returns J(theta), float
    '''
    m = len(y)
    X_theta_less_y = np.dot(X, theta) - y
    J = (1/(2*m)) * np.dot((X_theta_less_y).T, X_theta_less_y)
    return J[0, 0]<jupyter_output><empty_output><jupyter_text>Gradient Descent MethodGradient descent formula:Repeat until convergence {  $\quad \theta_j := \theta_j - \alpha \frac{\partial}{\partial \theta_j}J(\theta) $  }Substituting the partial derivative of the cost function:Repeat {  $ \quad \theta_j := \theta_j - \frac {\alpha}{m} \displaystyle \sum_{i=1}^{m} (h_{\theta}(x^{(i)}) - y^{(i)})x_j^{(i)} $  }Vectorized formula:$\theta := \theta - \frac {\alpha}{m} (X^T (X\theta - y))$Note that $X$ is the design matrix of features, and should be scaled appropriately to help the gradient descent algorithm converge more quickly.<jupyter_code>def gradient_descent(X, y, theta, alpha, num_iters):
    '''
    Gradient descent algorithm to iteratively find the minimum value
        of the cost function J(theta)
        Note: m = # of training observations, n = # of features
    X: mx(n+1) design matrix of scaled features
    y: mx1 vector of labels
    theta: (n+1)x1 vector of parameters
    alpha: the learning rate
    num_iters: number of iterations the algorithm should perfomr
    Output: returns (n+1)x1 vector of optimal theta parameters and
        an array holding the evaluated cost function value (J(theta))
        for each iteration
    '''
    m = len(y)
    J_history = np.zeros((num_iters, 1))
    
    for i in range(num_iters):
        # X' (n+1)xm matrix times h(x)-y mx1 vector -> (n+1)x1 vector
        # Scale vector by alpha/m
        adj = (alpha / m) * np.dot(X.T, (np.dot(X, theta) - y))

        # Simultaneously update thetas
        theta = theta - adj

        # Save the cost J in every iteration
        J_history[i] = calc_cost(X, y, theta)
    
    return (theta, J_history)<jupyter_output><empty_output><jupyter_text>Normal Equation MethodLinear regression has an analytical solution to find the optimal values for $\theta$ without using an iterative algorithm like gradient descent. You apply calculus to find the minima of the function: find the derivative (or partial derivatives with multiple variables), set to zero, and solve.The vectorized formula (where $X$ is the design matrix) for the solution is:$$\theta = (X^TX)^{-1} X^Ty$$<jupyter_code>def normal_equation(X, y):
    '''
    Applies the normal equation method to find optimal values for
        the parameters theta
        Note: m = # of training observations, n = # of features
    X: mx(n+1) design matrix of features (scaling unnecessary)
    y: mx1 vector of labels
    Output: returns (n+1)x1 vector of optimal theta parameters
    '''
    # theta = pinv(X' * X) * X'* y
    inv = np.linalg.pinv(np.dot(X.T, X))
    xTy = np.dot(X.T, y)
    theta = np.dot(inv, xTy)
    return theta<jupyter_output><empty_output><jupyter_text>Linear Regression Application<jupyter_code># Import libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Import Data - Housing Prices
# Source: Coursera Standford Machine Learning course, exercise 1
data = pd.read_csv('ex1data2.txt', header=None)
data.columns = ['SquareFeet', 'NumBedrooms', 'Price']

# Convert data into matrix or array format for linear algebra calculations
X = data[['SquareFeet', 'NumBedrooms']].as_matrix()
y = np.array(data['Price']).reshape(47, 1)
m = len(y)        # Number of training observations
n = X.shape[1]    # Number of features

data.head(10)
# Create design matrices - add new first column of 1's for x_0
x_0 = np.ones((m, 1))

# Gradient descent: normalize features first
X_norm, mean, s = feature_normalize(X)
# print('Mean:\n{}\nStd Dev:\n{}'.format(mean, s))
X_norm_design = np.concatenate((x_0, X_norm), axis=1)

# Normal equation: no need to normalize features
# Create design matrix with non-normalized features
X_not_norm_design = np.concatenate((x_0, X), axis=1)

print('Head of normalized design matrix:')
print(X_norm_design[:5]) 
print('Head of non-normalized design matrix:')
print(X_not_norm_design[:5])
# Find optimal theta parameters

# Gradient descent
num_iters = 1000    # Number of iterations for gradient descent
alpha = 0.01       # Gradient descent learning rate
theta_init = np.zeros((n + 1, 1))
theta_gd, J_history = gradient_descent(X_norm_design, y, theta_init, alpha, num_iters)
cost_gd = calc_cost(X_norm_design, y, theta_gd)

# Normal equation
theta_normal = normal_equation(X_not_norm_design, y)
cost_normal = calc_cost(X_not_norm_design, y, theta_normal)

print('Theta values from gradient descent:\n{}'.format(theta_gd))
print('Theta values from normal equation:\n{}'.format(theta_normal))
print('Final GD cost: {0:.0f}\nFinal NE cost: {1:.0f}'.format(cost_gd, cost_normal))
# Apply linear regression model to predict 1,650 sq-ft, 3 bedroom house
house = np.array([1650, 3])

# Normalize features, add x_0=1 column, apply gradient descent
house_norm = (house - mean) / s
house_norm = np.concatenate((np.ones(1), house_norm))
price_gd = np.dot(house_norm, theta_gd)

# Add x_0=1 column, apply normal equation
house_normal = np.concatenate((np.ones(1), house))
price_normal = np.dot(house_normal, theta_normal)

print('Predicted price of 1,650 sq-ft, 3 bedroom house:')
print('Price from gradient descent: ${0:.0f}'.format(price_gd[0]))
print('Price from normal equation: ${0:.0f}'.format(price_normal[0]))<jupyter_output>Predicted price of 1,650 sq-ft, 3 bedroom house:
Price from gradient descent: $293641
Price from normal equation: $293081
StarcoderdataJupyterStructuredCleanDedup