<jupyter_start><jupyter_text>1. You are provided the titanic dataset. Load the dataset and perform splitting into training and test sets with 70:30 ratio randomly using test train split.2. Use the Logistic regression created from scratch (from the prev question) in this question as well.3. Data cleaning plays a major role in this question. Report all the methods used by you in the ipynb.--> i. Check for missing valuesii. Drop Columns & Handle missing valuesiii. Create dummies for categorical featuresyou are free to perform other data cleaning to improve your results.4. Report accuracy score, Confusion matrix, heat map, classifiaction report and any other metrics you feel useful. dataset link : https://iiitaphyd-my.sharepoint.com/:f:/g/personal/apurva_jadhav_students_iiit_ac_in/Eictt5_qmoxNqezgQQiMWeIBph4sxlfA6jWAJNPnV2SF9Q?e=mQmYN0 (titanic.csv)<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, f1_score,confusion_matrix,r2_score
sns.set(style="darkgrid")
df = pd.read_csv('titanic.csv')
df.head()
print('Missing Values in the columns : \n')
print(df.isnull().sum())
df.describe(include='all')<jupyter_output><empty_output><jupyter_text>Data cleaning1. **Removal** :-- Remove *Name* column as this attribute does not affect the *Survived* status of the passenger. And moreover we can see that each person has a unique name hence there is no point considering this column.- Remove *Ticket* because there are 681 unique values of ticket and moreover if there is some correlation between the ticket and *Survived* status that can be captured by *Fare*.- Remove *Cabin* as there are lot of missing values<jupyter_code>df = df.drop(columns=['Name', 'Ticket', 'Cabin', 'PassengerId'])
s1 = sns.barplot(data = df, y='Survived' ,  hue='Sex' , x='Sex')
s1.set_title('Male-Female Survival')
plt.show()<jupyter_output><empty_output><jupyter_text>Females had a better survival rate than male.<jupyter_code>sns.pairplot(df, hue='Survived')<jupyter_output><empty_output><jupyter_text>Categorical dataFor categorical variables where no ordinal relationship exists, the integer encoding may not be enough, at best, or misleading to the model at worst.Forcing an ordinal relationship via an ordinal encoding and allowing the model to assume a natural ordering between categories may result in poor performance or unexpected results (predictions halfway between categories).In this case, a one-hot encoding can be applied to the ordinal representation. This is where the integer encoded variable is removed and one new binary variable is added for each unique integer value in the variable.  Dummy VariablesThe one-hot encoding creates one binary variable for each category.The problem is that this representation includes redundancy. For example, if we know that [1, 0, 0] represents “blue” and [0, 1, 0] represents “green” we don’t need another binary variable to represent “red“, instead we could use 0 values for both “blue” and “green” alone, e.g. [0, 0].This is called a dummy variable encoding, and always represents C categories with C-1 binary variables.<jupyter_code>from numpy import mean
s1 = sns.barplot(data = df, y='Survived' ,  hue='Embarked' , x='Embarked', estimator=mean)
s1.set_title('Survival vs Boarding place')
plt.show()
carrier_count = df['Embarked'].value_counts()
sns.barplot(x=carrier_count.index, y=carrier_count.values, alpha=0.9)
plt.title('Frequency Distribution of Boarding place')
plt.ylabel('Number of Occurrences', fontsize=12)
plt.xlabel('Places', fontsize=12)
plt.show()
df = pd.get_dummies(df, columns=['Sex', 'Embarked'], prefix=['Sex', 'Embarked'])
df.head()
print('Missing Values in the columns : \n')
print(df.isnull().sum())
df = df.fillna(df['Age'].mean())
print('Missing Values in the columns : \n')
print(df.isnull().sum())
df = df.astype(np.float64)
Y = df['Survived']
Y = np.array(Y)
df.drop(columns=['Survived'], inplace=True)
def standardise(df, col):
    df[col] = (df[col] - df[col].mean())/df[col].std()
    return df

for col in df.columns:
    df = standardise(df, col)
import copy
X = copy.deepcopy(df.to_numpy())
X.shape
x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, shuffle=True)
x_train.shape
class MyLogisticRegression:
    def __init__(self, train_data, Y):
        self.data = train_data  # It is assumed that data is normalized and shuffled (rows, cols)
        self.Y = Y[:, np.newaxis]
        self.b = np.random.randn()
        self.cols = self.data.shape[1]
        self.rows = self.data.shape[0]
        self.weights = np.random.randn(self.cols, 1)  # Initialising weights to 1, shape (cols, 1)
        self.num_iterations = 600
        self.learning_rate = 0.0001
        self.batch_size = 20
        self.errors = []
    
    @staticmethod
    def sigmoid(x):
        return 1/(1 + np.exp(-x))
        
    def calc_mini_batches(self):
        new_data = np.hstack((self.data, self.Y))
        np.random.shuffle(new_data)
    
        rem = self.rows % self.batch_size
        num = self.rows // self.batch_size
        till = self.batch_size * num
        if num > 0:
            dd = np.array(np.vsplit(new_data[ :till, :], num))
            X_batch = dd[:, :, :-1]
            Y_batch = dd[:, :, -1]
            
        return X_batch, Y_batch

    def update_weights(self, X, Y):
        Y_predicted = self.predict(X) # Remember that X has data stored along the row for one sample
        gradient = np.dot(np.transpose(X), Y_predicted - Y)
        self.b = self.b - np.sum(Y_predicted - Y)
        self.weights = self.weights - (self.learning_rate * gradient) # vector subtraction
    
    def print_error(self):
        Y_Predicted = self.predict(self.data)
        class_one = self.Y == 1
        class_two = np.invert(class_one)
        val = np.sum(np.log(Y_Predicted[class_one]))
        val += np.sum(np.log(1 - Y_Predicted[class_two]))
        self.errors.append(-val)
        print(-val)
    
    def gradient_descent(self):
        for j in range(self.num_iterations):
            X, Y = self.calc_mini_batches()
            num_batches = X.shape[0]
            for i in range(num_batches):
                self.update_weights(X[i, :, :], Y[i, :][:, np.newaxis])  # update the weights
            if (j+1)%100 == 0:
                self.print_error()
        plt.plot(self.errors)
        plt.style.use('ggplot')
        plt.xlabel('iteration')
        plt.ylabel('')
        plt.title('Error Vs iteration')
        plt.show()
    
    def predict(self, X):
        # X is 2 dimensional array, samples along the rows
        return self.sigmoid(np.dot(X, self.weights) + self.b)
reg = MyLogisticRegression(x_train, y_train)
reg.gradient_descent()
y_pred = reg.predict(x_test)
pred = y_pred >= 0.5
pred = pred.astype(int)
print('accuracy : {a}'.format(a=accuracy_score(y_test, pred)))
print('f1 score : {a}'.format(a = f1_score(y_test, pred)))
confusion_matrix(y_test, pred)
sns.heatmap(confusion_matrix(y_test, pred))
from sklearn.metrics import classification_report
print(classification_report(y_test, pred))<jupyter_output>precision    recall  f1-score   support

         0.0       0.82      0.86      0.84       153
         1.0       0.80      0.75      0.77       115

    accuracy                           0.81       268
   macro avg       0.81      0.80      0.80       268
weighted avg       0.81      0.81      0.81       268
StarcoderdataJupyterStructuredCleanDedup