<jupyter_start><jupyter_text>Dataset Source:About this fileBoston House Price dataset  columns:* CRIM     per capita crime rate by town* ZN       proportion of residential land zoned for lots over 25,000 sq.ft.* INDUS    proportion of non-retail business acres per town* CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)* NOX      nitric oxides concentration (parts per 10 million)* RM       average number of rooms per dwelling* AGE      proportion of owner-occupied units built prior to 1940* DIS      weighted distances to five Boston employment centres* RAD      index of accessibility to radial highways* TAX      full-value property-tax rate per 10,000* PTRATIO  pupil-teacher ratio by town* B         where Bk is the proportion of blacks by town* LSTAT    percentage lower status of the population* MEDV     Median value of owner-occupied homes in 1000$  Load Modules<jupyter_code>import numpy as np                  # linear algebra python library
import pandas as pd                 # data structure for tabular data.
import matplotlib.pyplot as plt     # visualization library

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Loading data<jupyter_code>filename = "housing.csv"
boston_data = pd.read_csv(filename, delim_whitespace=True, header=None)
header = ["CRIM","ZN","INDUS","CHAS","NOX","RM",
          "AGE","DIS","RAD","TAX","PTRATIO","B","LSTAT","MEDV"]

boston_data.columns = header
# display the first 10 rows of dataframe.
boston_data.head(10)<jupyter_output><empty_output><jupyter_text>Inspecting variable types<jupyter_code>boston_data.dtypes<jupyter_output><empty_output><jupyter_text>In many datasets, integer variables are cast as float. So, after inspectingthe data type of the variable, even if you get float as output, go aheadand check the unique values to make sure that those variables are discreteand not continuous.  Inspecting all variables inspecting distinct values of `RAD`(index of accessibility to radial highways).<jupyter_code>boston_data['RAD'].unique()<jupyter_output><empty_output><jupyter_text>inspecting distinct values of `CHAS` Charles River dummy variable (= 1 if tract bounds river; 0 otherwise).<jupyter_code>boston_data['CHAS'].unique()<jupyter_output><empty_output><jupyter_text>inspecting the first 20 distinct values of all continous variables as following:* CRIM     per capita crime rate by town* ZN       proportion of residential land zoned for lots over 25,000 sq.ft.* INDUS    proportion of non-retail business acres per town* NOX      nitric oxides concentration (parts per 10 million)* RM       average number of rooms per dwelling* AGE      proportion of owner-occupied units built prior to 1940* DIS      weighted distances to five Boston employment centres* TAX      full-value property-tax rate per 10,000* PTRATIO  pupil-teacher ratio by town* B         where Bk is the proportion of blacks by town* LSTAT    percentage lower status of the population* MEDV     Median value of owner-occupied homes in 1000$ CRIM per capita crime rate by town.<jupyter_code>boston_data['CRIM'].unique()[0:20]<jupyter_output><empty_output><jupyter_text>ZN proportion of residential land zoned for lots over 25,000 sq.ft.<jupyter_code>boston_data['ZN'].unique()[0:20]<jupyter_output><empty_output><jupyter_text>INDUS proportion of non-retail business acres per town<jupyter_code>boston_data['INDUS'].unique()[0:20]<jupyter_output><empty_output><jupyter_text>NOX nitric oxides concentration (parts per 10 million)<jupyter_code>boston_data['NOX'].unique()[0:20]<jupyter_output><empty_output><jupyter_text>RM average number of rooms per dwelling<jupyter_code>boston_data['RM'].unique()[0:20]<jupyter_output><empty_output><jupyter_text>AGE proportion of owner-occupied units built prior to 1940<jupyter_code>boston_data['AGE'].unique()[0:20]<jupyter_output><empty_output><jupyter_text>DIS weighted distances to five Boston employment centres<jupyter_code>boston_data['DIS'].unique()[0:20]<jupyter_output><empty_output><jupyter_text>TAX full-value property-tax rate per 10,000<jupyter_code>boston_data['TAX'].unique()[0:20]<jupyter_output><empty_output><jupyter_text>PTRATIO pupil-teacher ratio by town<jupyter_code>boston_data['PTRATIO'].unique()<jupyter_output><empty_output><jupyter_text>B where Bk is the proportion of blacks by town<jupyter_code>boston_data['B'].unique()[0:20]<jupyter_output><empty_output><jupyter_text>LSTAT percentage lower status of the population<jupyter_code>boston_data['LSTAT'].unique()[0:20]<jupyter_output><empty_output><jupyter_text>MEDV Median value of owner-occupied homes in 1000$<jupyter_code>boston_data['MEDV'].unique()[0:20]<jupyter_output><empty_output><jupyter_text>after we checked the dat type of each variable. we have 2 discrete numerical variable and 10 floating or continuous variales.  To understand wheather a variable is contious or discrete. we can also make a histogram for each:* CRIM     per capita crime rate by town* ZN       proportion of residential land zoned for lots over 25,000 sq.ft.* INDUS    proportion of non-retail business acres per town* CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)* NOX      nitric oxides concentration (parts per 10 million)* RM       average number of rooms per dwelling* AGE      proportion of owner-occupied units built prior to 1940* DIS      weighted distances to five Boston employment centres* RAD      index of accessibility to radial highways* TAX      full-value property-tax rate per 10,000* PTRATIO  pupil-teacher ratio by town* B         where Bk is the proportion of blacks by town* LSTAT    percentage lower status of the population* MEDV     Median value of owner-occupied homes in 1000$ making histogram for crime rate by town `CRIM` vatiable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['CRIM'])
bins = int(np.sqrt(n_data))
boston_data['CRIM'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>making histogram for  proportion of residential land zoned for lots over 25,000 sq.ft `ZN`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['ZN'])
bins = int(np.sqrt(n_data))
boston_data['ZN'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>making histogram for proportion of non-retail business acres per town `INDUS`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['INDUS'])
bins = int(np.sqrt(n_data))
boston_data['INDUS'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>making histogram for nitric oxides concentration (parts per 10 million) `NOX`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['NOX'])
bins = int(np.sqrt(n_data))
boston_data['NOX'].hist(bins=bins)<jupyter_output>22<jupyter_text>making histogram for average number of rooms per dwelling `RM`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['RM'])
bins = int(np.sqrt(n_data))
boston_data['RM'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>making histogram for proportion of owner-occupied units built prior to 1940 `AGE`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['AGE'])
bins = int(np.sqrt(n_data))
boston_data['AGE'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>making histogram for weighted distances to five Boston employment centres `DIS`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['DIS'])
bins = int(np.sqrt(n_data))
boston_data['DIS'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>making histogram for full-value property-tax rate per 10,000 `TAX`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['TAX'])
bins = int(np.sqrt(n_data))
boston_data['TAX'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>making histogram for pupil-teacher ratio by town `PTRATIO`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['PTRATIO'])
bins = int(np.sqrt(n_data))
boston_data['PTRATIO'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>making histogram  where Bk is the proportion of blacks by town `B`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['B'])
bins = int(np.sqrt(n_data))
boston_data['B'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>making histogram for percentage lower status of the population `LSTAT `, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['LSTAT'])
bins = int(np.sqrt(n_data))
boston_data['LSTAT'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>making histogram for Median value of owner-occupied homes in 1000$ ` MEDV`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['MEDV'])
bins = int(np.sqrt(n_data))
boston_data['MEDV'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>making histogram for index of accessibility to radial highways`RAD`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['RAD'])
bins = int(np.sqrt(n_data))
boston_data['RAD'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>by taking a look to histogram of features we noticing that the continuous variables values range is not discrete. making histogram for Charles River dummy variable (= 1 if tract bounds river; 0 otherwise) ` CHAS`, variable by dividing the variable range into intervals.<jupyter_code>n_data = len(boston_data['CHAS'])
bins = int(np.sqrt(n_data))
boston_data['CHAS'].hist(bins=bins)<jupyter_output><empty_output><jupyter_text>we noticing here the values of this variable is discrete.  Quantifying Missing Datacalculating the missing values in the dataset.<jupyter_code>boston_data.isnull().sum()<jupyter_output><empty_output><jupyter_text>There is no Missing Values   Determining the cardinality in cateogrical varaibles find unique values in each categorical variable<jupyter_code>boston_data.nunique()<jupyter_output><empty_output><jupyter_text>The nunique() method ignores missing values by default. If we want toconsider missing values as an additional category, we should set the    dropna argument to False: data.nunique(dropna=False). let's print out the unique category in Charles River dummy variable (= 1 if tract bounds river; 0 otherwise) ` CHAS`<jupyter_code>boston_data['CHAS'].unique()<jupyter_output><empty_output><jupyter_text>pandas nunique() can be used in the entire dataframe. pandas    unique(), on the other hand, works only on a pandas Series. Thus, weneed to specify the column name that we want to return the unique valuesfor.<jupyter_code>boston_data[['CHAS','RAD']].nunique().plot.bar(figsize=(12,6))
plt.xlabel("Variables")
plt.ylabel("Number Of Unique Values")
plt.title("Cardinality")
plt.show()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup