<jupyter_start><jupyter_text>DATA PREPARATION FOR MACHINE LEARNING MODELS
Using Absenteeism at work An UCI dataset  Programming with Python  Import Dataset from Kaggle Install Kaggle for upload dataset into google colab<jupyter_code>!pip install -q kaggle<jupyter_output><empty_output><jupyter_text>Upload Kaggle API key<jupyter_code>from google.colab import files
files.upload()
! mkdir ~/.kaggle
! cp kaggle.json ~/.kaggle/
! chmod 600 ~/.kaggle/kaggle.json<jupyter_output><empty_output><jupyter_text>Download dataset from Kaggle<jupyter_code>! kaggle datasets download -d 'loganalive/absenteeism-at-work-an-uci-dataset/download'  
!ls
!unzip -q absenteeism-at-work-an-uci-dataset.zip
!ls<jupyter_output>absenteeism-at-work-an-uci-dataset.zip	kaggle.json
Absenteeism_at_work.csv			sample_data<jupyter_text>Import Library<jupyter_code>import pandas as pd
import numpy as np
absent = pd.read_csv('Absenteeism_at_work.csv')
absent.head(20)
absent['Work load Average/day ']<jupyter_output><empty_output><jupyter_text>Dimensions of data<jupyter_code>shape = absent.shape
print (shape)<jupyter_output>(740, 21)<jupyter_text>Data type for each attribute<jupyter_code>types = absent.dtypes
print(types)
absent
# Null values in dataset
absent_data = pd.DataFrame(absent.isnull().sum())
absent_data = absent_data.rename(columns={0:"Absent_sum"})
absent_data["Absent Percent"] = (absent_data["Absent_sum"]/len(absent))*100
absent_data
absent.describe()
pd.set_option('display.width', 100)
pd.set_option('precision',3)
description = absent.describe()
print(description)
class_counts = absent.groupby('Absenteeism time in hours').size()
print(class_counts)
correlations = absent.corr(method = 'pearson')
print(correlations)<jupyter_output>ID  ...  Absenteeism time in hours
ID                               1.000e+00  ...                     -0.018
Reason for absence              -6.424e-02  ...                     -0.173
Month of absence                -4.346e-05  ...                      0.024
Day of the week                  3.447e-02  ...                     -0.124
Seasons                          9.849e-02  ...                     -0.006
Transportation expense          -2.242e-01  ...                      0.028
Distance from Residence to Work -4.862e-01  ...                     -0.088
Service time                    -2.727e-01  ...                      0.019
Age                              4.090e-02  ...                      0.066
Work load Average/day            9.246e-02  ...                      0.025
Hit target                       1.879e-02  ...                      0.027
Disciplinary failure             4.502e-03  ...                     -0.124
Education                [...]<jupyter_text>The most common method for calculating correlation is Pearson’s Correlation Coefficient, that assumes a normal distribution of the attributes involved. A correlation of -1 or 1 shows a full negative or positive correlation respectively. Whereas a value of 0 shows no correlation at all.The matrix lists all attributes across the top and down the side, to give correlation between all pairs of attributes (twice, because the matrix is symmetrical). There can show the diagonal line through the matrix from the top left to bottom right corners of the matrix shows perfect correlation of each attribute with itself.<jupyter_code>skew = absent.skew()
print(skew)<jupyter_output>ID                                 0.017
Reason for absence                -0.915
Month of absence                   0.069
Day of the week                    0.102
Seasons                           -0.039
Transportation expense             0.396
Distance from Residence to Work    0.312
Service time                      -0.005
Age                                0.698
Work load Average/day              0.961
Hit target                        -1.262
Disciplinary failure               3.952
Education                          2.109
Son                                1.086
Social drinker                    -0.273
Social smoker                      3.290
Pet                                2.736
Weight                             0.017
Height                             2.566
Body mass index                    0.305
Absenteeism time in hours          5.721
dtype: float64<jupyter_text>The skew result show a positive (right) or negative (left) skew. Values closer to zeroshow less skew.<jupyter_code>def unique(list1): 
    list_set = set(list1)  
    unique_list = (list(list_set)) 
    for x in unique_list: 
        print (x)
abtag=absent['Absenteeism time in hours']
unique(abtag)
abtag2=abtag[absent['Absenteeism time in hours']]
unique(abtag2)
# add categorical target column as per project requirement
absent['Absenteeism categories'] = np.where((absent['Absenteeism time in hours'] >= 0)&(absent['Absenteeism time in hours'] <= 20), "Group 0", 
                                              np.where((absent['Absenteeism time in hours'] >= 21)&(absent['Absenteeism time in hours'] <= 40), "Group 1",
                                                      np.where((absent['Absenteeism time in hours'] >= 41)&(absent['Absenteeism time in hours'] <= 60), "Group 2",
                                                              np.where((absent['Absenteeism time in hours'] >= 61)&(absent['Absenteeism time in hours'] <= 80), "Group 3",
                                                                      np.where((absent['Absenteeism time in hours'] >= 81)&(absent['Absenteeism time in hours'] <= 100), "Group 4",
                                                                                np.where((absent['Absenteeism time in hours'] >= 101)&(absent['Absenteeism time in hours'] <= 120),"Group 5",0))
                                                                              ))))
absent.head(20)
absent['Absenteeism categories'].tail()<jupyter_output><empty_output><jupyter_text>Formatting to proper data type<jupyter_code>absent['followUp_req'] = np.where(absent['Reason for absence']<= 21,1, 0)

absent['Reason for absence'] = absent['Reason for absence'].astype('category')
absent['Month of absence'] = absent['Month of absence'].astype('category')
absent['Day of the week'] = absent['Day of the week'].astype('category')
absent['Seasons'] = absent['Seasons'].astype('category')
absent['Disciplinary failure'] = absent['Disciplinary failure'].astype('category')
absent['Education'] = absent['Education'].astype('category')
absent['Social drinker'] = absent['Social drinker'].astype('category')
absent['Social smoker'] = absent['Social smoker'].astype('category')
absent['Pet'] = absent['Pet'].astype('category')
absent['followUp_req'] = absent['followUp_req'].astype('category')
absent['Absenteeism categories'] = absent['Absenteeism categories'].astype('category')
absent.info()
# store two datasets, one for continous and other categorical
dataset_continuous = absent.drop('Absenteeism categories', axis=1)
dataset_categorical = absent.drop('Absenteeism time in hours',axis=1)

print(dataset_continuous.shape)
print(dataset_categorical.shape)
# write the taining data to file

dataset_continuous.to_csv('cleanDataset_continuousTarget.csv',index=False)
dataset_categorical.to_csv('cleanDataset_categoricalTarget.csv',index=False)
# get the test dataset
test_path = 'Absenteeism_at_work.csv'
data_test = pd.read_csv(test_path, decimal=",")
# preprocess the test dataset
# adding new column named 'followUp_req' based on whether reason for absence required follow up or not
data_test['followUp_req'] = np.where(data_test['Reason for absence'] <= 21, 1, 0)

# add categorical target column as per project requirement

data_test['Absenteeism categories'] = np.where(data_test['Absenteeism time in hours'] == 0, "Group 0", 
                                              np.where(data_test['Absenteeism time in hours'] == 1, "Group 1",
                                                      np.where(data_test['Absenteeism time in hours'] == 2, "Group 2",
                                                              np.where(data_test['Absenteeism time in hours'] == 3, "Group 3",
                                                                      np.where((data_test['Absenteeism time in hours'] >= 4)&(data_test['Absenteeism time in hours'] <= 7), "Group 4",
                                                                               np.where(data_test['Absenteeism time in hours'] == 8, "Group 5",
                                                                                       np.where(data_test['Absenteeism time in hours'] >= 9, "Group 6",0))
                                                                              )))))

data_test['Reason for absence'] = data_test['Reason for absence'].astype('category').cat.codes
data_test['Month of absence'] = data_test['Month of absence'].astype('category').cat.codes
data_test['Day of the week'] = data_test['Day of the week'].astype('category').cat.codes
data_test['Seasons'] = data_test['Seasons'].astype('category').cat.codes
data_test['Disciplinary failure'] = data_test['Disciplinary failure'].astype('category').cat.codes
data_test['Education'] = data_test['Education'].astype('category').cat.codes
data_test['Social drinker'] = data_test['Social drinker'].astype('category').cat.codes
data_test['Social smoker'] = data_test['Social smoker'].astype('category').cat.codes
data_test['Pet'] = data_test['Pet'].astype('category').cat.codes
data_test['followUp_req'] = data_test['followUp_req'].astype('category').cat.codes
data_test['Absenteeism categories'] = data_test['Absenteeism categories'].astype('category').cat.codes
dataset_test = data_test

print(dataset_test.shape)
data_test.head()
dataset_test.to_csv('cleanDataset_categoricalTarget_test.csv',index=False)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup