<jupyter_start><jupyter_text>Odstraňovanie dát<jupyter_code># dependencies
import sys
import pandas as pd
import matplotlib.pyplot as plt

# local modules
sys.path.append("..")

from _ import constants
# load data
df_train = pd.read_csv(constants.TRAIN,sep=",")<jupyter_output><empty_output><jupyter_text>Atribút user_id  Počet používateľov, ktorí majú len jednu akciu Odstránim používateľov, ktorí majú jednu akciu - čo implicitne znamená aj jednu session. Takíto používatelia sú nevhodní na učenie.<jupyter_code>user_id_count = df_train.user_id.value_counts().reset_index(name="count").rename(columns={'index': 'user_id'})
len(df_train[df_train['user_id'].isin(user_id_count[(user_id_count['count'] == 1)]['user_id'])])
df_train = df_train[~df_train['user_id'].isin(user_id_count[(user_id_count['count'] == 1)]['user_id'])]
user_id_count = df_train.user_id.value_counts().reset_index(name="count").rename(columns={'index': 'user_id'})
user_id_groups = pd.DataFrame({
    'type': [
        'N>1000',
        '1000>N>100',
        '100>N>10',
        '10>N>1',
        'N=1'
    ],
    'count': [
        user_id_count[user_id_count['count'] > 1000].count()[0],
        user_id_count[(user_id_count['count'] > 100) & (user_id_count['count'] < 1000)].count()[0],
        user_id_count[(user_id_count['count'] > 10) & (user_id_count['count'] < 100)].count()[0],
        user_id_count[(user_id_count['count'] > 1) & (user_id_count['count'] < 10)].count()[0],
        user_id_count[user_id_count['count'] == 1].count()[0]
    ]
});

pi,ax = plt.subplots(figsize=[10,6]);
plt.title("Number of occurences (actions) of user in the dataset")
plt.pie(x=user_id_groups['count'], labels=user_id_groups['type'], explode=[0.2]*5, autopct="%.2f%%");
len(df_train.user_id)<jupyter_output><empty_output><jupyter_text>Atribút session<jupyter_code>session_action_count = df_train.groupby(['session_id'])['action_type'].count().reset_index(name="count")<jupyter_output><empty_output><jupyter_text>Odstránenie single-step sessions a excesívnych sessionsV prvom kroku sme síce odstránili používateľov, ktorí majú jednu session a zároveň jednu akciu, avšak my sme neodstránili sessions používateľov, ktoré majú jeden jediný krok v niektorej zo svojich sessions.Takisto by sme mohli vyhodiť tie sessions, ktoré sú nad 100 krokov. Vidíme, že tu su už dáta pomerne pekne distribuované. Na grafe môžme vidieť, že necelá polovica sessions má od 1-10 akcií. Vyše štvrtina je v rozpätií 10-100. Necelá štvrtina sú single-step sessions.<jupyter_code>df_train = df_train[~df_train['session_id'].isin(session_action_count[(session_action_count['count'] == 1) | (session_action_count['count'] > 100)]['session_id'])]
session_action_count = df_train.groupby(['session_id'])['action_type'].count().reset_index(name="count")
session_action_count_groups = pd.DataFrame({
   'type': [
        'N>1000',
        '1000>N>100',
        '100>N>10',
        '10>N>1',
        'N=1'
    ],
    'count': [
        session_action_count[session_action_count['count'] > 1000].count()[0],
        session_action_count[(session_action_count['count'] > 100) & (session_action_count['count'] < 1000)].count()[0],
        session_action_count[(session_action_count['count'] > 10) & (session_action_count['count'] < 100)].count()[0],
        session_action_count[(session_action_count['count'] > 1) & (session_action_count['count'] < 10)].count()[0],
        session_action_count[session_action_count['count'] == 1].count()[0]
    ]
});

pi,ax = plt.subplots(figsize=[10,6]);

plt.pie(x=session_action_count_groups['count'], labels=session_action_count_groups['type'], explode=[0.1]*5, autopct="%.2f%%")
plt.title('N - of actions per session ')
plt.show()
len(df_train)<jupyter_output><empty_output><jupyter_text>Odstránienie duplikovaných párov session_id - step<jupyter_code>df_train = df_train.reset_index().drop_duplicates(subset=['session_id','step'],keep='last').set_index('index')<jupyter_output><empty_output><jupyter_text>Uloženie do súboru<jupyter_code>df_train.to_parquet(constants.DROPPED, index=False)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup