<jupyter_start><jupyter_text>Viral Tweets Prediction ChallengeDevelop a machine learning model to predict the virality level of each tweet based on attributes such as tweet content, media attached to the tweet, and date/time published.  Import libraries<jupyter_code>import pandas as pd
import numpy as np

# Visualization
import seaborn as sns
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')

from sklearn import preprocessing
from sklearn.feature_selection import SelectFromModel
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
import lightgbm as lgb
from sklearn.metrics import accuracy_score
from sklearn.metrics import roc_auc_score

import random
import timeit
import math 
import collections 
from datetime import datetime<jupyter_output><empty_output><jupyter_text>Data retrieval<jupyter_code># Local storage path
tweets_path = "Dataset/Tweets/"
users_path = "Dataset/Users/"

# Training datasets
train_tweets = pd.read_csv(tweets_path + 'train_tweets.csv')
train_tweets_vectorized_media = pd.read_csv(tweets_path + 'train_tweets_vectorized_media.csv')
train_tweets_vectorized_text = pd.read_csv(tweets_path + 'train_tweets_vectorized_text.csv')

# Test dataset
test_tweets = pd.read_csv(tweets_path + 'test_tweets.csv')
test_tweets_vectorized_media = pd.read_csv(tweets_path + 'test_tweets_vectorized_media.csv')
test_tweets_vectorized_text = pd.read_csv(tweets_path + 'test_tweets_vectorized_text.csv')

# User dataset
users = pd.read_csv(users_path + 'users.csv')
user_vectorized_descriptions = pd.read_csv(users_path + 'user_vectorized_descriptions.csv')
user_vectorized_profile_images = pd.read_csv(users_path + 'user_vectorized_profile_images.csv')<jupyter_output><empty_output><jupyter_text>Dimensions of Data<jupyter_code># print dimensions of data
print('Dimensions:')
print('Train tweets:', train_tweets.shape)
print('Train tweets vectorized media:', train_tweets_vectorized_media.shape)
print('Train tweets vectorized text:', train_tweets_vectorized_text.shape)
print()

print('Test tweets:', test_tweets.shape)
print('Test tweets vectorized media:', test_tweets_vectorized_media.shape)
print('Test tweets vectorized text:', test_tweets_vectorized_text.shape)
print()

print('Users:', users.shape)
print('User vectorized descriptions:', user_vectorized_descriptions.shape)
print('User vectorized profile images:', user_vectorized_profile_images.shape)<jupyter_output>Dimensions:
Train tweets: (29625, 14)
Train tweets vectorized media: (21010, 2050)
Train tweets vectorized text: (29625, 769)

Test tweets: (12697, 13)
Test tweets vectorized media: (8946, 2050)
Test tweets vectorized text: (12697, 769)

Users: (52, 11)
User vectorized descriptions: (52, 769)
User vectorized profile images: (52, 2049)<jupyter_text>The dimensions show that users in the dataset may have multiple tweets.  Vectorized media has fewer rows than tweets, indicating that not all tweets have media, or that some tweets have multiple media.  Exploratory Data Analysis  Train Tweets<jupyter_code>train_tweets.head()<jupyter_output><empty_output><jupyter_text>Primary Keys: ```tweeet_id```, ```tweet_user_id```. 11 Features. Target variable: ```virality```.  Tweet data are connected thorugh ```tweet_id```.<jupyter_code>train_tweets.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 29625 entries, 0 to 29624
Data columns (total 14 columns):
 #   Column                  Non-Null Count  Dtype  
---  ------                  --------------  -----  
 0   tweet_id                29625 non-null  int64  
 1   tweet_user_id           29625 non-null  int64  
 2   tweet_created_at_year   29625 non-null  int64  
 3   tweet_created_at_month  29625 non-null  int64  
 4   tweet_created_at_day    29625 non-null  int64  
 5   tweet_created_at_hour   29625 non-null  int64  
 6   tweet_hashtag_count     29625 non-null  float64
 7   tweet_url_count         29625 non-null  float64
 8   tweet_mention_count     29625 non-null  float64
 9   tweet_has_attachment    29625 non-null  bool   
 10  tweet_attachment_class  29625 non-null  object 
 11  tweet_language_id       29625 non-null  int64  
 12  tweet_topic_ids         25340 non-null  object 
 13  virality                29625 non-null  int64  
dtypes: bool(1), float64(3), int64(8), obje[...]<jupyter_text>Tweet Creation Datetweet_created_at_year  tweet_created_at_day  tweet_created_at_month  tweet_created_at_hour<jupyter_code>fig, axs = plt.subplots(2, 2, figsize=(12, 8))

sns.histplot(train_tweets, x = 'tweet_created_at_year', discrete = True, ax = axs[0,0])
sns.histplot(train_tweets, x = 'tweet_created_at_day', discrete = True, ax = axs[0,1])
sns.histplot(train_tweets, x = 'tweet_created_at_month', discrete = True, ax = axs[1,0])
sns.histplot(train_tweets, x = 'tweet_created_at_hour', discrete = True, ax = axs[1,1])
plt.show()<jupyter_output><empty_output><jupyter_text>This data was produced during 2021, so the count for ```tweet_created_at_year``` for 2021 does not account for the full year unlike others.- Most tweets were made in the afternoon/evening.- Highest count of tweets during: 2020. December. 27th day of the month. 4pm.  Tweet Message Contenttweet_hashtag_count  tweet_url_count  tweet_mention_count<jupyter_code>fig, axs = plt.subplots(3, 1, figsize=(12, 10))

sns.histplot(x = 'tweet_hashtag_count', data = train_tweets, discrete = True, ax = axs[0])
sns.histplot(x = 'tweet_url_count', data = train_tweets, discrete = True, ax = axs[1])
sns.histplot(x = 'tweet_mention_count', data = train_tweets, discrete = True, ax = axs[2])
plt.show()<jupyter_output><empty_output><jupyter_text>Most tweets have 0 hashtags, 1 url, and 0 mentions.  Tweet Attatchmenttweet_has_attachment   tweet_attachment_class<jupyter_code>fig, axs = plt.subplots(2, 1, figsize=(10, 6))

sns.countplot(x = 'tweet_has_attachment', data = train_tweets, ax = axs[0])
sns.countplot(x = 'tweet_attachment_class', data = train_tweets, ax = axs[1])
plt.show()<jupyter_output><empty_output><jupyter_text>Most tweets have an attatchment.Most attatchments are class A.<jupyter_code>fig, axs = plt.subplots(1, 1, figsize=(8, 3))
sns.countplot(x = 'tweet_language_id', data = train_tweets, ax = axs)
plt.show()<jupyter_output><empty_output><jupyter_text>A high amount of tweets in language_id 0, which is presumed to be english.  Tweet Virality<jupyter_code>sns.countplot(x = 'virality', data = train_tweets)
plt.show()<jupyter_output><empty_output><jupyter_text>For virality, 1 is low whereas 5 is high.  Most tweets have a virality of 1.  Correlation Matrix<jupyter_code>corrmat = train_tweets.corr()[2:] 
sns.heatmap(corrmat, square=True);
df_corr = train_tweets.corr()['virality'][2:-1]
top_features = df_corr.sort_values(ascending=False, key=abs)
top_features<jupyter_output><empty_output><jupyter_text>The heatmap shows that some features have correlation with each otherThe correaltion numbers show a low correlation between virality and features, meaning they cannot be used linearly to predict virality.  Train Tweets Vecotrized Media<jupyter_code>train_tweets_vectorized_media.head()<jupyter_output><empty_output><jupyter_text>Train Tweets Vectorized Text<jupyter_code>train_tweets_vectorized_text.head()<jupyter_output><empty_output><jupyter_text>Each column in Vectorized Text/Media represents one coordinate in the numeric feature space  Users<jupyter_code>users.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 52 entries, 0 to 51
Data columns (total 11 columns):
 #   Column                 Non-Null Count  Dtype
---  ------                 --------------  -----
 0   user_id                52 non-null     int64
 1   user_like_count        52 non-null     int64
 2   user_followers_count   52 non-null     int64
 3   user_following_count   52 non-null     int64
 4   user_listed_on_count   52 non-null     int64
 5   user_has_location      52 non-null     bool 
 6   user_tweet_count       52 non-null     int64
 7   user_has_url           52 non-null     bool 
 8   user_verified          52 non-null     int64
 9   user_created_at_year   52 non-null     int64
 10  user_created_at_month  52 non-null     int64
dtypes: bool(2), int64(9)
memory usage: 3.9 KB<jupyter_text>User data are related thorugh ```user_id```.Users have 10 features.  User Countuser_like_count  user_followers_count  user_following_count  user_listed_on_count  user_tweet_count<jupyter_code>fig, axs = plt.subplots(2, 3, figsize=(18, 8))

sns.histplot(users, x = 'user_like_count', ax = axs[0,0])
sns.histplot(users, x = 'user_followers_count', ax = axs[0,1])
sns.histplot(users, x = 'user_following_count', ax = axs[0,2])
sns.histplot(users, x = 'user_listed_on_count', ax = axs[1,0])
sns.histplot(users, x = 'user_tweet_count', ax = axs[1,1])
axs[1][2].set_visible(False)
plt.show()<jupyter_output><empty_output><jupyter_text>- A large portion of users have between 0-100k followers.- A large portion of users follow between 0 to 100k users.- Most users are listed in 0-5000 lists.- Most users have a tweet count around 500k.  User Hasuser_has_location  user_has_url  user_verified<jupyter_code>fig, axs = plt.subplots(1, 3, figsize=(16, 6))

sns.countplot(x = 'user_has_location', data = users, ax = axs[0])
sns.countplot(x = 'user_has_url', data = users, ax = axs[1])
sns.countplot(x = 'user_verified', data = users, ax = axs[2])
plt.show()<jupyter_output><empty_output><jupyter_text>Most users have their location listed on their account, have a URL in their bio, and most are not verified. For the binary data: most of the users have their location and url listed on their accounts. Most of them are not verified.  User Creation Date<jupyter_code>fig, axs = plt.subplots(2, 1, figsize=(12, 8))

sns.histplot(users, x = 'user_created_at_year', discrete = True, ax = axs[0])
sns.histplot(users, x = 'user_created_at_month', discrete = True, ax = axs[1])
plt.show()<jupyter_output><empty_output><jupyter_text>Most accounts were created in 2011, and in August.  User Vectorized Descriptions<jupyter_code>user_vectorized_descriptions.head()<jupyter_output><empty_output><jupyter_text>User Vectorized Profile Images<jupyter_code>user_vectorized_profile_images.head()<jupyter_output><empty_output><jupyter_text>Data Preprocessing & Wrangling  Missing Data<jupyter_code>train_tweets.isnull().sum()<jupyter_output><empty_output><jupyter_text>Only ```tweet_topic_ids``` have null values. These will treated as another tweet_topic_id by filling them with ```["0"]```.<jupyter_code>train_tweets.fillna({'tweet_topic_ids':"['0']"}, inplace=True)
train_tweets_vectorized_media.isnull().sum().sum()
train_tweets_vectorized_text.isnull().sum().sum()
users.isnull().sum()
user_vectorized_descriptions.isnull().sum().sum()
user_vectorized_profile_images.isnull().sum().sum()<jupyter_output><empty_output><jupyter_text>Categorical Variables  Train Tweets  One-hot encoding<jupyter_code>topic_ids = (
    train_tweets.tweet_topic_ids.str.strip('[]').str.split('\s*,\s*').explode().str.get_dummies().sum(level=0).add_prefix('topic_id_')
) 
topic_ids.rename(columns = lambda x: x.replace("'", ""), inplace=True)
year = pd.get_dummies(train_tweets.tweet_created_at_year, prefix='year')
month = pd.get_dummies(train_tweets.tweet_created_at_month , prefix='month')
day = pd.get_dummies(train_tweets.tweet_created_at_day, prefix='day')
attachment = pd.get_dummies(train_tweets.tweet_attachment_class, prefix='attatchment')
language = pd.get_dummies(train_tweets.tweet_language_id, prefix='language')<jupyter_output><empty_output><jupyter_text>Cyclical Encoding<jupyter_code>hour_sin = np.sin(2 * np.pi * train_tweets['tweet_created_at_hour']/24.0)
hour_sin.name = 'hour_sin'
hour_cos = np.cos(2 * np.pi * train_tweets['tweet_created_at_hour']/24.0)
hour_cos.name = 'hour_cos'
# Join encoded data to train data.
columns_drop = [
                "tweet_topic_ids",
                "tweet_created_at_year",
                "tweet_created_at_month",
                "tweet_created_at_day",
                "tweet_attachment_class",
                "tweet_language_id",
                "tweet_created_at_hour",
               ]
encoded = [topic_ids, year, month, day, attachment, language, hour_sin, hour_cos]

train_tweets_final = train_tweets.drop(columns_drop, 1).join(encoded)
train_tweets_final.head()<jupyter_output><empty_output><jupyter_text>Users  One-hot encoding<jupyter_code>year = pd.get_dummies(users.user_created_at_year, prefix='year')
month = pd.get_dummies(users.user_created_at_month , prefix='month')
# Join encoded data to train data.
columns_drop = [
                "user_created_at_year",
                "user_created_at_month",
               ]
dfs = [year, month]

users_final = users.drop(columns_drop, 1).join(dfs)
users_final.head()<jupyter_output><empty_output><jupyter_text>Normalize Data Machine learning algorithms perform better or converage faster when the features are on a small scale. Let's normalize the counts.  Train Tweets<jupyter_code># Normalize using reprocessing.normalize
#scaled_tweet_hashtag_count = preprocessing.normalize([train_tweets_final["tweet_hashtag_count"]])
#train_tweets_final["tweet_hashtag_count"] = scaled_tweet_hashtag_count[0]

#scaled_tweet_url_count = preprocessing.normalize([train_tweets_final["tweet_url_count"]])
#train_tweets_final["tweet_url_count"] = scaled_tweet_url_count[0]

#scaled_tweet_mention_count = preprocessing.normalize([train_tweets_final["tweet_mention_count"]])
#train_tweets_final["tweet_mention_count"] = scaled_tweet_mention_count[0]
#train_tweets_final.head()<jupyter_output><empty_output><jupyter_text>User<jupyter_code>#users_final["user_like_count"] = preprocessing.normalize([users_final["user_like_count"]])[0]
#users_final["user_followers_count"] = preprocessing.normalize([users_final["user_followers_count"]])[0]
#users_final["user_following_count"] = preprocessing.normalize([users_final["user_following_count"]])[0]
#users_final["user_listed_on_count"] = preprocessing.normalize([users_final["user_listed_on_count"]])[0]
#users_final["user_tweet_count"] = preprocessing.normalize([users_final["user_tweet_count"]])[0]
#users_final.head()<jupyter_output><empty_output><jupyter_text>Feature SelectionUse lasso regression for feature selection.  Train Tweets Media<jupyter_code>print("train_tweets shape:", train_tweets.shape)
print("train_tweets_vectorized_media shape:", train_tweets_vectorized_media.shape)

# Match row number between train tweets and vectorized media
vectorized_media_df = pd.merge(train_tweets, train_tweets_vectorized_media, on='tweet_id', how='right')
# Drop extra columns
vectorized_media_df.drop(train_tweets.columns.difference(['virality']), axis=1, inplace=True)
vectorized_media_df.head()
# Set the target as well as dependent variables from image data.
y = vectorized_media_df['virality']
x = vectorized_media_df.loc[:, vectorized_media_df.columns.str.contains("img_")] 

# Run Lasso regression for feature selection.
sel_model = SelectFromModel(LogisticRegression(C=1, penalty='l1', solver='liblinear'))

# time the model fitting
start = timeit.default_timer()

# Fit the trained model on our data
sel_model.fit(x, y)

stop = timeit.default_timer()
print('Time: ', stop - start) 

# get index of good features
sel_index = sel_model.get_support()

# count the no of columns selected
counter = collections.Counter(sel_model.get_support())
counter
media_ind_df = pd.DataFrame(x[x.columns[(sel_index)]])
train_tweets_media_final = pd.concat([train_tweets_vectorized_media[['media_id', 'tweet_id']], media_ind_df], axis=1)
train_tweets_media_final.head()<jupyter_output><empty_output><jupyter_text>Train Tweets Text<jupyter_code>print("train_tweets shape:", train_tweets.shape)
print("train_tweets_vectorized_text:", train_tweets_vectorized_media.shape)

# Match row number between train tweets and vectorized text
vectorized_text_df = pd.merge(train_tweets, train_tweets_vectorized_text, on='tweet_id', how='right')
# Drop extra columns
vectorized_text_df.drop(train_tweets.columns.difference(['virality']), axis=1, inplace=True)
vectorized_text_df.head()
# Set the target as well as dependent variables from image data.
y = vectorized_text_df['virality']
x = vectorized_text_df.loc[:, train_tweets_vectorized_text.columns.str.contains("feature_")] 

# time the model fitting
start = timeit.default_timer()

# Fit the trained model on our data
sel_model.fit(x, y)

stop = timeit.default_timer()
print('Time: ', stop - start) 

# get index of good features
sel_index = sel_model.get_support()

# count the no of columns selected
counter = collections.Counter(sel_model.get_support())
counter
text_ind_df = pd.DataFrame(x[x.columns[(sel_index)]])
train_tweets_text_final = pd.concat([train_tweets_vectorized_text[['tweet_id']], text_ind_df], axis=1)
train_tweets_text_final.head()<jupyter_output><empty_output><jupyter_text>User Descriptions<jupyter_code># Find the median virality for each user to reduce features
average_virality_df = train_tweets.groupby('tweet_user_id').agg(pd.Series.median)['virality']<jupyter_output><empty_output><jupyter_text>Obtain median of virality since each user may have multiple tweets.<jupyter_code>descriptions_df = pd.merge(average_virality_df, user_vectorized_descriptions, left_on='tweet_user_id', right_on='user_id', how='right')
descriptions_df.head()
# Set the target as well as dependent variables from image data.
y = descriptions_df['virality']
x = descriptions_df.loc[:, descriptions_df.columns.str.contains("feature_")] 

# time the model fitting
start = timeit.default_timer()

# Fit the trained model on our data
sel_model.fit(x, y)

stop = timeit.default_timer()
print('Time: ', stop - start) 

# get index of good features
sel_index = sel_model.get_support()

# count the no of columns selected
counter = collections.Counter(sel_model.get_support())
counter
desc_ind_df = pd.DataFrame(x[x.columns[(sel_index)]])
user_descriptions_final = pd.concat([user_vectorized_descriptions[['user_id']], desc_ind_df], axis=1)
user_descriptions_final.head()<jupyter_output><empty_output><jupyter_text>User Profile Images<jupyter_code>profile_images_df = pd.merge(average_virality_df, user_vectorized_profile_images, left_on='tweet_user_id', right_on='user_id', how='right')
profile_images_df.head()
# Set the target as well as dependent variables from image data.
y = profile_images_df['virality']
x = profile_images_df.loc[:, profile_images_df.columns.str.contains("feature_")] 

# time the model fitting
start = timeit.default_timer()

# Fit the trained model on our data
sel_model.fit(x, y)

stop = timeit.default_timer()
print('Time: ', stop - start) 

# get index of good features
sel_index = sel_model.get_support()

# count the no of columns selected
counter = collections.Counter(sel_model.get_support())
counter
user_prof_ind_df = pd.DataFrame(x[x.columns[(sel_index)]])
user_profile_images_final = pd.concat([user_vectorized_profile_images[['user_id']], user_prof_ind_df], axis=1)
user_profile_images_final.head()
print("Shape:")
print("train_tweets:", train_tweets_final.shape)
print("train_tweets_media:", train_tweets_media_final.shape) # join on tweet id
print("train_tweets_text:", train_tweets_text_final.shape) # join on tweet id
print("")
print("user", users_final.shape) 
print("user_description", user_descriptions_final.shape) # join on user id
print("user_profile", user_profile_images_final.shape) # join on user id
# Group media by tweet_id (since there are multiple media id for a single tweet)
media_df = train_tweets_media_final.groupby('tweet_id').mean()
# tweets_vectorized_text and user_vectorized_profile_images has same column names. 
# rename columns in tweets_vectorized_text
cols = train_tweets_text_final.columns[train_tweets_text_final.columns.str.contains('feature_')]
train_tweets_text_final.rename(columns = dict(zip(cols, 'text_' + cols)), inplace=True)
train_tweets_text_final.head()
# Merge all tables on the column 'user_id' for user data and tweet_id for tweet data

# Join tweets data
tweet_df = pd.merge(media_df, train_tweets_text_final, on = 'tweet_id', how = 'right')
tweet_df.fillna(0, inplace=True)

# Join users data
user_df = pd.merge(users_final, user_profile_images_final, on='user_id')

# Join tweets data on train_tweets
tweet_df_final = pd.merge(train_tweets_final, tweet_df, on = 'tweet_id')

# Join with the users data
final_df = pd.merge(tweet_df_final, user_df, left_on = 'tweet_user_id', right_on='user_id')

final_df.shape<jupyter_output><empty_output><jupyter_text>Preprocessing Test Data  Test Tweets<jupyter_code>test_tweets.isnull().sum()
test_tweets.fillna({'tweet_topic_ids':"['0']"}, inplace=True)
topic_ids = (
    test_tweets['tweet_topic_ids'].str.strip('[]').str.split('\s*,\s*').explode()
    .str.get_dummies().sum(level=0).add_prefix('topic_id_')
) 
topic_ids.rename(columns = lambda x: x.replace("'", ""), inplace=True)
year = pd.get_dummies(test_tweets.tweet_created_at_year, prefix='year')
month = pd.get_dummies(test_tweets.tweet_created_at_month , prefix='month')
day = pd.get_dummies(test_tweets.tweet_created_at_day, prefix='day')
attachment = pd.get_dummies(test_tweets.tweet_attachment_class, prefix='attatchment')
language = pd.get_dummies(test_tweets.tweet_language_id, prefix='language')
hour_sin = np.sin(2*np.pi*test_tweets['tweet_created_at_hour']/24.0)
hour_sin.name = 'hour_sin'
hour_cos = np.cos(2*np.pi*test_tweets['tweet_created_at_hour']/24.0)
hour_cos.name = 'hour_cos'
columns_drop = [
                "tweet_topic_ids",
                "tweet_created_at_year",
                "tweet_created_at_month",
                "tweet_created_at_day",
                "tweet_attachment_class",
                "tweet_language_id",
                "tweet_created_at_hour",
              ]
dfs = [
        topic_ids,
        year,
        month,
        day,
        attachment,
        language,
        hour_sin,
        hour_cos,
      ]

test_tweets_final = test_tweets.drop(columns_drop, 1).join(dfs)
test_tweets_final.head()
# columns missing in train from test
cols_test = set(test_tweets_final.columns) - set(train_tweets_final.columns)
cols_test
for col in cols_test:
    final_df[col] = 0
# columns missing in test from train
cols_train = set(train_tweets_final.columns) - set(test_tweets_final.columns)
cols_train.remove('virality') # remove virality from columns to add to test
cols_train
for col in cols_train:
    test_tweets_final[col] = 0
test_tweets_media_final = pd.concat([test_tweets_vectorized_media[['media_id', 'tweet_id']], media_ind_df], axis=1)
test_tweets_text_final = pd.concat([test_tweets_vectorized_text[['tweet_id']], text_ind_df], axis=1)

media_df = test_tweets_media_final.groupby('tweet_id').mean()

cols = test_tweets_text_final.columns[test_tweets_text_final.columns.str.contains('feature_')]
test_tweets_text_final.rename(columns = dict(zip(cols, 'text_' + cols)), inplace=True)

# Join tweets data
tweet_df = pd.merge(media_df, test_tweets_text_final, on = 'tweet_id', how = 'right')
tweet_df.fillna(0, inplace=True)

# Join users data
user_df = pd.merge(users_final, user_profile_images_final, on='user_id')

# Join tweets data on train_tweets
tweet_df_final = pd.merge(test_tweets_final, tweet_df, on = 'tweet_id')

# Join with user data
p_final_df = pd.merge(tweet_df_final, user_df, left_on = 'tweet_user_id', right_on='user_id')

p_final_df.shape
final_df.shape # Has one more than test because of virality column
final_df.to_csv('base_final_df.csv', index=False)
p_final_df.to_csv('base_p_final_df.csv', index=False)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup