<jupyter_start><jupyter_text>KNN with ClusteringThe K Nearest Neighbors approach for recommendation usually works by, given a certain user, finding similar ones and analyzing their iteraction history. However, since we do not have a `user_id` in the dataset we used a workaround. With *clusters*, we can find the most itnteracted items for each cluster instead of each user. Whenever we wish to find similar users, the KNN algorithm can provide similar *feature combinations* and we can match those to specific clusters. The iteraction list can then be used to create scores for each item. Therefore, we'll need to save through BentoML the following data:* An **Index Map** to map an item_id into an index (e.g. 1, 2, 7, 45, etc.)* A **Clustering Algorithm** to map new user features to clusters* A **Popularity Matrix** holding the amount of iteractions per cluster per item with shape (clusters, items)* A **KNN Algorithm** to provide the K nearest neighbors of a set of `user_features`In this notebook we will setup these elements. However, the actual recommendation happens in `knn_cluster.py` that will answer to the BentoML api when requested.  Importing Libraries<jupyter_code>import pandas as pd
import numpy as np

from sklearn.cluster import KMeans
from sklearn.neighbors import KNeighborsClassifier

from preprocessing import preprocess, read_sample
from knn_cluster import ClusteredKNN<jupyter_output><empty_output><jupyter_text>Acquire preprocessed Data<jupyter_code>#df = preprocess("Sample")
df = read_sample("/media/backup/datasets/yahoo/yahoo_dataset_clicked.csv", p=1)

df.head(6)
import re
import ast
def literal_eval(element):
    if isinstance(element, str):
        return ast.literal_eval(re.sub('\s+',',',element))
    return element

df['User_Features'] = df['User_Features'].apply(literal_eval)
df['Article_List'] = df['Article_List'].apply(literal_eval)<jupyter_output><empty_output><jupyter_text>ClusteringFor the cluster, we will need the users' features<jupyter_code>users = np.asarray(df.loc[:,'User_Features']) # acquire only the features
users = np.stack(users, axis=0) # stack them to make an array (iteractions, features)
users.shape<jupyter_output><empty_output><jupyter_text>Now we can intialize the clustering algorithm, decide how many clusters we want, and compute<jupyter_code>N_CLUSTERS = 10
kmeans = KMeans(n_clusters = N_CLUSTERS, n_jobs=-1)
kmeans.fit(users)<jupyter_output><empty_output><jupyter_text>K Nearest NeighborsFor the KNN algorithm, we will need the users' features, as well as their clusters, referred here as `labels`<jupyter_code>labels = kmeans.labels_
knn = KNeighborsClassifier(n_jobs=-1)
knn.fit(users, labels)
1+1<jupyter_output><empty_output><jupyter_text>The KNN Algorithm will return a list of indexes of the k closest elementsFirst, we take a sample from the dataset<jupyter_code>sample = df.sample(1).loc[:,'User_Features']
sample_features = np.stack(sample,axis=0)<jupyter_output><empty_output><jupyter_text>Now we can run the KNN with the features sampled<jupyter_code>idxs = knn.kneighbors(X=sample_features, n_neighbors=7, return_distance=False)
idxs<jupyter_output><empty_output><jupyter_text>Index MapFirst, we get all articles in a list<jupyter_code>articles = df['Clicked_Article'].unique()<jupyter_output><empty_output><jupyter_text>Then, we iterate over them creating a dictionary for the index map.<jupyter_code>index_map = {}
idx = 1 # idx starts at 1 so that 0 is used for when the article is not found in the index map
for art in articles:
    index_map[art] = idx
    idx+=1
# index_map<jupyter_output><empty_output><jupyter_text>Popularity MatrixIn order to get our Popularity Matrix, we need to replace the `article_id` by the index and add to which cluster an iteraction belongs to<jupyter_code>df['Clicked_Article'].replace(index_map, inplace=True)
df['Cluster'] = kmeans.labels_
df.head(5)<jupyter_output><empty_output><jupyter_text>Now we can locate the iteractions with clicks and group them by cluster and item.<jupyter_code>popular_by_cluster = df.loc[(df['Click']==1)].groupby(['Cluster','Clicked_Article']).size().sort_values(ascending=False)
popular_by_cluster<jupyter_output><empty_output><jupyter_text>As this returns us with a pandas Series type, we need to create a matrix (numpy array) to hold these values.<jupyter_code>pop_matrix = np.zeros((N_CLUSTERS,len(index_map)+1), dtype='int')

for c in range(N_CLUSTERS):
#     pop_matrix[c][0] = -1
    for i in popular_by_cluster[c].index:
        pop_matrix[c][i] = popular_by_cluster[c][i]

pop_matrix<jupyter_output><empty_output><jupyter_text>Saving ArtifactsNow that we have all our elements to make recommendations, we can save them so our recommender can load them.<jupyter_code>model = ClusteredKNN()
model.pack("index_map", index_map)
model.pack("pop_matrix", pop_matrix)
model.pack("knn", knn)
model.pack("cluster_path", kmeans)<jupyter_output><empty_output><jupyter_text>After packing what our recommender will need, we can test it with a small sample.Notice that, in this case, we must pass the number of neighbors to be used as `model.rank(sample, n_neighbors)`<jupyter_code>test_articles = [565648, 563115, 552077, 564335, 565589, 563938, 560290, 563643, 560620, 565822, 563787, 555528, 565364, 559855, 560518]
model.rank({'Timestamp': 123456789, 'Clicked_Article': 565822, 'Click': 1, 'User_Features': sample_features[0], 'Article_List': np.asarray(test_articles)})<jupyter_output><empty_output><jupyter_text>In order to check wether the recommendation is correct, we can do it ourselvesFirst, we get the neighbors for our feature set<jupyter_code>idxs = knn.kneighbors(X=sample_features, n_neighbors=10, return_distance=False)
idxs<jupyter_output><empty_output><jupyter_text>Then, we get the clusters for each of these neighbors<jupyter_code>knclusters = kmeans.labels_[idxs]
knclusters<jupyter_output><empty_output><jupyter_text>It is likely that most neighbors will belong to the same cluster, but it should not always be the case. Now we can look at our popularity matrix and gather the clicks for each cluster<jupyter_code>clicks = [pop_matrix[c] for c in knclusters]
clicks = np.asarray(clicks[0])
clicks<jupyter_output><empty_output><jupyter_text>It is intended that clusters that appear more than once have their information added multiple times.This is because we will mean over these values. Therefore, the clusters that appear multiple times should be weighted differently.<jupyter_code>mean_scores = np.mean(clicks, axis=0)
mean_scores<jupyter_output><empty_output><jupyter_text>To get the scores specifically about the items in the recommended list, we need to translate the ids into indexes and look at those in the `mean_scores`<jupyter_code>indexes = [index_map[art] for art in test_articles]
scores = [mean_scores[idx] for idx in indexes] 
scores<jupyter_output><empty_output><jupyter_text>Sorting out the values we can check if the model functioned properly<jupyter_code>sorted(zip(scores, test_articles),reverse=True)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup