<jupyter_start><jupyter_text>Comparison between recommendations of pipeplines\**Update 1/Dec/2020** add top n vs gpbucb, add comparison between with and without kernel normalisation.**Update 27/Nov/2020:** kernel normalisation over all 4138 unique sequences. Adjust parameters [centering_flag, unit_norm_flag] for kernel normalisation. **Update 15/Nov/2020:** recommend based on round 0 + 1 data. This notebook shows the recommendations from different pipelines by scatter plotting the ucb scores/ucb ranks of the union set of top n recommendations from different pipelines.Particularly, this notebook shows the comparison betweeen recommendations from different hyperparameters (alpha, sigma_0).<jupyter_code># direct to proper path
import os
import sys
module_path = os.path.abspath(os.path.join('../..'))
if module_path not in sys.path:
    sys.path.append(module_path)
    
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import itertools
from collections import defaultdict
import math
import json

from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import PairwiseKernel, DotProduct, RBF 
from sklearn.kernel_ridge import KernelRidge
from sklearn.metrics import r2_score, mean_squared_error, make_scorer
from sklearn.model_selection import KFold
from sklearn_extra.cluster import KMedoids

from codes.embedding import Embedding
from codes.environment import Rewards_env
from codes.ucb import GPUCB, Random
from codes.evaluations import evaluate, plot_eva
from codes.regression import *
from codes.kernels_for_GPK import *
from codes.batch_ucb import *
import codes.plot_format as plotFormat

from ipywidgets import IntProgress
from IPython.display import display
import warnings
%matplotlib inline
Path = '../../data/Results_Microplate_partialTrue_normTrue_roundRep_formatSeq_logTrue.csv'

known_df = pd.read_csv(Path)
known_df.head()
run_new_experiment = False


if run_new_experiment:
    for alpha in [2]:
        for sigma_0 in [1]:
            for beta in [2]:
                for centering_flag in [True,False]:
                    for unit_norm_flag in [True,False]:
                        top_n = Top_n_ucb(known_df, kernel_name='WD_Kernel_Shift', l=6, s=1, sigma_0 = sigma_0,
                                        embedding='label', alpha=alpha, rec_size=None, beta=beta,
                                        centering_flag = centering_flag, unit_norm_flag = unit_norm_flag)

                        top_n_rec_df = top_n.run_experiment()

                        # top_n_rec_df['index'] = top_n_rec_df['level_0']
                        # top_n_rec_df = top_n_rec_df.drop(columns = ['level_0'])

                        with pd.ExcelWriter('all_ucb_pred.xlsx', engine="openpyxl", mode='a') as writer:  
                            top_n_rec_df.to_excel(writer, sheet_name=plotFormat.valid_name('train_samples_alpha' + str(alpha) + '_sigma0_' + str(sigma_0) + '_beta_' + str(beta) + '_cen' + str(centering_flag) + '_uninorm' + str(unit_norm_flag)))
type(True)
if run_new_experiment:

    top_n = Top_n_ucb(known_df, kernel_name='RBF', embedding='onehot', alpha=2, rec_size=None, beta=2)

    top_n_rec_df = top_n.run_experiment()
    # top_n_rec_df['index'] = top_n_rec_df['level_0']
    # top_n_rec_df = top_n_rec_df.drop(columns = ['level_0'])

    with pd.ExcelWriter('all_ucb_pred_kernelAdjust.xlsx', engine="openpyxl", mode='a') as writer:  
        top_n_rec_df.to_excel(writer, sheet_name=plotFormat.valid_name('train_samples_RBF_alpha' + str(alpha) + '_beta_' + str(beta)))<jupyter_output><empty_output><jupyter_text>Read prediction data; Plot ucb scoresTake two prediction list from different pipelines, take the union set of top n ucb scores, and scatter plot - the ucb scores for both pipeline predictions. x and y axis represents the predicted ucb scores from each pipeline, the same point represents the same sequence. - the rank of ucb scores (top UCB value is rank 1, the second rank 2, etc.) for both pipeline predictions. The units on the axis should be ranks (the plots will now be backwards, with the best items near the origin).<jupyter_code>def scatterplot_union_ucb_scores(df1, df2, df1_name, df2_name, rec_size = 90):
    """For two rec df from different pipelines,
    take the union set and scatter plot ucb scores.
    
    Input dataframe has at least two columns: index, ucb
    """
    union_rec_index = list(set(df1[:rec_size]['index']).union(set(df2[:rec_size]['index'])))
    print('union set size: ', len(union_rec_index))
    new_df1 = df1.loc[df1['index'].isin(union_rec_index)][['index', 'ucb']]
    new_df2 = df2.loc[df2['index'].isin(union_rec_index)][['index', 'ucb']]
    new_df1['rank'] = new_df1.index + 1
    new_df2['rank'] = new_df2.index + 1
    
    merge_df = new_df1.merge(new_df2, on = 'index')
    merge_df = merge_df.rename(columns = {'ucb_x': 'ucb_' + str(df1_name), 'ucb_y': 'ucb_' + str(df2_name),
                                         'rank_x': 'rank_' + str(df1_name), 'rank_y': 'rank_' + str(df2_name)})
    
    fig,a = plt.subplots(1, 2, figsize = (12,4))
    sns.scatterplot(x = 'ucb_'  + str(df1_name), y = 'ucb_' + str(df2_name), data = merge_df, ax = a[0])
    
    a[0].set_title('ucb scatter')
    sns.scatterplot(x = 'rank_' + str(df1_name), y = 'rank_' + str(df2_name), data = merge_df, ax = a[1])
    a[1].set_title('rank scatter')
    #a[1].set_xlim(-10,300)
    #a[1].set_ylim(-10,300)
sheet_to_df_map = pd.read_excel('all_ucb_pred_kernelAdjust.xlsx', sheet_name=None)
sheet_to_df_map.keys()<jupyter_output><empty_output><jupyter_text>Kernle Normalisation?<jupyter_code>scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[0],list(sheet_to_df_map.values())[1], 
                            list(sheet_to_df_map.keys())[0],list(sheet_to_df_map.keys())[1])

scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[2],list(sheet_to_df_map.values())[3], 
                            list(sheet_to_df_map.keys())[2],list(sheet_to_df_map.keys())[3])

scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[0],list(sheet_to_df_map.values())[2], 
                            list(sheet_to_df_map.keys())[0],list(sheet_to_df_map.keys())[2])

scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[1],list(sheet_to_df_map.values())[3], 
                            list(sheet_to_df_map.keys())[1],list(sheet_to_df_map.keys())[3])<jupyter_output>union set size:  131<jupyter_text>RBF?<jupyter_code>scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[0],list(sheet_to_df_map.values())[-1], 
                            list(sheet_to_df_map.keys())[0],list(sheet_to_df_map.keys())[-1])

scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[1],list(sheet_to_df_map.values())[-1], 
                            list(sheet_to_df_map.keys())[1],list(sheet_to_df_map.keys())[-1])

scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[2],list(sheet_to_df_map.values())[-1], 
                            list(sheet_to_df_map.keys())[2],list(sheet_to_df_map.keys())[-1])

scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[3],list(sheet_to_df_map.values())[-1], 
                            list(sheet_to_df_map.keys())[3],list(sheet_to_df_map.keys())[-1])
pd.set_option('mode.chained_assignment', None) # Turn off warnings

def scatterplot_for_diff_beta(df, name1, name2, rec_size = 90):
    """For the rec df pred mean and std, generating ucb with different beta 
    take the union set and scatter plot ucb scores.
    
    Input dataframe has at least two columns: index, ucb
    """
    df1 = df[['index', 'ucb_' + str(name1)]].copy()
    df1 = df1.sort_values(by = 'ucb_' + str(name1), ascending = False).reset_index()
    df2 = df[['index', 'ucb_' + str(name2)]].copy()
    df2 = df2.sort_values(by = 'ucb_' + str(name2), ascending = False).reset_index()
    
    union_rec_index = list(set(df1[:rec_size]['index']).union(set(df2[:rec_size]['index'])))
    print('union set size: ', len(union_rec_index))
    new_df1 = df1.loc[df1['index'].isin(union_rec_index)]
    new_df2 = df2.loc[df2['index'].isin(union_rec_index)]
    new_df1['rank'] = new_df1.index + 1
    new_df2['rank'] = new_df2.index + 1
    
    merge_df = new_df1.merge(new_df2, on = 'index')
    merge_df = merge_df.rename(columns = {'ucb_x': 'ucb_' + str(name1), 'ucb_y': 'ucb_' + str(name2),
                                         'rank_x': 'rank_' + str(name1), 'rank_y': 'rank_' + str(name2)})
#     print(merge_df)
    fig,a = plt.subplots(1, 2, figsize = (12,4))
    sns.scatterplot(x = 'ucb_'  + str(name1), y = 'ucb_' + str(name2), data = merge_df, ax = a[0])   
    a[0].set_title('ucb scatter ' + str(name1) + ' vs ' + str(name2))
    sns.scatterplot(x = 'rank_' + str(name1), y = 'rank_' + str(name2), data = merge_df, ax = a[1])
    a[1].set_title('rank scatter' + str(name1) + ' vs ' + str(name2))
    #a[1].set_xlim(-10,300)
    #a[1].set_ylim(-10,300)<jupyter_output><empty_output><jupyter_text>Generating 90 recs  Top-n v.s. GPUCB<jupyter_code>sheet_to_df_map = pd.read_excel('batch_ucb.xlsx', sheet_name=None)
scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[0],list(sheet_to_df_map.values())[1], 
                            list(sheet_to_df_map.keys())[0],list(sheet_to_df_map.keys())[1])
scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[2],list(sheet_to_df_map.values())[3], 
                            list(sheet_to_df_map.keys())[2],list(sheet_to_df_map.keys())[3])<jupyter_output>union set size:  92<jupyter_text>With vs without kernel norms<jupyter_code>scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[0],list(sheet_to_df_map.values())[2], 
                            list(sheet_to_df_map.keys())[0],list(sheet_to_df_map.keys())[2])
scatterplot_union_ucb_scores(list(sheet_to_df_map.values())[1],list(sheet_to_df_map.values())[3], 
                            list(sheet_to_df_map.keys())[1],list(sheet_to_df_map.keys())[3])<jupyter_output>union set size:  149
StarcoderdataJupyterStructuredCleanDedup