<jupyter_start><jupyter_text>Scopus searches of machine learning algorithms in chemistry literature In this notebook, we search for occurrences of maching learning (ML) algorithm names in literature and visualize the findings. Literature topics are categorized based on the [ACS technical divisions.](https://www.acs.org/content/acs/en/membership-and-networks/join/websites.html) We consider a piece of literature to belong to a division if the title, abstract, or keywords list mention any keyword from the lists written in `acs_divisions.py` These lists contain words found in the titles of symposia for each technical division listed in the "calls for abstracts" for fall and spring ACS national meetings from 2017-2020. We search for literature with the Scopus API via URL requests. To do the Scopus searches in this notebook (which is optional), you need an API key. [You can get one here.](https://dev.elsevier.com/) Once you get a key, add it to the file `config.json`<jupyter_code>import requests
import json
import urllib
import pandas as pd
import numpy as np

from acs_divisions import divisions
from utils import make_df, make_request, get_all_results

# Load configuration, this should contain your API Key
con_file = open("config.json")
config = json.load(con_file)
con_file.close()<jupyter_output><empty_output><jupyter_text>Below are the names of machine learning algorithms that we will search. Some names appear in papers that don't explicitly mention ML, thus we have lists that may contain the algorithm name and the words "machine learning."<jupyter_code>ml_terms = [['naive bayes'],['bayesian network*'],['genetic algorithm*'],['particle swarm'],['decision tree*'],
           ['random forest*'],['neural network*'],['back propagation','machine learning'],['nearest neigh*','machine learning'],['support vector*'],
           ['regression','machine learning'],['gaussian process','kriging']]<jupyter_output><empty_output><jupyter_text>Below is a function to make query strings. This takes a list of keywords, and checks if any or all are found based on the boolean operator provided. A search code is also supplied. The default value is "TITLE-ABS-KEY."<jupyter_code>def make_query_string(keywords,boolop="OR",search_code="TITLE-ABS-KEY"):
    """
    Creates a Scopus query string used to search for ANY/ALL
    keyword(s) from a list of keywords in a document's title, abstract, or 
    keyword list using the search code: "TITLE-ABS-KEY"
    """
    qstring = ''
    for i in range(len(keywords)):
        
        if i != len(keywords) - 1:
            qstring += f'\"{keywords[i]}\"' + f' {boolop} '
        
        else:
            qstring += f'\"{keywords[i]}\"'

    qstring = search_code + "(" + qstring + ")"
    
    return qstring<jupyter_output><empty_output><jupyter_text>Here we're looping through each division and find occurences of each machine learning algorithm. As mentioned above, some algorithm names need to be searched with the term "machine learning" because those names may occur in documents that aren't related to machine learning, ie. "regression." **This step is optional because this cells takes a good amount of time to run. Skip to the cells below the text cell that begins with "! ! ! !" to get the search results.**<jupyter_code>df = pd.DataFrame()
results = {}

for acronym in divisions:
    
    division_dict = {}

    for term in ml_terms:

        keywords = divisions[acronym]['keywords']
    
        plain_query = make_query_string(keywords)
        
        if acronym == 'GEOC':
            
            plain_query += " OR " + make_query_string(['mining','geo*'],"AND")
        
        if term[-1] == 'machine learning':
            
            plain_query += " AND " + make_query_string(term,"AND")
            
        else:
            
            plain_query += " AND " + make_query_string(term)
        
        request = make_request(plain_query, config['apikey'])
        #print(request)
        num_results = request['search-results']['opensearch:totalResults']
        print(acronym, term[0], num_results)
        
        df.loc[acronym,term[0]] = num_results
        
        division_dict.update({term[0]:request})
        
    results.update({acronym:division_dict})
    
        #print(plain_query)
        #print()<jupyter_output>AGFD naive bayes 861
AGFD bayesian network* 2510
AGFD genetic algorithm* 10447
AGFD particle swarm 2129
AGFD decision tree* 7374
AGFD random forest* 5661
AGFD neural network* 25773
AGFD back propagation 134
AGFD nearest neigh* 716
AGFD support vector* 13031
AGFD regression 2136
AGFD gaussian process 2817
AGRO naive bayes 214
AGRO bayesian network* 1285
AGRO genetic algorithm* 3634
AGRO particle swarm 882
AGRO decision tree* 3413
AGRO random forest* 2950
AGRO neural network* 11404
AGRO back propagation 70
AGRO nearest neigh* 199
AGRO support vector* 3874
AGRO regression 1115
AGRO gaussian process 5548
ANYL naive bayes 1044
ANYL bayesian network* 5507
ANYL genetic algorithm* 13583
ANYL particle swarm 4891
ANYL decision tree* 6538
ANYL random forest* 4545
ANYL neural network* 53473
ANYL back propagation 150
ANYL nearest neigh* 879
ANYL support vector* 17199
ANYL regression 2294
ANYL gaussian process 4258
BIOT naive bayes 112
BIOT bayesian network* 662
BIOT genetic algorithm* 1993
BIOT par[...]<jupyter_text>Saving a CSV file with records of the number of results for each search. This will be used for constructing the visualization.<jupyter_code>df.to_csv('search_results.csv')<jupyter_output><empty_output><jupyter_text>Saving the raw search results to a JSON file.<jupyter_code>with open('search_results.json','w') as out:
    json.dump(results,out)<jupyter_output><empty_output><jupyter_text>**! ! ! !** Here you can read the results of the Scopus searches above.<jupyter_code>df_search_results = pd.read_csv('search_results.csv')<jupyter_output><empty_output><jupyter_text>Here, we're looping through every keyword in `acs_divisions.py` and counting how many times each are found in literature. This helps us identify the top keywords that contribute to our final results. **This step is optional because this cells takes a LONG time to run. Skip to the cells below the text cell that begins with "! ! ! !" to get the search results.**<jupyter_code>df = pd.DataFrame()
iloc = -1
for div in divisions:
    for keyword in divisions[div]['keywords']:
        
        div_dict = {'Division':div,
                   'Keyword':keyword}

        num_results_total = 0
        for term in ml_terms:
            
            iloc += 1
            
            plain_query = make_query_string([keyword])
            
            if term[-1] == 'machine learning':
                
                plain_query += " AND " + make_query_string(term,"AND")
                
            else:
                
                plain_query += " AND " + make_query_string(term)
                
            request = make_request(plain_query, config['apikey'])
            num_results = request['search-results']['opensearch:totalResults']
            print(div, keyword, term[0], num_results)
        
            div_dict.update({term[0]:num_results})
            num_results_total += int(num_results)
            
        div_dict.update({'Total':num_results_total})
        df = df.append(div_dict,ignore_index = True)
                
            #print(plain_query)<jupyter_output><empty_output><jupyter_text>Doing a separate search for the number of occurences searching for "mining" AND "geo*"<jupyter_code>mining_dict = {'Division':'GEOC',
              'Keyword':'mining'}

num_results_total = 0
for term in ml_terms:
            
            plain_query = make_query_string(['mining','geo*'],"AND")
            
            if term[-1] == 'machine learning':
                
                plain_query += " AND " + make_query_string(term,"AND")
                
            else:
                
                plain_query += " AND " + make_query_string(term)
                
            request = make_request(plain_query, config['apikey'])
            num_results = request['search-results']['opensearch:totalResults']
            mining_dict.update({term[0]:num_results})
            num_results_total += int(num_results)
            print(term[0], num_results)
                
            #print(plain_query)
            
mining_dict.update({'Total':num_results_total})
df = df.append(mining_dict,ignore_index = True)
df.to_csv('occurences_of_individual_keywords.csv')<jupyter_output><empty_output><jupyter_text>**! ! ! !** Here you can read the results of the searches from the above cells.<jupyter_code>df_keywords = pd.read_csv('occurences_of_individual_keywords.csv').drop(columns='Unnamed: 0')
df_keywords<jupyter_output><empty_output><jupyter_text>Here we find the top five frequently found keywords for each division.<jupyter_code>df_top_five = pd.DataFrame()

for div in divisions:
    
    five = df_keywords.loc[df_keywords['Division'] == div]['Total'].nlargest(5).index
    five_df = df_keywords.iloc[five][['Division','Keyword','Total']]
    df_top_five = pd.concat([df_top_five,five_df])
df_top_five<jupyter_output><empty_output><jupyter_text>Write new CSV with top fives<jupyter_code>df_top_five.to_csv('top_five_keywords.csv')<jupyter_output><empty_output><jupyter_text>List of ML algorithm names to label the heatmap properly.<jupyter_code>ml_terms_formatted = ['Back Propagation + ML','Nearest Neighbors + ML','Naive Bayes','Regression + ML',
                      'Bayesian Networks','Random Forest','Particle Swarm','Gaussian Process/Kriging',
                      'Decision Tree','Support Vectors','Genetic Algorithms','Neural Networks']<jupyter_output><empty_output><jupyter_text>Now we'll read in the data from the Scopus searches and organize the data for our visualization.<jupyter_code>df = pd.read_csv('search_results.csv')<jupyter_output><empty_output><jupyter_text>Rearranging rows so that they are in order of decreasing average number of occurences.<jupyter_code>df = df.reindex(df.mean(axis=1).sort_values().index[::-1], axis=0)
divisions = df['Unnamed: 0'].values
df = df.drop(labels=['Unnamed: 0'], axis=1)<jupyter_output><empty_output><jupyter_text>Rearranging columns so that they are in order of increasing average number of occurrences.<jupyter_code>df = df.reindex(df.mean().sort_values().index, axis=1)<jupyter_output><empty_output><jupyter_text>Extracting the numerical values to a separate array and taking the natural log of the values.<jupyter_code>data = df.values
data = data + 1
data = np.log(data)<jupyter_output><empty_output><jupyter_text>Reading in top key words from individual searches.<jupyter_code>top_key = pd.read_csv('top_five_keywords.csv')
top_key<jupyter_output><empty_output><jupyter_text>Here we'll construct a dataframe that contains the top five keywords in a more organized manner.<jupyter_code>df1 = pd.DataFrame()
for div in divisions:
    
    div_df = top_key.loc[top_key['Division'] == div]['Keyword']
    
    df_dict = {'Division':div}
    
    for i,row in enumerate(div_df):
        
        df_dict.update({str(i+1):[row]})
     
    temp_df = pd.DataFrame(df_dict)
    df1 = pd.concat([df1,temp_df])
df1 = df1.fillna('None')<jupyter_output><empty_output><jupyter_text>Saving to CSV<jupyter_code>df1.to_csv('top_five_keywords_reformatted.csv')<jupyter_output><empty_output><jupyter_text>Constructing a new dataframe that contains percentages with the top keywords<jupyter_code>df3 = pd.read_csv('occurences_of_individual_keywords.csv')
df1['Total All Keywords'] = df1['Division'].apply(lambda x: df3.loc[df3['Division'] == x]['Total'].sum())
df1.head()
df4 = pd.DataFrame()
for div in divisions:
    df_dict = {'Division':div}
    for num in ['1','2','3','4','5']:
        div_dict = df1.loc[df1['Division'] == div]
        kw = div_dict[num].values[0]
        
        if kw != 'None': 
            kw_tot = df3.loc[(df3['Division'] == div) & (df3['Keyword'] == kw)]['Total'].values[0]
            div_tot = div_dict['Total All Keywords'].values[0]
            percentage = np.round(100 * ( kw_tot / div_tot), 1)
            df_dict.update({num:kw+f' ({percentage}%)'})
            
        else:
            df_dict.update({num:'None'})
            
    df4 = df4.append(df_dict, ignore_index=True)
df4 = df4[['Division','1','2','3','4','5']]
df4.head()
df4.to_csv('top_five_keywords_with_percentages.csv')<jupyter_output><empty_output><jupyter_text>Finally, we can visualize the results in a heatmap below.<jupyter_code>from matplotlib import pyplot as plt
from mpl_toolkits.axes_grid1 import make_axes_locatable

fig, ax = plt.subplots(figsize=[20,10])
im = ax.imshow(data, cmap='hot_r')
#im = ax.pcolor(data, cmap='hot_r', edgecolors='k', linewidths=0.5)

ax.set_xticks(np.arange(len(ml_terms_formatted)))
ax.set_yticks(np.arange(len(divisions)))

ax.set_xticklabels(ml_terms_formatted, fontweight = 'bold', fontsize = 12)
ax.set_yticklabels(divisions, fontweight = 'bold', fontsize = 12)

plt.setp(ax.get_xticklabels(), rotation=45, ha="right",
         rotation_mode="anchor")

divider = make_axes_locatable(ax)
cax = divider.append_axes("right", size="10%", pad=0.25)

cb = plt.colorbar(im, cax=cax)
cb.set_label(label='ln(No. of Occurrences)',weight='bold',size=12)

for i in range(len(divisions)):
    for j in range(len(ml_terms_formatted)):
        if data[i, j] == 0:
            text = ax.text(j, i, 'X',ha="center", va="center", color="k", size=18)

#table = plt.table(cellText=text_data,loc='upper center',colWidths=[5]*5,rowLabels=divisions)
            
plt.show()
plt.tight_layout()
#plt.savefig('heatmap.png',format='png',dpi=600)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup