<jupyter_start><jupyter_text>What do we know about vaccines and therapeutics? What has been published concerning research and development and evaluation efforts of vaccines and therapeutics? Prevention for healthcare workers:* Efforts to develop prophylaxis clinical studies and prioritize in healthcare workers Therapeutics:* Effectiveness of drugs being developed and tried to treat COVID-19 patients.* Clinical and bench trials to investigate less common viral inhibitors against COVID-19 such as naproxen, clarithromycin, and minocyclinethat that may exert effects on viral replication.* Capabilities to discover a therapeutic (not vaccine) for the disease, and clinical effectiveness studies to discover therapeutics, to include antiviral agents. Vaccine* Methods evaluating potential complication of Antibody-Dependent Enhancement (ADE) in vaccine recipients.* Exploration of use of best animal models and their predictive value for a human vaccine.* Efforts targeted at a universal coronavirus vaccine.* Approaches to evaluate risk for enhanced disease after vaccination* Assays to evaluate vaccine immune response and process development for vaccines, alongside suitable animal models [in conjunction with therapeutics] Distribution:* Alternative models to aid decision makers in determining how to prioritize and distribute scarce, newly proven therapeutics as production ramps up. This could include identifying approaches for expanding production capacity to ensure equitable and timely distribution to populations in need.  Imports and Constants<jupyter_code>import re
import pandas as pd
import numpy as np
from datetime import datetime
from gensim.test.utils import datapath
from gensim.models import KeyedVectors
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
from gensim.models import Word2Vec
from transformers import AutoTokenizer, AutoModel, AutoModelWithLMHead
import torch
PREV = "Efforts to develop prophylaxis clinical studies and prioritize in healthcare workers"

THERAPEUTIC_1 =  "Effectiveness of drugs being developed and tried to treat COVID-19 patients."
THERAPEUTIC_2 =  "Clinical and bench trials to investigate less common viral inhibitors against COVID-19 such as naproxen, clarithromycin, and minocyclinethat that may exert effects on viral replication."
THERAPEUTIC_3 = "Capabilities to discover a therapeutic for the disease, and clinical effectiveness studies to discover therapeutics, to include antiviral agents."

VACCINE_1 = "Methods evaluating potential complication of Antibody-Dependent Enhancement (ADE) in vaccine recipients."
VACCINE_2 =  "Exploration of use of best animal models and their predictive value for a human vaccine."
VACCINE_3 =  "Efforts targeted at a universal coronavirus vaccine."
VACCINE_4 =  "Approaches to evaluate risk for enhanced disease after vaccination"
VACCINE_5 =  "Assays to evaluate vaccine immune response and process development for vaccines, alongside suitable animal models [in conjunction with therapeutics]"

DISTRIBUTION = "Alternative models to aid decision makers in determining how to prioritize and distribute scarce, newly proven therapeutics as production ramps up. This could include identifying approaches for expanding production capacity to ensure equitable and timely distribution to populations in need."

QUESTIONS = [PREV, THERAPEUTIC_1, THERAPEUTIC_2, THERAPEUTIC_3, 
             VACCINE_1, VACCINE_2, VACCINE_3, VACCINE_4, VACCINE_5, DISTRIBUTION]
FILE_NAME = '../data/metadata_with_ner_19_04.csv'

PUBMED_W2V_30_PATH = './models/PubMed-shuffle-win-2.bin'<jupyter_output><empty_output><jupyter_text>Read Dataset<jupyter_code>df = pd.read_csv(FILE_NAME)
df.head(5)<jupyter_output><empty_output><jupyter_text>Articles representations Comparaison  Load Pretrained W2V model https://www.sciencedirect.com/science/article/pii/S2590177X19300563fn5https://github.com/cambridgeltl/BioNLP-2016    http://bioasq.org/news/bioasq-releases-continuous-space-word-vectors-obtained-applying-word2vec-pubmed-abstracts<jupyter_code># cleaning made to train the model
def clean_text_for_w2v(text):
    bioclean = lambda t: re.sub('[.,?;*!%^&_+():-\[\]{}]', '', t.replace('"', '').replace('/', '').replace('\\', '').replace("'",'').strip().lower()).split()
    return bioclean(text)

def transform_vectors_mean(corpus, model, size):
    result = []

    for sent in corpus:
        s = []
        for w in sent:
            try:
                s.append(model[w])
            except KeyError:
                s.append(np.zeros(size))

        result.append(np.mean(s, axis=0))

    return result
pubmed_w2v_model = KeyedVectors.load_word2vec_format(PUBMED_W2V_30_PATH, binary=True)
pubmed_w2v_model.most_similar('vaccine')
pubmed_w2v_model.most_similar('therapeutics')<jupyter_output><empty_output><jupyter_text>Vizualisation<jupyter_code>COLORS = ['blue', 'red', 'green', 'orange']
TARGET_NAMES = ['preventions', 'therapeutics', 'vaccine', 'distribution']

MASK_P = [True, False, False, False,False, False,False,False,False,False]
MASK_T = [False, True, True, True,False, False,False,False,False,False]
MASK_V = [False, False, False, False,True, True,True,True,True,False]
MASK_D = [False, False, False, False,False, False,False,False,False,True]
questions_cleaned = [clean_text_for_w2v(t) for t in QUESTIONS]
questions_w2v = transform_vectors_mean(questions_cleaned, pubmed_w2v_model, 200)

pca = PCA(n_components=2)
principalComponents_q = pca.fit_transform(questions_w2v)
plt.figure(figsize=(8, 8))
for color, i, target_name in zip(COLORS, [MASK_P, MASK_T, MASK_V, MASK_D], TARGET_NAMES):
    plt.scatter(principalComponents_q[i, 0], principalComponents_q[i, 1],
                color=color, lw=2, label=target_name)

plt.legend(loc="best", shadow=False, scatterpoints=1)

plt.show()<jupyter_output><empty_output><jupyter_text>ClinicalBERT https://towardsdatascience.com/illustrated-self-attention-2d627e33b20ahttps://www.aclweb.org/anthology/W19-1909/https://github.com/EmilyAlsentzer/clinicalBERThttps://github.com/huggingface/transformers/issues/1950<jupyter_code>def get_embeddings_bert(text, model):
    input_ids = torch.tensor(tokenizer.encode(text, max_length=512)).unsqueeze(0)  # Batch size 1
    # 512 max lenght for the model
    outputs = model(input_ids)
    last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple
    sentence_embedding = torch.mean(last_hidden_states[0], dim=0)
    
    return sentence_embedding.tolist()
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
clinical_bert_model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
questions_clinical_bert_emb = [get_embeddings_bert(q, clinical_bert_model) for q in QUESTIONS]
pca = PCA(n_components=2)
principalComponents_q = pca.fit_transform(questions_clinical_bert_emb)
plt.figure(figsize=(8, 8))
for color, i, target_name in zip(COLORS, [MASK_P, MASK_T, MASK_V, MASK_D], TARGET_NAMES):
    plt.scatter(principalComponents_q[i, 0], principalComponents_q[i, 1],
                color=color, lw=2, label=target_name)

plt.legend(loc="best", shadow=False, scatterpoints=1)

plt.show()<jupyter_output><empty_output><jupyter_text>BioBERT https://github.com/dmis-lab/biobert<jupyter_code>tokenizer = AutoTokenizer.from_pretrained("monologg/biobert_v1.0_pubmed_pmc")
bio_bert = AutoModel.from_pretrained("monologg/biobert_v1.0_pubmed_pmc")
questions_bio_bert_emb = [get_embeddings_bert(q, bio_bert) for q in QUESTIONS]
pca = PCA(n_components=2)
principalComponents_q = pca.fit_transform(questions_bio_bert_emb)
plt.figure(figsize=(8, 8))
for color, i, target_name in zip(COLORS, [MASK_P, MASK_T, MASK_V, MASK_D], TARGET_NAMES):
    plt.scatter(principalComponents_q[i, 0], principalComponents_q[i, 1],
                color=color, lw=2, label=target_name)

plt.legend(loc="best", shadow=False, scatterpoints=1)

plt.show()<jupyter_output><empty_output><jupyter_text>SciBert https://github.com/allenai/scibert<jupyter_code>tokenizer = AutoTokenizer.from_pretrained('allenai/scibert_scivocab_cased')
sci_bert_model = AutoModel.from_pretrained('allenai/scibert_scivocab_cased')
questions_sci_bert_emb = [get_embeddings_bert(q, sci_bert_model) for q in QUESTIONS]
pca = PCA(n_components=2)
principalComponents_q = pca.fit_transform(questions_sci_bert_emb)
plt.figure(figsize=(8, 8))
for color, i, target_name in zip(COLORS, [MASK_P, MASK_T, MASK_V, MASK_D], TARGET_NAMES):
    plt.scatter(principalComponents_q[i, 0], principalComponents_q[i, 1],
                color=color, lw=2, label=target_name)

plt.legend(loc="best", shadow=False, scatterpoints=1)

plt.show()<jupyter_output><empty_output><jupyter_text>Bio RoBERTa https://arxiv.org/pdf/1907.11692.pdfhttps://huggingface.co/allenai/biomed_roberta_base<jupyter_code>tokenizer = AutoTokenizer.from_pretrained("allenai/biomed_roberta_base")
biomed_roberta_model = AutoModel.from_pretrained("allenai/biomed_roberta_base")
questions_biomed_roberta_emb = [get_embeddings_bert(q, biomed_roberta_model) for q in QUESTIONS]
pca = PCA(n_components=2)
principalComponents_q = pca.fit_transform(questions_biomed_roberta_emb)
plt.figure(figsize=(8, 8))
for color, i, target_name in zip(COLORS, [MASK_P, MASK_T, MASK_V, MASK_D], TARGET_NAMES):
    plt.scatter(principalComponents_q[i, 0], principalComponents_q[i, 1],
                color=color, lw=2, label=target_name)

plt.legend(loc="best", shadow=False, scatterpoints=1)

plt.show()<jupyter_output><empty_output><jupyter_text>GPT-2https://huggingface.co/mrm8488/GPT-2-finetuned-covid-bio-medrxiv<jupyter_code>tokenizer = AutoTokenizer.from_pretrained("mrm8488/GPT-2-finetuned-covid-bio-medrxiv")
gpt_2 = AutoModelWithLMHead.from_pretrained("mrm8488/GPT-2-finetuned-covid-bio-medrxiv")
questions_biomed_roberta_emb = [get_embeddings_bert(q, gpt_2) for q in QUESTIONS]
pca = PCA(n_components=2)
principalComponents_q = pca.fit_transform(questions_biomed_roberta_emb)
plt.figure(figsize=(8, 8))
for color, i, target_name in zip(COLORS, [MASK_P, MASK_T, MASK_V, MASK_D], TARGET_NAMES):
    plt.scatter(principalComponents_q[i, 0], principalComponents_q[i, 1],
                color=color, lw=2, label=target_name)

plt.legend(loc="best", shadow=False, scatterpoints=1)

plt.show()<jupyter_output><empty_output><jupyter_text>Biomed RoBERTa or GPT-2 seems to be the best representation (hint)  Try to create a W2V with just our articles (abstract) https://machinelearningmastery.com/develop-word-embeddings-python-gensim/<jupyter_code># define training data
sentences = [clean_text_for_w2v(text) for text in df['abstract']]
titles = [clean_text_for_w2v(text) for text in df['title']]
training = sentences + titles
# train model
model = Word2Vec(training, min_count=5, size=200)
print(model)
# summarize vocabulary
words = list(model.wv.vocab)
# save model
model.save('./models/w2v_200_covid_articles_model.bin')
# load model
w2v_covid_article_model = Word2Vec.load('./models/w2v_200_covid_articles_model.bin')
print(w2v_covid_article_model)<jupyter_output>Word2Vec(vocab=44649, size=200, alpha=0.025)
Word2Vec(vocab=44649, size=200, alpha=0.025)<jupyter_text>Have a small look of the word space created<jupyter_code>X = model[model.wv.vocab][50:70]

pca = PCA(n_components=2)
result = pca.fit_transform(X)
plt.scatter(result[:, 0], result[:, 1])
words = list(w2v_covid_article_model.wv.vocab)[50:70]
for i, word in enumerate(words):
    plt.annotate(word, xy=(result[i, 0], result[i, 1]))
plt.show()
w2v_covid_article_model.most_similar('vaccine')
w2v_covid_article_model.most_similar('therapeutics')<jupyter_output>/Users/samuel/anaconda/envs/venv_zencity_ai/lib/python3.6/site-packages/ipykernel/__main__.py:1: DeprecationWarning: Call to deprecated `most_similar` (Method will be removed in 4.0.0, use self.wv.most_similar() instead).
  if __name__ == '__main__':<jupyter_text>Vizualition questions with our model<jupyter_code>questions_cleaned = [clean_text_for_w2v(t) for t in QUESTIONS]
questions_w2v = transform_vectors_mean(questions_cleaned, w2v_covid_article_model, 200)

pca = PCA(n_components=2)
principalComponents_q = pca.fit_transform(questions_w2v)
plt.figure(figsize=(8, 8))
for color, i, target_name in zip(COLORS, [MASK_P, MASK_T, MASK_V, MASK_D], TARGET_NAMES):
    plt.scatter(principalComponents_q[i, 0], principalComponents_q[i, 1],
                color=color, lw=2, label=target_name)

plt.legend(loc="best", shadow=False, scatterpoints=1)
plt.show()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup