<jupyter_start><jupyter_text>Extracting BERT embeddings from texts for readability assessment.<jupyter_code>!pip install -U sentence-transformers
from sentence_transformers import SentenceTransformer
from transformers import AutoTokenizer, AutoModel
import torch
import pickle
import numpy<jupyter_output><empty_output><jupyter_text>Read contents from a specific corpus. Corpus should be 1 per line and in .txt format.<jupyter_code>titles = []
contents = []

with open("commoncore.txt","r") as file:
  file_contents = file.readlines()
  for i in file_contents:
    parsed_text = i.split(';',2)
    parsed_text[2] = parsed_text[2].strip()
    print(parsed_text[2])
    titles.append(parsed_text[0].strip())
    contents.append(parsed_text[2])<jupyter_output>The policeman on the beat moved up the avenue impressively. The impressiveness was habitual and not for show, for spectators were few. The time was barely o'clock at night, but chilly gusts of wind with a taste of rain in them had well nigh de-peopled the streets. Trying doors as he went, twirling his club with many intricate and artful movements, turning now and then to cast his watchful eye adown the pacific thoroughfare, the officer, with his stalwart form and slight swagger, made a fine picture of a guardian of the peace. The vicinity was one that kept early hours. Now and then you might see the lights of a cigar store or of an all-night lunch counter but the majority of the doors belonged to business places that had long since been closed. When about midway of a certain block the policeman suddenly slowed his walk. In the doorway of a darkened hardware store a man leaned, with an unlighted cigar in his mouth. As the policeman walked up to him the man spoke up quickly. It's all rig[...]<jupyter_text>Mean Pooling - take attention mask into account for correct averaging.<jupyter_code>def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] #First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, 1)
    sum_mask = torch.clamp(input_mask_expanded.sum(1), min=1e-9)
    return sum_embeddings / sum_mask<jupyter_output><empty_output><jupyter_text>Load BERT models from Huggingface model repository.<jupyter_code># FOR FILIPINO
# tokenizer = AutoTokenizer.from_pretrained("jcblaise/bert-tagalog-base-cased")
# model = AutoModel.from_pretrained("jcblaise/bert-tagalog-base-cased")

# FOR ENGLISH
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
model = AutoModel.from_pretrained("bert-base-cased")<jupyter_output><empty_output><jupyter_text>Preprocess and compute embeddings<jupyter_code>#Tokenize sentences
encoded_input = tokenizer(contents, padding=True, truncation=True, max_length=512, return_tensors='pt')

#Compute token embeddings
with torch.no_grad():
    model_output = model(**encoded_input)

#Perform pooling. In this case, mean pooling
sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])<jupyter_output><empty_output><jupyter_text>Transform embeddings to numpy format. Show example at index 0.<jupyter_code>sentence_embeddings_np = sentence_embeddings.numpy()
sentence_embeddings_np[0]<jupyter_output><empty_output><jupyter_text>Save in csv format. Can be added to csv files of linguistic features for readability assessment.<jupyter_code>numpy.savetxt('commoncore_sbert_embeddings.csv', sentence_embeddings_np, delimiter=',')<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup