<jupyter_start><jupyter_text>ELMO/ARAVEC/FASTTEXT Baseline for OSACT4 - Task A In this notebook, we will walk you through the process of reproducing the ELMO/ARAVEC/FASTTEXT baseline for the OSACT4 task A.  Loading Required Modules We start by loading the needed python libraries.<jupyter_code>import os
import numpy as np
import tensorflow as tf
from tensorflow import keras
import pandas as pd
from sklearn.metrics import f1_score
from sklearn.preprocessing import LabelEncoder
from embed_classer import embed<jupyter_output><empty_output><jupyter_text>Loading Data Using pandas, we can load and inspect the training and testing datasets as follows:<jupyter_code>df_train = pd.read_csv("../../data/osact4/OSACT2020-sharedTask-train.txt", sep="\t", quotechar='▁', header=None, names=["text", "offensive", "hate"])
df_dev = pd.read_csv("../../data/osact4/OSACT2020-sharedTask-dev.txt", sep="\t", quotechar='▁', header=None, names=["text", "offensive", "hate"])
df_test = pd.read_csv("../../private_datasets/offensive/tweets_v1.0.txt", sep="\t", quotechar='▁', header=None, names=["text"])<jupyter_output>/code/alue/alue/env/lib/python3.7/site-packages/ipykernel_launcher.py:1: ParserWarning: Falling back to the 'python' engine because ord(quotechar) > 127, meaning the quotechar is larger than one byte, and the 'c' engine does not support such quotechars; you can avoid this warning by specifying engine='python'.
  """Entry point for launching an IPython kernel.
/code/alue/alue/env/lib/python3.7/site-packages/ipykernel_launcher.py:2: ParserWarning: Falling back to the 'python' engine because ord(quotechar) > 127, meaning the quotechar is larger than one byte, and the 'c' engine does not support such quotechars; you can avoid this warning by specifying engine='python'.
  
/code/alue/alue/env/lib/python3.7/site-packages/ipykernel_launcher.py:3: ParserWarning: Falling back to the 'python' engine because ord(quotechar) > 127, meaning the quotechar is larger than one byte, and the 'c' engine does not support such quotechars; you can avoid this warning by specifying engine='python'.
  This is s[...]<jupyter_text>Below we list the 5 first entries in the training data.<jupyter_code>df_train.head()<jupyter_output><empty_output><jupyter_text>Below we list the 5 first entries in the development data.<jupyter_code>df_dev.head()<jupyter_output><empty_output><jupyter_text>Below we list the 5 first entries in the testing data.<jupyter_code>df_test.head()<jupyter_output><empty_output><jupyter_text>Model Preparation We start by setting the randomisation seed and the maximum sentence length:<jupyter_code>tf.random.set_seed(123)
max_sentence_len = 20
model_type = "fasttext"

if model_type == "aravec":
    model_path = '../pretrained/full_uni_sg_300_twitter.mdl'
    size = 300
elif model_type == "fasttext":
    model_path = '../pretrained/cc.ar.300.bin'
    size = 300
elif model_type == "elmo":
    model_path= '../pretrained'
    size = 1024<jupyter_output><empty_output><jupyter_text>Next we load our model of choice:<jupyter_code>embedder = embed(model_type, model_path)<jupyter_output>/code/alue/alue/env/lib/python3.7/site-packages/torch/cuda/__init__.py:52: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 10010). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at  /pytorch/c10/cuda/CUDAFunctions.cpp:109.)
  return torch._C._cuda_getDeviceCount() > 0
2021-03-22 15:27:13,148 INFO: char embedding size: 3773
2021-03-22 15:27:14,241 INFO: word embedding size: 221272
2021-03-22 15:27:17,920 INFO: Model(
  (token_embedder): ConvTokenEmbedder(
    (word_emb_layer): EmbeddingLayer(
      (embedding): Embedding(221272, 100, padding_idx=3)
    )
    (char_emb_layer): EmbeddingLayer(
      (embedding): Embedding(3773, 50, padding_idx=3770)
    )
    (convolutions): ModuleList(
      (0): Conv1d(50, 32,[...]<jupyter_text>Then we define the input and output to the model:<jupyter_code>sentence = keras.Input(shape=(max_sentence_len, size), name='sentence')
label = keras.Input(shape=(1,), name='label')<jupyter_output><empty_output><jupyter_text>This is followed by defining the structure of the network:<jupyter_code>forward_layer = tf.keras.layers.LSTM(size)
backward_layer = tf.keras.layers.LSTM(size, go_backwards=True)
masking_layer = tf.keras.layers.Masking()
rnn = tf.keras.layers.Bidirectional(forward_layer, backward_layer=backward_layer)
logits = rnn(sentence)
logits = keras.layers.Dense(1, activation=tf.nn.sigmoid)(logits)<jupyter_output><empty_output><jupyter_text>Then we construct and compile the model:<jupyter_code>model = keras.Model(inputs=sentence, outputs=logits)
model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>Model Training First we perpare the inputs and outputs to be fed to the model during training:<jupyter_code>X_train = embedder.embed_batch(df_train["text"].tolist(), max_sentence_len)
X_dev = embedder.embed_batch(df_dev["text"].tolist(), max_sentence_len)
le = LabelEncoder()
le.fit(df_train["offensive"])
Y_train = le.transform(df_train["offensive"])
Y_dev = le.transform(df_dev["offensive"])<jupyter_output>2021-03-22 15:27:36,134 INFO: 110 batches, avg len: 21.3
2021-03-22 15:27:59,126 INFO: Finished 1000 sentences.
2021-03-22 15:28:22,459 INFO: Finished 2000 sentences.
2021-03-22 15:28:42,378 INFO: Finished 3000 sentences.
2021-03-22 15:29:01,707 INFO: Finished 4000 sentences.
2021-03-22 15:29:21,189 INFO: Finished 5000 sentences.
2021-03-22 15:29:49,623 INFO: Finished 6000 sentences.
2021-03-22 15:30:13,424 INFO: Finished 7000 sentences.
2021-03-22 15:30:41,732 INFO: 16 batches, avg len: 21.6
2021-03-22 15:31:15,202 INFO: Finished 1000 sentences.<jupyter_text>Next we fit the data:<jupyter_code>model.fit(X_train, Y_train, epochs=5, batch_size=32, validation_data=(X_dev, Y_dev))<jupyter_output>Epoch 1/5
219/219 [==============================] - 286s 1s/step - loss: 0.5030 - accuracy: 0.7887 - val_loss: 0.3262 - val_accuracy: 0.8660
Epoch 2/5
219/219 [==============================] - 275s 1s/step - loss: 0.3106 - accuracy: 0.8626 - val_loss: 0.3534 - val_accuracy: 0.8570
Epoch 3/5
219/219 [==============================] - 272s 1s/step - loss: 0.2800 - accuracy: 0.8831 - val_loss: 0.2752 - val_accuracy: 0.8860
Epoch 4/5
219/219 [==============================] - 271s 1s/step - loss: 0.2033 - accuracy: 0.9178 - val_loss: 0.2701 - val_accuracy: 0.8870
Epoch 5/5
219/219 [==============================] - 274s 1s/step - loss: 0.1314 - accuracy: 0.9540 - val_loss: 0.3358 - val_accuracy: 0.8740<jupyter_text>Submission Preperation We perpare the features for each testset instance as follows:<jupyter_code>X_test = embedder.embed_batch(df_test["text"].tolist(), max_sentence_len)<jupyter_output>2021-03-22 15:55:06,228 INFO: 16 batches, avg len: 16.2
2021-03-22 15:55:27,962 INFO: Finished 1000 sentences.<jupyter_text>We predict and evaluate the prediction as follows:<jupyter_code>predictions = (model.predict(X_test)>0.5).astype(int)<jupyter_output><empty_output><jupyter_text>We perpare the predictions as a pandas dataframe.<jupyter_code>df_preds = pd.DataFrame(data=le.inverse_transform(predictions), columns=["prediction"])
if not os.path.exists("./predictions/{}".format(model_type)):
    os.makedirs("./predictions/{}".format(model_type), exist_ok=True)
df_preds.to_csv("./predictions/{}/offensive.tsv".format(model_type), index=False, header=False, sep="\t")<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup