<jupyter_start><jupyter_text>Now check the synchronic validity of each model on some benchmark tasks and datasets.<jupyter_code>!wget https://raw.githubusercontent.com/AdrienGuille/DistributionalSemantics/master/evaluation_data/wordsim353.tsv
!wget https://raw.githubusercontent.com/nicholas-leonard/word2vec/master/questions-words.txt
model = eqtime_models[0]
similarities = model.wv.evaluate_word_pairs('wordsim353.tsv', dummy4unknown=False)
analogy_scores = model.wv.evaluate_word_analogies ("questions-words.txt", dummy4unknown=False)
print (similarities[0], similarities[1], analogy_scores[0])
correlations = list ()
analogy_accuracies = list ()
for i in range (1,6):
    similarities = eqtime_models[i].wv.evaluate_word_pairs('wordsim353.tsv', dummy4unknown=True)
    analogy_scores = eqtime_models[i].wv.evaluate_word_analogies ("questions-words.txt", dummy4unknown=True)
    print (i, similarities[0], similarities[1], analogy_scores[0])
    correlations.append (similarities[1].correlation)
    analogy_accuracies.append (analogy_scores[0])
import numpy as np
correlations = [0.4334708888875906, 0.4360834647415774, 0.4371147985338671, 0.43634716424164294, 0.4355623410528807]
analogy_accuracies = [0.19274457634056488, 0.19448424068767908, 0.19535407286123618, 0.19315390912812117, 0.19223291035611953]
print (np.mean (correlations), np.std(correlations))
print (np.mean (analogy_accuracies), np.std(analogy_accuracies))<jupyter_output>0.43571573149151177 0.0012291599239472335
0.19359394187474416 0.0011539627801072797<jupyter_text>Now let's evaluate on the models that have approx. equal number of documents in each of the splits.<jupyter_code>correlations = list ()
analogy_accuracies = list ()
for i, model in enumerate (eqdocs_models):
    similarities = model.wv.evaluate_word_pairs('wordsim353.tsv', dummy4unknown=False)
    analogy_scores = model.wv.evaluate_word_analogies ("questions-words.txt", dummy4unknown=False)
    
    print (i, similarities[0], similarities[1].correlation, analogy_scores[0])
    
    correlations.append (similarities[1].correlation)
    analogy_accuracies.append (analogy_scores[0])
print (np.mean(correlations), np.std (correlations), np.min (correlations), np.max (correlations))
print (np.mean(analogy_accuracies), np.std (analogy_accuracies), np.min (analogy_accuracies), np.max (analogy_accuracies))<jupyter_output>0.520576813438865 0.01784918059365494 0.4879675646695335 0.5526897528764858
0.29838960533884645 0.02115169689860722 0.27043808209727493 0.3473179502547198
StarcoderdataJupyterStructuredCleanDedup