<jupyter_start><jupyter_text>Getting started with Word2Vec in Gensim and making it work!The idea behind Word2Vec is pretty simple. We are making and assumption that you can tell the meaning of a word by the company it keeps. This is analogous to the saying *show me your friends, and I'll tell who you are*. So if you have two words that have very similar neighbors (i.e. the usage context is about the same), then these words are probably quite similar in meaning or are at least highly related. For example, the words `shocked`,`appalled` and `astonished` are typically used in a similar context. In this tutorial, you will learn how to use the Gensim implementation of Word2Vec and actually get it to work! I have heard a lot of complaints about poor performance etc, but its really a combination of two things, (1) your input data and (2) your parameter settings. Note that the training algorithms in this package were ported from the [original Word2Vec implementation by Google](https://arxiv.org/pdf/1301.3781.pdf) and extended with additional functionality.<jupyter_code>!pip install gensim<jupyter_output>Collecting gensim
[?25l  Downloading https://files.pythonhosted.org/packages/c2/db/677f0c8a1c49b44e7a999c2fdbcba576017c10d3d77d11c29ee3fa1b291e/gensim-3.8.1-cp35-cp35m-manylinux1_x86_64.whl (24.2MB)
[K     |████████████████████████████████| 24.2MB 8.0kB/s  eta 0:00:01   |████████████▋                   | 9.6MB 3.6MB/s eta 0:00:05     |████████████████▌               | 12.4MB 3.0MB/s eta 0:00:04     |███████████████████             | 14.3MB 3.0MB/s eta 0:00:04     |███████████████████████████████ | 23.5MB 10.1MB/s eta 0:00:01
[?25hCollecting smart-open>=1.8.1
[?25l  Downloading https://files.pythonhosted.org/packages/0c/09/735f2786dfac9bbf39d244ce75c0313d27d4962e71e0774750dc809f2395/smart_open-1.9.0.tar.gz (70kB)
[K     |████████████████████████████████| 71kB 460kB/s eta 0:00:011
[?25hRequirement already satisfied: six>=1.5.0 in /home/nbuser/anaconda3_420/lib/python3.5/site-packages (from gensim) (1.11.0)
Requirement already satisfied: scipy>=0.18.1 in /home/nbuser/anaconda3_420/l[...]<jupyter_text>Imports and loggingFirst, we start with our imports and get logging established:<jupyter_code># imports needed and set up logging
import gzip
import gensim 
import logging

logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)<jupyter_output><empty_output><jupyter_text>Dataset Next, is our dataset. The secret to getting Word2Vec really working for you is to have lots and lots of text data. In this case I am going to use data from the [OpinRank](http://kavita-ganesan.com/entity-ranking-data/) dataset. This dataset has full user reviews of cars and hotels. I have specifically concatenated all of the hotel reviews into one big file which is about 97MB compressed and 229MB uncompressed. We will use the compressed file for this tutorial. Each line in this file represents a hotel review. You can download the OpinRank Word2Vec dataset here.To avoid confusion, while gensim’s word2vec tutorial says that you need to pass it a sequence of sentences as its input, you can always pass it a whole review as a sentence (i.e. a much larger size of text), and it should not make much of a difference. Now, let's take a closer look at this data below by printing the first line. You can see that this is a pretty hefty review.<jupyter_code>data_file="reviews_data.txt.gz"

with gzip.open ('reviews_data.txt.gz', 'rb') as f:
    for i,line in enumerate (f):
        if(i>3):
            break
        else:
            print(line)<jupyter_output><empty_output><jupyter_text>Read files into a listNow that we've had a sneak peak of our dataset, we can read it into a list so that we can pass this on to the Word2Vec model. Notice in the code below, that I am directly reading the compressed file. I'm also doing a mild pre-processing of the reviews using `gensim.utils.simple_preprocess (line)`. This does some basic pre-processing such as tokenization, lowercasing, etc and returns back a list of tokens (words). Documentation of this pre-processing method can be found on the official [Gensim documentation site](https://radimrehurek.com/gensim/utils.html).<jupyter_code>def read_input(input_file):
    """This method reads the input file which is in gzip format"""
    
    logging.info("reading file {0}...this may take a while".format(input_file))
    
    with gzip.open (input_file, 'rb') as f:
        for i, line in enumerate (f): 

            if (i%10000==0):
                logging.info ("read {0} reviews".format (i))
            # do some pre-processing and return a list of words for each review text
            yield gensim.utils.simple_preprocess (line)

# read the tokenized reviews into a list
# each review item becomes a serries of words
# so this becomes a list of lists
documents = list (read_input (data_file))
logging.info ("Done reading data file")<jupyter_output>2018-01-28 00:27:46,482 : INFO : reading file reviews_data.txt.gz...this may take a while
2018-01-28 00:27:46,484 : INFO : read 0 reviews
2018-01-28 00:27:48,868 : INFO : read 10000 reviews
2018-01-28 00:27:51,350 : INFO : read 20000 reviews
2018-01-28 00:27:54,287 : INFO : read 30000 reviews
2018-01-28 00:27:57,177 : INFO : read 40000 reviews
2018-01-28 00:28:00,147 : INFO : read 50000 reviews
2018-01-28 00:28:03,028 : INFO : read 60000 reviews
2018-01-28 00:28:05,508 : INFO : read 70000 reviews
2018-01-28 00:28:08,176 : INFO : read 80000 reviews
2018-01-28 00:28:10,532 : INFO : read 90000 reviews
2018-01-28 00:28:12,768 : INFO : read 100000 reviews
2018-01-28 00:28:14,962 : INFO : read 110000 reviews
2018-01-28 00:28:17,314 : INFO : read 120000 reviews
2018-01-28 00:28:19,624 : INFO : read 130000 reviews
2018-01-28 00:28:21,985 : INFO : read 140000 reviews
2018-01-28 00:28:24,178 : INFO : read 150000 reviews
2018-01-28 00:28:26,464 : INFO : read 160000 reviews
2018-01-28 00:28:29,481[...]<jupyter_text>Training the Word2Vec modelTraining the model is fairly straightforward. You just instantiate Word2Vec and pass the reviews that we read in the previous step (the `documents`). So, we are essentially passing on a list of lists. Where each list within the main list contains a set of tokens from a user review. Word2Vec uses all these tokens to internally create a vocabulary. And by vocabulary, I mean a set of unique words.After building the vocabulary, we just need to call `train(...)` to start training the Word2Vec model. Training on the [OpinRank](http://kavita-ganesan.com/entity-ranking-data/) dataset takes about 10 minutes so please be patient while running your code on this dataset.Behind the scenes we are actually training a simple neural network with a single hidden layer. But, we are actually not going to use the neural network after training. Instead, the goal is to learn the weights of the hidden layer. These weights are essentially the word vectors that we’re trying to learn.<jupyter_code>model = gensim.models.Word2Vec (documents, size=150, window=10, min_count=2, workers=10)
model.train(documents,total_examples=len(documents),epochs=10)<jupyter_output>2018-01-26 22:57:00,707 : WARNING : consider setting layer size to a multiple of 4 for greater performance
2018-01-26 22:57:00,709 : INFO : collecting all words and their counts
2018-01-26 22:57:00,710 : INFO : PROGRESS: at sentence #0, processed 0 words, keeping 0 word types
2018-01-26 22:57:01,045 : INFO : PROGRESS: at sentence #10000, processed 1655714 words, keeping 25777 word types
2018-01-26 22:57:01,399 : INFO : PROGRESS: at sentence #20000, processed 3317863 words, keeping 35016 word types
2018-01-26 22:57:01,803 : INFO : PROGRESS: at sentence #30000, processed 5264072 words, keeping 47518 word types
2018-01-26 22:57:02,188 : INFO : PROGRESS: at sentence #40000, processed 7081746 words, keeping 56675 word types
2018-01-26 22:57:02,560 : INFO : PROGRESS: at sentence #50000, processed 9089491 words, keeping 63744 word types
2018-01-26 22:57:02,905 : INFO : PROGRESS: at sentence #60000, processed 11013723 words, keeping 76781 word types
2018-01-26 22:57:03,202 : INFO : PROGRESS: a[...]<jupyter_text>Now, let's look at some output This first example shows a simple case of looking up words similar to the word `dirty`. All we need to do here is to call the `most_similar` function and provide the word `dirty` as the positive example. This returns the top 10 similar words.<jupyter_code>w1 = "dirty"
model.wv.most_similar (positive=w1)<jupyter_output>2018-01-26 23:03:42,416 : INFO : precomputing L2-norms of word weight vectors<jupyter_text>That looks pretty good, right? Let's look at a few more. Let's look at similarity for `polite`, `france` and `shocked`.<jupyter_code># look up top 6 words similar to 'polite'
w1 = ["polite"]
model.wv.most_similar (positive=w1,topn=6)

# look up top 6 words similar to 'france'
w1 = ["france"]
model.wv.most_similar (positive=w1,topn=6)

# look up top 6 words similar to 'shocked'
w1 = ["shocked"]
model.wv.most_similar (positive=w1,topn=6)<jupyter_output><empty_output><jupyter_text>That's, nice. You can even specify several positive examples to get things that are related in the provided context and provide negative examples to say what should not be considered as related. In the example below we are asking for all items that *relate to bed* only:<jupyter_code># get everything related to stuff on the bed
w1 = ["bed",'sheet','pillow']
w2 = ['couch']
model.wv.most_similar (positive=w1,negative=w2,topn=10)<jupyter_output><empty_output><jupyter_text>Similarity between two words in the vocabulary You can even use the Word2Vec model to return the similarity between two words that are present in the vocabulary.<jupyter_code># similarity between two different words
model.wv.similarity(w1="dirty",w2="smelly")
# similarity between two identical words
model.wv.similarity(w1="dirty",w2="dirty")
# similarity between two unrelated words
model.wv.similarity(w1="dirty",w2="clean")<jupyter_output><empty_output><jupyter_text>Under the hood, the above three snippets computes the cosine similarity between the two specified words using word vectors of each. From the scores, it makes sense that `dirty` is highly similar to `smelly` but `dirty` is dissimilar to `clean`. If you do a similarity between two identical words, the score will be 1.0 as the range of the cosine similarity score will always be between [0.0-1.0]. You can read more about cosine similarity scoring [here](https://en.wikipedia.org/wiki/Cosine_similarity).  Find the odd one outYou can even use Word2Vec to find odd items given a list of items.<jupyter_code># Which one is the odd one out in this list?
model.wv.doesnt_match(["cat","dog","france"])
# Which one is the odd one out in this list?
model.wv.doesnt_match(["bed","pillow","duvet","shower"])<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup