<jupyter_start><jupyter_text>[LEGALST-123] Lab 16: Intro to Text Analysis---In this lab, students will learn about dominant language models in natural language processing and the basics of how to implement it in Python. We'll be using the data you extracted from the last lab (un-debates-2001-clean.csv).<jupyter_code># dependencies
import numpy as np
import pandas as pd<jupyter_output><empty_output><jupyter_text>OverviewHere we will discuss one widely used representation of text:- Bag-of-Words Encoding: encodes text by the frequency of each wordThis model was very popular in early text analysis, and continues to be used today. In fact, the models that have replaced it are still very difficult to actually interpret, giving the BoW approach a slight advantage if we want to understand why the model makes certain decisions. Once we have our BoW model we can analyze it in a high-dimensional vector space, which gives us more insights into the similarities and clustering of different texts.<jupyter_code>## retrieve our data
data = pd.read_csv('data/un-debates-2001-clean.csv', encoding='utf-8-sig', index_col=0).reset_index(drop=True)
data.head(5)
## let's store our text and our tokens into a list
text_list = ...
tokens_list = ...<jupyter_output><empty_output><jupyter_text>Bag-of-Words EncodingThe bag-of-words encoding is widely used and a standard representation for text in many of the popular text clustering algorithms.__Key Things to Note:__1. __Stop words are removed.__ Stop-words are words like 'is' and 'about' that in isolation contain very little information about the meaning of the sentence.2. __Word order information is lost.__ 3. __Capitalization and punctuation__ are typically removed.4. __Sparse Encoding:__ is necessary to represent the bag-of-words efficiently. There are millions of possible words (including terminology, names, and misspellings) and so instantiating a 0 for every word that is not in each record would be incredibly inefficient.Why is it called a __bag-of-words__? __SOLUTION:__  Implementing the Bag-of-words Model  Review of TokensIf you remember from the last lab, we created tokens and added it to our table. Normally at this point of the stage, you would create tokens for yourself to use, so let's introduce a new term called `Counter`.The easiest way to count tokens is using the `Counter` object from `collections`. This will give you back a dictionary with the token counts:<jupyter_code>from collections import Counter
# extract the first speech in tokens_list, split it by whitespace, then put it into a Counter
first_speech = ...
counter = Counter(first_speech) 
counter<jupyter_output><empty_output><jupyter_text>The `most_common()` method can be called on a Counter to return the most common tokens and their counts. What are the most common tokens in the first speech? What do these common words tell you about the content or tone of the speech?<jupyter_code>...<jupyter_output><empty_output><jupyter_text>Document-Term MatrixWe can use sklearn to construct a bag-of-words representation of text. Create an instance of [CountVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html).<jupyter_code>from sklearn.feature_extraction.text import CountVectorizer

# Construct the tokenizer.
cv = ...<jupyter_output><empty_output><jupyter_text>After creating the CountVectorizer object, use `fit_transform` on it. `fit_transform` takes in the list of documents we want to represent (in this case, the list of tokenized text).<jupyter_code>dtm = ...
dtm<jupyter_output><empty_output><jupyter_text>What's this? A sparse matrix just means that many cells in the table don't have value. We can get a better look at what's going on by turning the sparse matrix into a data frame. First, get the list of words in our 'bag-of-words' by using `get_feature_names()` on your CountVectorizer.<jupyter_code># create labels for columns.
word_list = ...
word_list<jupyter_output><empty_output><jupyter_text>You can then de-sparsify the sparse matrix by turning it into an array. Try using `toarray()` on it.<jupyter_code># de-sparsify by turning dtm into an array
desparse = ...
desparse<jupyter_output><empty_output><jupyter_text>You now have everything you need to convert your sparse matrix to a DataFrame. Double-check the [documentation](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html) for a reminder of how to construct the frame.<jupyter_code># create a dataframe with words as columns and the sparse matrix data as the data
dtm_df = ...
dtm_df.head()<jupyter_output><empty_output><jupyter_text>This is what we call a Document Term Matrix, a core concept in NLP and text analysis.As you can see, there are columns for each word in the entire list. Each row is for each text. The values are the word count for that word in the corresponding text. Note that there are many 0s, hence the matrix is 'sparse'. Why are there so many zeros? __SOLUTION__:  To see the frequency of a word across documents, index that word's column.<jupyter_code># can easily find the frequencies for each of the given words
dtm_df['zone']
# what's the total number of times the word 'zone' pops up?
...
# how many words appear in the 100th document?
...<jupyter_output><empty_output><jupyter_text>NormalizationLet's see if we can take another step and try to make equal comparisons across each of the texts. We can normalize the values by dividing each word count by the total number of words in the text. We'll need to sum on axis=1, or summing the row, as each row is a text), as opposed to summing up the column.Once we have the total number of words in the text, we can get a percentage of words that one particular word accounts for, thus applying this method to every other word across the matrix.<jupyter_code># see if you can fill this out on your own following the steps listed above.
row_sums = ... # sum up the desparse on axis=1
normed = ... # divide this over the total number of row_sums
dtm_df = ... # create a data frame using the word_list and the new data
dtm_df.head()<jupyter_output><empty_output><jupyter_text>When would it be most important to normalize word counts?1. When you have a lot of documents2. When you have very few documents3. When the documents are of many different lengths4. When the documents are all around the same lengthWhy? __SOLUTION__:  StreamliningOverall, this was a lot of work and if it is as common as we say it is in NLP, shouldn't someone have streamlined it before? In fact, we can simply instruct CountVectorizer not to include stopwords at all (so we could use it on our non-tokenized text), and another function, TfidfTransformer, normalizes easily.<jupyter_code>from sklearn.feature_extraction.text import TfidfTransformer

# fill out this beginning part
# when you create your CountVectorizer, set the stop_words argument equal to 'english'
cv = ...
dtm = ...

# this is what allows us to easily streamline
tt = TfidfTransformer(norm='l1',use_idf=False)
dtm_tf = tt.fit_transform(dtm)
dtm_tf<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup