<jupyter_start><jupyter_text>GA Data Science Final Project - 3- NLP  Import the data<jupyter_code>import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer<jupyter_output><empty_output><jupyter_text>Convert datatypes to `str`<jupyter_code>df = pd.read_csv('issue_comments_jupyter_copy.csv')
df['org'] = df['org'].astype('str')
df['repo'] = df['repo'].astype('str')
df['comments'] = df['comments'].astype('str')
df['user'] = df['user'].astype('str')<jupyter_output><empty_output><jupyter_text>Scikit Learn Count Vectorizer First, save the Series 'comments' to the variable `comments`.<jupyter_code>comments = df.comments
comments.head()<jupyter_output><empty_output><jupyter_text>*comments* is a long column with many rows and I want to use CountVectorizer on all the comments to generate counts of all the words used, so I'll convert all the comments to strings and concatenate those strings using a space as a separator, then save the whole long string to a text file and also store it as the variable **all_comments**CountVectorizer documentation: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.htmlwhich states: "Convert a collection of text documents to a matrix of token countsThis implementation produces a sparse representation of the counts using scipy.sparse.coo_matrix.If you do not provide an a-priori dictionary and you do not use an analyzer that does some kind of feature selection then the number of features will be equal to the vocabulary size found by analyzing the data."  Second, convert all rows in `comments` to a string.<jupyter_code>with open ('all_comments.txt',"wb") as fd:
    all_comments = comments.str.cat(sep=' ')
    fd.write (all_comments)
cvec = CountVectorizer()

cvec.fit([all_comments])
df2 = pd.DataFrame(cvec.transform([all_comments]).todense(), columns=cvec.get_feature_names())
df2.transpose().sort_values(0, ascending=False).head(25)<jupyter_output><empty_output><jupyter_text>The output shows a multitude of stop words that we can take out later. Frequency Distribution Curve<jupyter_code>import nltk
fdist1 = nltk.FreqDist(df2)
fdist1

vocabulary1 = fdist1.keys()
vocabulary1[:50]<jupyter_output><empty_output><jupyter_text>From Loper, et al. 2009: "When we first invoke FreqDist, we pass the name of the text as an argument. We can inspect the total number of words ('outcomes') that have been counted up. The expression keys() gives us a list of all the distinct types in the text, and we can look at the first 50 of these by slicing the list.<jupyter_code>fdist1.plot(50, cumulative=True)
fdist1.hapaxes()<jupyter_output><empty_output><jupyter_text>Depending on output: Since neither frequent nor infrequent words help, we need to try something else. Beginning to see that this is a very noisy data set.  NLP Bag of WordsUsing **segmentation** to identify sentences within `all_comments`<jupyter_code>from nltk.tokenize import PunktSentenceTokenizer
sent_detector = PunktSentenceTokenizer()
sent_detector.sentences_from_text(all_comments.decode('utf8'))<jupyter_output><empty_output><jupyter_text>Lemmatization<jupyter_code>from nltk.stem import WordNetLemmatizer
from sklearn.linear_model import LogisticRegression

lemmatizer = WordNetLemmatizer()
lemmatizer.lemmatize(df2)<jupyter_output><empty_output><jupyter_text>Stemming  Term Frequency - Inverse Document Frequency - TF-IDF<jupyter_code>from sklearn.feature_extraction.text import TfidfVectorizer
tvec = TfidfVectorizer(stop_words='english')
tvec.fit([all_comments])
tfidf_data = pd.DataFrame(tvec.transform([all_comments]).todense(), 
    columns=tvec.get_feature_names(), index=['all_comments'])
tfidf_data.transpose().sort_values('all_comments', ascending=False).head(10).transpose()<jupyter_output><empty_output><jupyter_text>"tf–idf, short for term frequency–inverse document frequency, is a numerical statistic that is intended to reflect how important a word is to a document in a collection or corpus. What this output means: the words above are most important across the Jupyter GitHub comments.   Topic Modeling: LDA  Import libraries<jupyter_code>from gensim import corpora, models, matutils
from sklearn.feature_extraction import text
from sklearn.feature_extraction.text import CountVectorizer
from collections import defaultdict
import pandas as pd<jupyter_output><empty_output><jupyter_text>Instantiate CountVectorizer and fit data to model<jupyter_code>#circling back to add stop words
stop_words = text.ENGLISH_STOP_WORDS.union(['jupyter', 'notebook', 'https', 'github', 'com', 'html', 'http', 'org','ellisonbg','don'])
vectorizer = CountVectorizer(stop_words=stop_words, min_df=3)
X = vectorizer.fit_transform(comments.dropna())<jupyter_output><empty_output><jupyter_text>Tokens that were saved after stopwords removed<jupyter_code>vectorizer.vocabulary_<jupyter_output><empty_output><jupyter_text>Counts of tokens<jupyter_code>docs = pd.DataFrame(X.toarray(), columns=vectorizer.get_feature_names()).head()
docs
docs.shape<jupyter_output><empty_output><jupyter_text>Set up LDA model - set up the vocabulary<jupyter_code>vocab = {v: k for k, v in vectorizer.vocabulary_.iteritems()}
vocab
vectorizer<jupyter_output><empty_output><jupyter_text>Set up the actual LDA model<jupyter_code>lda = models.LdaModel(
    matutils.Sparse2Corpus(X, documents_columns=False),
    #corpus,
    num_topics = 5,
    passes = 20,
    id2word = vocab
)<jupyter_output><empty_output><jupyter_text>*Third pass looking at the topics, with added stop words 'jupyter', 'notebook', 'https', 'github', 'com', 'html', 'http', 'org','ellisonbg','don'*<jupyter_code>lda.print_topics(num_topics=5, num_words=5)<jupyter_output><empty_output><jupyter_text>*High-level labels*<jupyter_code>#First pass: funny but not helpful topics:
topics_labels = {
    0: "Installing Python and Jupyter"
    1: "I think I just like the Jupyter Notebook"
    2: "JavaScript and .py Files"
    3: "Cells"
    4: "GitHub"
}

#Third pass topics
topics_labels = {
    0: "Viewing and writing notifications"
    1: "Python files and packages"
    2: "PR and issue closure"
    3: "User interface"
    4: "Thinking"
}
for ti, topic in enumerate(lda.show_topics(num_topics = 5)):
    print("Topic: %d" % (ti))
    print(topic)
    print()<jupyter_output>Topic: 0
(0, u'0.028*"notifications" + 0.027*"view" + 0.024*"directly" + 0.024*"reply" + 0.023*"wrote" + 0.022*"email" + 0.021*"issuecomment" + 0.021*"pull" + 0.017*"issues" + 0.016*"brian"')
()
Topic: 1
(1, u'0.025*"file" + 0.024*"python" + 0.024*"py" + 0.019*"line" + 0.015*"packages" + 0.014*"lib" + 0.013*"ipython" + 0.013*"local" + 0.012*"usr" + 0.011*"users"')
()
Topic: 2
(2, u'0.024*"thanks" + 0.015*"issue" + 0.010*"pr" + 0.009*"think" + 0.009*"just" + 0.009*"ll" + 0.009*"like" + 0.008*"try" + 0.008*"work" + 0.007*"issues"')
()
Topic: 3
(3, u'0.040*"js" + 0.017*"notebookapp" + 0.013*"png" + 0.013*"assets" + 0.012*"cloud" + 0.012*"githubusercontent" + 0.012*"docker" + 0.012*"error" + 0.012*"17" + 0.009*"static"')
()
Topic: 4
(4, u'0.012*"think" + 0.011*"like" + 0.011*"cell" + 0.009*"use" + 0.008*"just" + 0.007*"kernel" + 0.007*"way" + 0.006*"code" + 0.006*"want" + 0.006*"make"')
()
StarcoderdataJupyterStructuredCleanDedup