<jupyter_start><jupyter_text>![Callysto.ca Banner](https://github.com/callysto/curriculum-notebooks/blob/master/callysto-notebook-banner-top.jpg?raw=true)  Analysing Text StatisticsLet's try out some statistical analysis of text, including [natural language processing](https://en.wikipedia.org/wiki/Natural_language_processing), using a [public domain](https://en.wikipedia.org/wiki/Public_domain) book from [Project Gutenberg](http://www.gutenberg.org).The example we'll use is the [most downloaded](http://www.gutenberg.org/browse/scores/top) book, [Pride and Prejudice by Jane Austen](http://www.gutenberg.org/ebooks/1342). Running this first code cell will import and display the contents of the book.Feel free to change the link in the following code cell if you'd like to explore another book, but make sure you are using the `Plain Text UTF-8` link.<jupyter_code>gutenberg_text_link = 'http://gutenberg.org/files/1342/1342-0.txt'

import requests
r = requests.get(gutenberg_text_link) # get the online book file
r.encoding = 'utf-8' # specify the type of text encoding in the file
text = r.text.split('***')[2] # get the part after the header
text = text.replace("’","'").replace("“",'"').replace("”",'"') # replace any 'smart quotes'
book_title = r.text[r.text.index('Title:')+7:r.text.index('Author:')-4] # find the book title
print(text)<jupyter_output><empty_output><jupyter_text>Making a DataFrame of ChaptersNow that we have the text of the book, let's split it into chapters. We'll use the Python [library](https://en.wikipedia.org/wiki/Library_(computing)) called [pandas](https://pandas.pydata.org) to create a [dataframe](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html) that includes the text and length of each chapter (number of characters, including spaces and punctuation).<jupyter_code>import pandas as pd
chapters = pd.DataFrame() # create an empty data frame that we will append to
for chapter in text.split('Chapter'):
    if len(chapter)>500: # so that we are getting actual book chapters
        chapter = chapter.replace('\r','').replace('\n','') # delete the 'new line' characters
        chapters = chapters.append({'Chapter Text':chapter, 'Length':len(chapter)}, ignore_index=True)
chapters.index=chapters.index+1 # set the index to the chapter number
chapters<jupyter_output><empty_output><jupyter_text>Visualizing Chapter LengthsFrom that dataframe we can create a bar graph of chapter lengths.<jupyter_code>import plotly.express as px
px.bar(chapters, y='Length', title='Characters per Chapter in '+book_title).update_xaxes(title='Chapter')<jupyter_output><empty_output><jupyter_text>Counting Words by TypeWe'll use the [spaCy](https://spacy.io) natural language processing library to identify the [parts of speech](https://spacy.io/api/annotationpos-tagging) in the text. For this example we'll just look at adjectives, verbs, nouns, and proper nouns, but you can add to the list on the first line in the code cell.This will take a while to run, and will result in a dataframe containing the number and percent of each of those parts of speech in each chapter.<jupyter_code>word_types = ['ADJ', 'VERB', 'NOUN', 'PROPN'] # https://spacy.io/api/annotation#pos-tagging

# uncomment the following two installation lines if you get errors
#!pip install spacy --user
#!python -m spacy download en_core_web_sm

import en_core_web_sm
nlp = en_core_web_sm.load() # set up natural language processing
parts_of_speech_df = pd.DataFrame(columns=word_types) # create an empty dataframe with column names
for i in range(1,len(chapters)+1): # iterate through the chapters dataframe
    parts_of_speech_list = [] # create an empty list
    word_count = 0
    for token in nlp(chapters['Chapter Text'][i]): # loop through each token in the chapter
        word_count += 1 # increment the word counter
        part_of_speech = token.pos_
        if part_of_speech in word_types: # if it is in the list of types we made
            parts_of_speech_list.append(part_of_speech)
    word_type_count = {} # create an empty dictionary
    for word_type in word_types:
        word_type_count.update({word_type:parts_of_speech_list.count(word_type)}) # add to that dictionary
    word_type_count.update({'Words':word_count})
    parts_of_speech_df = parts_of_speech_df.append(word_type_count, ignore_index=True) # add to dataframe
parts_of_speech_df = parts_of_speech_df.astype(int) # convert values to integers
parts_of_speech_df.index = parts_of_speech_df.index+1 # set the dataframe index to Chapter number
parts_of_speech_df['Adjectives %'] = parts_of_speech_df['ADJ']/parts_of_speech_df['Words']*100
parts_of_speech_df['Verbs %'] = parts_of_speech_df['VERB']/parts_of_speech_df['Words']*100
parts_of_speech_df['Nouns %'] = parts_of_speech_df['NOUN']/parts_of_speech_df['Words']*100
parts_of_speech_df['Proper Nouns %'] = parts_of_speech_df['PROPN']/parts_of_speech_df['Words']*100
parts_of_speech_df<jupyter_output><empty_output><jupyter_text>We can also plot the proportional occurrences of those parts of speech.<jupyter_code>px.line(parts_of_speech_df, y=['Adjectives %', 'Verbs %', 'Nouns %', 'Proper Nouns %'], 
        title='Proportion of Word Types in '+book_title).update_xaxes(title='Chapter')<jupyter_output><empty_output><jupyter_text>Most Common VerbsTo get an idea of the most common words in the text we can look at a part of speech, verbs for example, and count which are the most frequent.This will also take some time to run.<jupyter_code>word_type = 'VERB'

from collections import Counter
words_df = pd.DataFrame()
for i in range(1,len(chapters)+1): # loop through the chapters dataframe
    word_list = []
    for token in nlp(chapters['Chapter Text'][i]):
        if token.pos_ == word_type: # if the token is a VERB
            word_list.append(token.lemma_.strip().lower())
    words_df = words_df.append(Counter(word_list), ignore_index=True) # add the list to the dataframe
words_df.index = words_df.index+1 # set the dataframe index to Chapter number
words_df.sum().sort_values(ascending=False)<jupyter_output><empty_output><jupyter_text>In our example text there are 1233 unique verbs. Let's look at the `10` most common verbs.<jupyter_code>words_df.sum().sort_values(ascending=False).head(10)<jupyter_output><empty_output><jupyter_text>We can also choose a verb and plot its frequency by chapter as a percent of the total number of words.<jupyter_code>word = 'say'

words_df['%'] = words_df[word]/parts_of_speech_df['Words']*100 # calculate the percent of the words in each chapter
px.bar(words_df, y='%', title='Percent Frequency of the Word "'+word+'" by Chapter in '+book_title).update_xaxes(title='Chapter').update_yaxes(title='Percent Freqency')<jupyter_output><empty_output><jupyter_text>Most Common NamesWe can also look at character names and how often they occur in each chapter. The spaCy library does a fairly good job of identifying names, but you may see some false positives (words that aren't actually character names).<jupyter_code>names_df = pd.DataFrame()
for i in range(1,len(chapters)+1):
    names_list = []
    for token in nlp(chapters['Chapter Text'][i]):
        #if token.pos_ == 'PROPN':
        if token.ent_type_ == 'PERSON': # seems to be more reliable than part_of_speech == proper_noun
            names_list.append(token.text)
    names_df = names_df.append(Counter(names_list), ignore_index=True)
names_df<jupyter_output><empty_output><jupyter_text>List of Character NamesWe can check out the list of words identified as names.<jupyter_code>for name in names_df.columns:
    print(name)<jupyter_output><empty_output><jupyter_text>Cleaning DataIf you'd like to remove columns that may categorized incorrectly or are uncommon, we can drop columns with fewer than five occurrences.<jupyter_code>for column in names_df.columns:
    if names_df[column].sum() < 5: # if there are fewer than five occurences
        names_df.drop(columns=column, inplace=True)
names_df<jupyter_output><empty_output><jupyter_text>Visualization of Name FrequenciesLet's make a bar graph of the top `20` most frequently mentioned characters.<jupyter_code>d = names_df.sum().sort_values(ascending=False).head(20)
px.bar(d, title='Character Name Frequencies in '+book_title).update_yaxes(title='Frequency').update_xaxes(title='Name').update(layout_showlegend=False)<jupyter_output><empty_output><jupyter_text>Cumulative Name Frequencies over TimeSince we have the text divided into chapters, let's visualize the cumulative mentions of the top `3` character names throughout the book.<jupyter_code>how_many_names = 3
main_character_names = names_df.sum().sort_values(ascending=False).head(how_many_names).index # list the top three
main_characters = names_df[main_character_names].fillna(value=0) # create a new dataframe for top three
main_characters.index = main_characters.index+1
title = 'Cumulative Character Mentions of Time in '+book_title
px.line(main_characters.cumsum(), title=title).update_yaxes(title='Total Mentions').update_xaxes(title='Chapter')<jupyter_output><empty_output><jupyter_text>Sentiment Analysis[Sentiment analysis](https://en.wikipedia.org/wiki/Sentiment_analysis) is categorizing text based on its tone (negative, neutral, or positive).For this we will use the [vaderSentiment](https://github.com/cjhutto/vaderSentiment) library, then visualize the positive and negative sentiment by chapter.<jupyter_code>#!pip install vaderSentiment --user
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
analyzer = SentimentIntensityAnalyzer()
sentiment_df = pd.DataFrame()
for i in range(1,len(chapters)+1):
    senti = analyzer.polarity_scores(chapters['Chapter Text'][i]) # analyze the sentiment of chapter
    sentiment_df = sentiment_df.append(senti, ignore_index=True) # add to dataframe
sentiment_df['neg'] = -sentiment_df['neg'] # change the sign of the negative sentiment
sentiment_df.index = sentiment_df.index+1
px.bar(sentiment_df, y=['pos', 'neg'], title=book_title+' Sentiment Analysis by Chapter').update(layout_showlegend=False)<jupyter_output><empty_output><jupyter_text>ReadabilityOne last library to introduce, [textstat](https://github.com/shivam5992/textstat) for checking the readability, complexity, and grade level of text. It includes a [number of functions](https://github.com/shivam5992/textstatlist-of-functions), but we'll only use a few of them. This will take a while to run.<jupyter_code>#!pip install --user textstat
import textstat
textstats = pd.DataFrame()
for i in range(1,len(chapters)+1):
    text = chapters['Chapter Text'][i]
    textstats_data = {'Flesch-Kincaid Grade':textstat.flesch_kincaid_grade(text),
                        'Gunning Fog Index':textstat.gunning_fog(text),
                        'Linsear Write Formula':textstat.linsear_write_formula(text),
                        'Readability':textstat.text_standard(text, float_output=True)}
    textstats = textstats.append(textstats_data, ignore_index=True)
textstats.index = textstats.index+1
textstats<jupyter_output><empty_output><jupyter_text>Now that we have a dataframe of readability information, we can plot and describe the readability statistics.<jupyter_code>px.line(textstats, y='Readability', title=book_title+' Readability by Chapter').update_xaxes(title='Chapter')
textstats.describe()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup