<jupyter_start><jupyter_text>Text SummarizationText summarization is a subdomain of Natural Language Processing (NLP) that deals with extracting summaries from huge chunks of texts. There are two main types of techniques used for text summarization: - NLP-based techniques and - Deep learning-based techniques. We will use in this notebook a simple NLP-based technique for text summarization. We will simply use Python's NLTK library for summarizing Wikipedia articles.  https://dev.to/davidisrawi/build-a-quick-summarizer-with-python-and-nltkhttps://stackabuse.com/text-summarization-with-nltk-in-python/ - from nltk.corpus import stopwords- from nltk.tokenize import word_tokenize, sent_tokenize  First model Four steps to build a summarizer- 1 - Remove stop words for the analysis- 2 - Create frequency table of words- 3 - Assign score to each sentence depending on the words is contains and the frequency table- 4 - Build summary by adding every sentence above a certain score threshold __Stop words__ are any word that does not add a value to the meaning of sentence. __Corpus__ means a collection of text. __Tokenizers__ divides a text into a series of tokens.There are three main tokenizers: word, sentence and regex tokenizer.<jupyter_code>import nltk
nltk.download('stopwords')
nltk.download('punkt')
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize, sent_tokenize
from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
text = "So, keep working. Keep striving. Never give up. Fall down seven times, get up eight. Ease is a greater threat to progress than hardship. Ease is a greater threat to progress than hardship. So, keep moving, keep growing, keep learning. See you at work."<jupyter_output><empty_output><jupyter_text>First - create two arrays<jupyter_code># one array for stop words 
stopWords = set(stopwords.words("english"))
# and other array for every word in the body of the text
words = word_tokenize(text)<jupyter_output><empty_output><jupyter_text>Second - create a dictionary for the word frequency table.Words there are not in stopWords array.<jupyter_code>freqTable = dict()
for word in words:
    word = word.lower()
    if word in stopWords:
        continue
    if word in freqTable:
        freqTable[word] += 1
    else:
        freqTable[word] = 1<jupyter_output><empty_output><jupyter_text>Third - assigning a score to every sentence.Building a sentence tokenizer<jupyter_code>sentences = sent_tokenize(text)
sentences<jupyter_output><empty_output><jupyter_text>Going through every sentence and giving it a score depending on the words it has.<jupyter_code>sentenceValue = dict()
for sentence in sentences:
    for wordValue in freqTable:
        if wordValue[0] in sentence.lower():
            if sentence in sentenceValue:
                sentenceValue[sentence] += wordValue[1]<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup