<jupyter_start><jupyter_text>Run saved model on wikipedia page to classify text as needing citationAuthor: <NAME>Revision: 5/11/18 - DRM - translate .py files into .ipynb, misc formatting1. loads in text from wikipedia page with python wikipedia library2. breaks text into sentences3. gets parts of speech and stems words to create binary feature vector4. tests each binary feature vector representing a sentence with the model5.  saves output in a text file where each line is a sentence with a 0 or 1 to indicate if the sentence needs citation or not.<jupyter_code># import relevant libraries
import wikipedia
import nltk
import pandas as pd
import numpy as np
import pickle
from nltk.stem.snowball import SnowballStemmer<jupyter_output><empty_output><jupyter_text>Create p_stemmer of class PorterStemmer<jupyter_code>p_stemmer = SnowballStemmer("english", ignore_stopwords=True)<jupyter_output><empty_output><jupyter_text>pull text from wikipedia pageNote: you can change the name if desired.<jupyter_code>wikiPage = wikipedia.page("Forbidden City") 
content = wikiPage.content # Content of page<jupyter_output><empty_output><jupyter_text>extract sentences from paragraph<jupyter_code>fullSents = nltk.sent_tokenize(content)<jupyter_output><empty_output><jupyter_text>extract words from sentences and find parts of speech of words<jupyter_code>wordsInSents = list()
posInSents = list()
for sent in fullSents:
    sent_token = nltk.word_tokenize(sent) # word tokenize the sentence
    sent_pos = nltk.pos_tag(sent_token) # tag with part of speech
    stemWords = list() # create list to store stemmed words
    onlyPOS = list() # create list to store sentence parts of speech
    for item in sent_pos:
        word = item[0]
        stemWords.append(p_stemmer.stem(word.lower())) # lower case and stem words
        onlyPOS.append(item[1])
    wordsInSents.append(stemWords)
    posInSents.append(onlyPOS)<jupyter_output><empty_output><jupyter_text>go through each word and the pos of each word to see if it matches the 600 feature names of the finalized model to create the 600 element binary vector<jupyter_code>featureNames = np.load("../Data/FeatureName_finalized_model.npy")
featureMat = np.zeros((len(wordsInSents), len(featureNames)))
for i in range(0,len(wordsInSents)):
    words = wordsInSents[i]
    POSes = posInSents[i]
    for j in range(0,len(featureNames)):
        name = featureNames[j]
        isWord = name in words
        isPOS = name in POSes
        if isWord or isPOS:
            featureMat[i,j] = 1<jupyter_output><empty_output><jupyter_text>feed the binary vector into the model to get a prediction<jupyter_code>loaded_model = pickle.load(open("../Data/finalized_model.sav", "rb"))
myPredictions = loaded_model.predict(featureMat)<jupyter_output><empty_output><jupyter_text>write out each sentence and its prediction into a new text file<jupyter_code>file = open("../Data/classifiedText_wikipedia.txt", "w")

for i in range(0,len(fullSents)):
    sent = fullSents[i]
#     sent = sent.encode("ascii", "ignore")
    sent_complete = sent + "    " + str(myPredictions[i]) + "\n"
    file.write(sent_complete)

file.close()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup