<jupyter_start><jupyter_text>Text extraction from pdfs<jupyter_code>## Extraction from google drive folders
input_folder = os.path.join(PROJECT_ROOT, "tasks", "extract_text", "input")
output_folder = os.path.join(PROJECT_ROOT, "tasks", "extract_text", "output")

filepaths = [os.path.join(input_folder, file) for file in os.listdir(input_folder) if os.path.splitext(file)[1] == ".pdf"]
filepaths
# pytesseract extraction
for file in filepaths:
    pages = get_image1(file)
    text = ""  # initialize document text
    for pageNum, imgBlob in enumerate(pages):
        page_text = pytesseract.image_to_string(imgBlob, lang="spa")  # extract text
        text += wrap_pagenum(page_text, pageNum)  # wrap page number
    split_paragraphs(text)  # split document text into paragraphscommand string or list of command arguments to run inside the container after is created. The commands execute from the workspaceFolder
    export_ocr(text, file, "pytesseract", output_folder)  # write extracted text to disk<jupyter_output><empty_output><jupyter_text>Spellchecking with pyspellcheckerThis model will classify each word as misspelled if it isn't contained in a spanish vocabulary.Then it uses a Levenshtein Distance algorithm to find permutations within an edit distance of 2 from the original word. It then compares all permutations (insertions, deletions, replacements, and transpositions) to known words in a word frequency list. Those words that are found more often in the frequency list are more likely the correct results.<jupyter_code>words = word_tokenize(text)
spell = SpellChecker(language='es')

# find those words that may be misspelled
misspelled = spell.unknown(words)

for word in misspelled:
    # Get the one `most likely` answer
    corr = spell.correction(word)
    # Get a list of `likely` options
    cand = spell.candidates(word)
    
    print(f"Mispelled word: {word} - Possible candidates: {cand} - Correction: {corr}")<jupyter_output><empty_output><jupyter_text>Spellcheking using JamSpell- **Training the model is required as there's no pre-trained spanish jamspell model**. Instructions on how to do it at https://github.com/bakwc/JamSpelltrain.<jupyter_code># Example of how to use JamSpell
corrector = jamspell.TSpellCorrector()

# Check if model file exists
modelFile = 'en.bin'
if os.path.exists(modelFile):
    corrector.LoadLangModel('en.bin')  # load the trained model

    corrector.FixFragment('I am the begt spell cherken!')
    # u'I am the best spell checker!'

    corrector.GetCandidates(['i', 'am', 'the', 'begt', 'spell', 'cherken'], 3)
    # (u'best', u'beat', u'belt', u'bet', u'bent', ... )

    corrector.GetCandidates(['i', 'am', 'the', 'begt', 'spell', 'cherken'], 5)
    # (u'checker', u'chicken', u'checked', u'wherein', u'coherent', ...)
else:
    print("modelFile contains inexistent path!")<jupyter_output><empty_output><jupyter_text>Extraction for S3 Bucket folder<jupyter_code># Connection to AWS
KEY = os.environ.get("S3_BUCKET")
SECRET = os.environ.get("SECRET_KEY")

s3 = boto3.resource(
    service_name = 's3',
    region_name = 'us-east-2',
    aws_access_key_id = KEY,
    aws_secret_access_key = SECRET
)
folder = 1 #This is the number of the folder in the S3 Bucket there are 10, from 1 to 10
i = 0
for obj in s3.Bucket('wri-latin-talent').objects.all().filter(Prefix='full'):
    if set_folder_tag(folder) in obj.key and obj.key.replace("full/" + set_folder_tag(folder), "") != "": #To run only over the desired folders
        print(i, "**", obj.key)
        key = "text-extraction/" + obj.key.replace("full/"+  set_folder_tag(folder), "") + ".txt"
        file = obj.get()['Body'].read() #get the file from S3
        pages = get_image2(file)
        text = ""  # initialize document text
        for pageNum, imgBlob in enumerate(pages):
            page_text = pytesseract.image_to_string(imgBlob, lang="spa")
            text += wrap_pagenum(page_text, pageNum)  # wrap page number
        content = split_paragraphs(text)  # extract text
        s3.Object('wri-latin-talent', key).put(Body = content)#This will save all the contents in the string variable "content" into a txt file in the Pre-processed folder
        i += 1<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup