<jupyter_start><jupyter_text>Putting It All TogetherIn the previous notebooks, I've written code for a webcrawler that pulls articles related to Biblical archeology, extracts the text from those articles, creates a list of each the specific Bible verse references, and predicts if the website is truly relevant. In this notebook, I would like to put all of that together in a single program that I can then use to create a dataset in which each relevant article is connected directly to the Bible verses that are mentioned in that article. The end goal is to create a resource that enriches Bible study with archeological discoveries.<jupyter_code>import pandas as pd
from googlesearch import search 
import time<jupyter_output><empty_output><jupyter_text>I think the best way to encapulate what I've done before is by defining functions that can be used in the flow of this notebook. The firt function I want to define is a webcrawler that accepts a keyword argument and then searches for websites that are related to that keyword and Biblical archeology.This site describes the google module: https://towardsdatascience.com/current-google-search-packages-using-python-3-7-a-simple-tutorial-3606e459e0d4<jupyter_code>def crawler(keyword, limit):
    websites = []
    
    query = "archeololgy OR artifact + Bible" + keyword
    
    for j in search(query, stop=limit):
        time.sleep(3)        
        websites.append(j)
        
    return pd.Series(websites)
site_list = crawler('Israel', 5)
site_list
type(site_list)<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup