Add custom scraping rules#
The SitemapLoader uses beautifulsoup4 for the scraping process, and it scrapes every element on the page by default. The SitemapLoader constructor accepts a custom scraping function. This feature can be helpful to tailor the scraping process to your specific needs; for example, you might want to avoid scraping headers or navigation elements.
The following example shows how to develop and use a custom function to avoid navigation and header elements.
Import the beautifulsoup4 library and define the custom function.
pip install beautifulsoup4
from bs4 import BeautifulSoup
def remove_nav_and_header_elements(content: BeautifulSoup) -> str:
    # Find all 'nav' and 'header' elements in the BeautifulSoup object
    nav_elements = content.find_all('nav')
    header_elements = content.find_all('header')
    # Remove each 'nav' and 'header' element from the BeautifulSoup object
    for element in nav_elements + header_elements:
        element.decompose()
    return str(content.get_text())
Add your custom function to the SitemapLoader object.
loader = SitemapLoader(
    "https://langchain.readthedocs.io/sitemap.xml",
    filter_urls=["https://python.langchain.com/en/latest/"],
    parsing_function=remove_nav_and_header_elements
)
Local Sitemap#
The sitemap loader can also be used to load local files.
sitemap_loader = SitemapLoader(web_path="example_data/sitemap.xml", is_local=True)
docs = sitemap_loader.load()
Fetching pages: 100%|####################################################################################################################################| 3/3 [00:00<00:00,  3.91it/s]
previous
PDF
next
Subtitle
 Contents
  
Filtering sitemap URLs
Add custom scraping rules
Local Sitemap
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
rtdocs_stable/api.python.langchain.com/en/stable/modules/indexes/document_loaders/examples/sitemap.html