Filtering sitemap URLs#
Sitemaps can be massive files, with thousands of URLs.  Often you don’t need every single one of them.  You can filter the URLs by passing a list of strings or regex patterns to the url_filter parameter.  Only URLs that match one of the patterns will be loaded.
loader = SitemapLoader(
    "https://langchain.readthedocs.io/sitemap.xml",
    filter_urls=["https://python.langchain.com/en/latest/"]
)
documents = loader.load()
documents[0]
rtdocs_stable/api.python.langchain.com/en/stable/modules/indexes/document_loaders/examples/sitemap.html