│   45 │   │   │   except Exception as e:                                                          │
│   46 │   │   │   │   raise RuntimeError(f"Error loading {self.file_path}") from e                │
│   47                                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Error loading ../../../../../tests/integration_tests/examples/example-non-utf8.txt
The file example-non-utf8.txt uses a different encoding the load() function fails with a helpful message indicating which file failed decoding.
With the default behavior of TextLoader any failure to load any of the documents will fail the whole loading process and no documents are loaded.
B. Silent fail#
We can pass the parameter silent_errors to the DirectoryLoader to skip the files which could not be loaded and continue the load process.
loader = DirectoryLoader(path, glob="**/*.txt", loader_cls=TextLoader, silent_errors=True)
docs = loader.load()
Error loading ../../../../../tests/integration_tests/examples/example-non-utf8.txt
doc_sources = [doc.metadata['source']  for doc in docs]
doc_sources
['../../../../../tests/integration_tests/examples/whatsapp_chat.txt',
 '../../../../../tests/integration_tests/examples/example-utf8.txt']
C. Auto detect encodings#
We can also ask TextLoader to auto detect the file encoding before failing, by passing the autodetect_encoding to the loader class.
text_loader_kwargs={'autodetect_encoding': True}
loader = DirectoryLoader(path, glob="**/*.txt", loader_cls=TextLoader, loader_kwargs=text_loader_kwargs)
docs = loader.load()
doc_sources = [doc.metadata['source']  for doc in docs]
doc_sources
['../../../../../tests/integration_tests/examples/example-non-utf8.txt',
 '../../../../../tests/integration_tests/examples/whatsapp_chat.txt',
rtdocs_stable/api.python.langchain.com/en/stable/modules/indexes/document_loaders/examples/file_directory.html