docs = loader.load()
len(docs)
1
If you need to load Python source code files, use the PythonLoader.
from langchain.document_loaders import PythonLoader
loader = DirectoryLoader('../../../../../', glob="**/*.py", loader_cls=PythonLoader)
docs = loader.load()
len(docs)
691
Auto detect file encodings with TextLoader#
In this example we will see some strategies that can be useful when loading a big list of arbitrary files from a directory using the TextLoader class.
First to illustrate the problem, let’s try to load multiple text with arbitrary encodings.
path = '../../../../../tests/integration_tests/examples'
loader = DirectoryLoader(path, glob="**/*.txt", loader_cls=TextLoader)
A. Default Behavior#
loader.load()
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /data/source/langchain/langchain/document_loaders/text.py:29 in load                             │
│                                                                                                  │
│   26 │   │   text = ""                                                                           │
│   27 │   │   with open(self.file_path, encoding=self.encoding) as f:                             │
│   28 │   │   │   try:                                                                            │
│ ❱ 29 │   │   │   │   text = f.read()                                                             │
│   30 │   │   │   except UnicodeDecodeError as e:                                                 │
│   31 │   │   │   │   if self.autodetect_encoding:                                                │
│   32 │   │   │   │   │   detected_encodings = self.detect_file_encodings()                       │
│                                                                                                  │
│ /home/spike/.pyenv/versions/3.9.11/lib/python3.9/codecs.py:322 in decode                         │
rtdocs_stable/api.python.langchain.com/en/stable/modules/indexes/document_loaders/examples/file_directory.html