Combine lines with common metadata into chunks
:param lines: Line of text / associated header metadata
split_text(text: str) → List[langchain.text_splitter.LineType][source]#
Split markdown file
:param text: Markdown file
class langchain.text_splitter.MarkdownTextSplitter(**kwargs: Any)[source]#
Attempts to split the text along Markdown-formatted headings.
class langchain.text_splitter.NLTKTextSplitter(separator: str = '\n\n', **kwargs: Any)[source]#
Implementation of splitting text that looks at sentences using NLTK.
split_text(text: str) → List[str][source]#
Split incoming text and return chunks.
class langchain.text_splitter.PythonCodeTextSplitter(**kwargs: Any)[source]#
Attempts to split the text along Python syntax.
class langchain.text_splitter.RecursiveCharacterTextSplitter(separators: Optional[List[str]] = None, keep_separator: bool = True, **kwargs: Any)[source]#
Implementation of splitting text that looks at characters.
Recursively tries to split by different characters to find one
that works.
classmethod from_language(language: langchain.text_splitter.Language, **kwargs: Any) → langchain.text_splitter.RecursiveCharacterTextSplitter[source]#
static get_separators_for_language(language: langchain.text_splitter.Language) → List[str][source]#
split_text(text: str) → List[str][source]#
Split text into multiple components.
class langchain.text_splitter.SentenceTransformersTokenTextSplitter(chunk_overlap: int = 50, model_name: str = 'sentence-transformers/all-mpnet-base-v2', tokens_per_chunk: Optional[int] = None, **kwargs: Any)[source]#
Implementation of splitting text that looks at tokens.
rtdocs_stable/api.python.langchain.com/en/stable/reference/modules/text_splitter.html