.rst
.pdf
Text Splitters
Text Splitters#
Note
Conceptual Guide
When you want to deal with long pieces of text, it is necessary to split up that text into chunks.
As simple as this sounds, there is a lot of potential complexity here. Ideally, you want to keep the semantically related pieces of text together. What “semantically related” means could depend on the type of text.
This notebook showcases several ways to do that.
At a high level, text splitters work as following:
Split the text up into small, semantically meaningful chunks (often sentences).
Start combining these small chunks into a larger chunk until you reach a certain size (as measured by some function).
Once you reach that size, make that chunk its own piece of text and then start creating a new chunk of text with some overlap (to keep context between chunks).
That means there are two different axes along which you can customize your text splitter:
How the text is split
How the chunk size is measured
For an introduction to the default text splitter and generic functionality see:
Getting Started
Usage examples for the text splitters:
Character
Code (including HTML, Markdown, Latex, Python, etc)
NLTK
Recursive Character
spaCy
tiktoken (OpenAI)
Most LLMs are constrained by the number of tokens that you can pass in, which is not the same as the number of characters.
In order to get a more accurate estimate, we can use tokenizers to count the number of tokens in the text.
We use this number inside the ..TextSplitter classes.
This implemented as the from_<tokenizer> methods of the ..TextSplitter classes:
Hugging Face tokenizer
tiktoken (OpenAI) tokenizer
previous
Twitter
next
Getting Started
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
rtdocs_stable/api.python.langchain.com/en/stable/modules/indexes/text_splitters.html