.ipynb
.pdf
Tiktoken
Tiktoken#
tiktoken is a fast BPE tokeniser created by OpenAI.
How the text is split: by tiktoken tokens
How the chunk size is measured: by tiktoken tokens
#!pip install tiktoken
# This is a long document we can split up.
with open('../../../state_of_the_union.txt') as f:
    state_of_the_union = f.read()
from langchain.text_splitter import TokenTextSplitter
text_splitter = TokenTextSplitter(chunk_size=10, chunk_overlap=0)
texts = text_splitter.split_text(state_of_the_union)
print(texts[0])
Madam Speaker, Madam Vice President, our
previous
spaCy
next
Hugging Face tokenizer
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on Jun 16, 2023.
rtdocs_stable/api.python.langchain.com/en/stable/modules/indexes/text_splitters/examples/tiktoken_splitter.html