from langchain.embeddings import HuggingFaceHubEmbeddings
For a more detailed walkthrough of this, see this notebook
Tokenizer#
There are several places you can use tokenizers available through the transformers package.
By default, it is used to count tokens for all LLMs.
You can also use it to count tokens when splitting documents with
from langchain.text_splitter import CharacterTextSplitter
CharacterTextSplitter.from_huggingface_tokenizer(...)
For a more detailed walkthrough of this, see this notebook
Datasets#
The Hugging Face Hub has lots of great datasets that can be used to evaluate your LLM chains.
For a detailed walkthrough of how to use them to do so, see this notebook
previous
Helicone
next
iFixit
 Contents
  
Installation and Setup
Wrappers
LLM
Embeddings
Tokenizer
Datasets
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on Jun 16, 2023.
rtdocs_stable/api.python.langchain.com/en/stable/integrations/huggingface.html