.ipynb
.pdf
DocArrayHnswSearch
 Contents 
Setup
Using DocArrayHnswSearch
Similarity search
Similarity search with score
DocArrayHnswSearch#
DocArrayHnswSearch is a lightweight Document Index implementation provided by Docarray that runs fully locally and is best suited for small- to medium-sized datasets. It stores vectors on disk in hnswlib, and stores all other data in SQLite.
This notebook shows how to use functionality related to the DocArrayHnswSearch.
Setup#
Uncomment the below cells to install docarray and get/set your OpenAI api key if you haven’t already done so.
# !pip install "docarray[hnswlib]"
# Get an OpenAI token: https://platform.openai.com/account/api-keys
# import os
# from getpass import getpass
# OPENAI_API_KEY = getpass()
# os.environ["OPENAI_API_KEY"] = OPENAI_API_KEY
Using DocArrayHnswSearch#
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import DocArrayHnswSearch
from langchain.document_loaders import TextLoader
documents = TextLoader('../../../state_of_the_union.txt').load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)
embeddings = OpenAIEmbeddings()
db = DocArrayHnswSearch.from_documents(docs, embeddings, work_dir='hnswlib_store/', n_dim=1536)
Similarity search#
query = "What did the president say about Ketanji Brown Jackson"
docs = db.similarity_search(query)
print(docs[0].page_content)
rtdocs_stable/api.python.langchain.com/en/stable/modules/indexes/vectorstores/examples/docarray_hnsw.html