.ipynb
.pdf
Llama-cpp
 Contents 
Installation
CPU only installation
Installation with OpenBLAS / cuBLAS / CLBlast
Usage
CPU
GPU
Llama-cpp#
llama-cpp is a Python binding for llama.cpp.
It supports several LLMs.
This notebook goes over how to run llama-cpp within LangChain.
Installation#
There is a banch of options how to install the llama-cpp package:
only CPU usage
CPU + GPU (using one of many BLAS backends)
CPU only installation#
!pip install llama-cpp-python
Installation with OpenBLAS / cuBLAS / CLBlast#
lama.cpp supports multiple BLAS backends for faster processing. Use the FORCE_CMAKE=1 environment variable to force the use of cmake and install the pip package for the desired BLAS backend (source).
Example installation with cuBLAS backend:
!CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python
IMPORTANT: If you have already installed a cpu only version of the package, you need to reinstall it from scratch: condiser the following command:
!CMAKE_ARGS="-DLLAMA_CUBLAS=on" FORCE_CMAKE=1 pip install --upgrade --force-reinstall llama-cpp-python
Usage#
Make sure you are following all instructions to install all necessary model files.
You don’t need an API_TOKEN!
from langchain.llms import LlamaCpp
from langchain import PromptTemplate, LLMChain
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
Consider using a template that suits your model! Check the models page on HuggingFace etc. to get a correct prompting template.
template = """Question: {question}
rtdocs_stable/api.python.langchain.com/en/stable/modules/models/llms/integrations/llamacpp.html