.rst
.pdf
Deploying LLMs in Production
 Contents 
Outline
Designing a Robust LLM Application Service
Monitoring
Fault tolerance
Zero down time upgrade
Load balancing
Maintaining Cost-Efficiency and Scalability
Self-hosting models
Resource Management and Auto-Scaling
Utilizing Spot Instances
Independent Scaling
Batching requests
Ensuring Rapid Iteration
Model composition
Cloud providers
Infrastructure as Code (IaC)
CI/CD
Deploying LLMs in Production#
In today’s fast-paced technological landscape, the use of Large Language Models (LLMs) is rapidly expanding. As a result, it’s crucial for developers to understand how to effectively deploy these models in production environments. LLM interfaces typically fall into two categories:
Case 1: Utilizing External LLM Providers (OpenAI, Anthropic, etc.)In this scenario, most of the computational burden is handled by the LLM providers, while LangChain simplifies the implementation of business logic around these services. This approach includes features such as prompt templating, chat message generation, caching, vector embedding database creation, preprocessing, etc.
Case 2: Self-hosted Open-Source ModelsAlternatively, developers can opt to use smaller, yet comparably capable, self-hosted open-source LLM models. This approach can significantly decrease costs, latency, and privacy concerns associated with transferring data to external LLM providers.
Regardless of the framework that forms the backbone of your product, deploying LLM applications comes with its own set of challenges. It’s vital to understand the trade-offs and key considerations when evaluating serving frameworks.
Outline#
This guide aims to provide a comprehensive overview of the requirements for deploying LLMs in a production setting, focusing on:
Designing a Robust LLM Application Service
Maintaining Cost-Efficiency
Ensuring Rapid Iteration
rtdocs_stable/api.python.langchain.com/en/stable/additional_resources/deploy_llms.html