Maintaining Cost-Efficiency
Ensuring Rapid Iteration
Understanding these components is crucial when assessing serving systems. LangChain integrates with several open-source projects designed to tackle these issues, providing a robust framework for productionizing your LLM applications. Some notable frameworks include:
Ray Serve
BentoML
Modal
These links will provide further information on each ecosystem, assisting you in finding the best fit for your LLM deployment needs.
Designing a Robust LLM Application Service#
When deploying an LLM service in production, it’s imperative to provide a seamless user experience free from outages. Achieving 24/7 service availability involves creating and maintaining several sub-systems surrounding your application.
Monitoring#
Monitoring forms an integral part of any system running in a production environment. In the context of LLMs, it is essential to monitor both performance and quality metrics.
Performance Metrics: These metrics provide insights into the efficiency and capacity of your model. Here are some key examples:
Query per second (QPS): This measures the number of queries your model processes in a second, offering insights into its utilization.
Latency: This metric quantifies the delay from when your client sends a request to when they receive a response.
Tokens Per Second (TPS): This represents the number of tokens your model can generate in a second.
Quality Metrics: These metrics are typically customized according to the business use-case. For instance, how does the output of your system compare to a baseline, such as a previous version? Although these metrics can be calculated offline, you need to log the necessary data to use them later.
Fault tolerance#
rtdocs_stable/api.python.langchain.com/en/stable/additional_resources/deploy_llms.html