.ipynb
.pdf
Ray Serve
 Contents 
Goal of this notebook
Setup Ray Serve
General Skeleton
Example of deploying and OpenAI chain with custom prompts
Ray Serve#
Ray Serve is a scalable model serving library for building online inference APIs. Serve is particularly well suited for system composition, enabling you to build a complex inference service consisting of multiple chains and business logic all in Python code.
Goal of this notebook#
This notebook shows a simple example of how to deploy an OpenAI chain into production. You can extend it to deploy your own self-hosted models where you can easily define amount of hardware resources (GPUs and CPUs) needed to run your model in production efficiently. Read more about available options including autoscaling in the Ray Serve documentation.
Setup Ray Serve#
Install ray with pip install ray[serve].
General Skeleton#
The general skeleton for deploying a service is the following:
# 0: Import ray serve and request from starlette
from ray import serve
from starlette.requests import Request
# 1: Define a Ray Serve deployment.
@serve.deployment
class LLMServe:
    def __init__(self) -> None:
        # All the initialization code goes here
        pass
    async def __call__(self, request: Request) -> str:
        # You can parse the request here
        # and return a response
        return "Hello World"
# 2: Bind the model to deployment
deployment = LLMServe.bind()
# 3: Run the deployment
serve.api.run(deployment)
# Shutdown the deployment
serve.api.shutdown()
Example of deploying and OpenAI chain with custom prompts#
Get an OpenAI API key from here. By running the following code, you will be asked to provide your API key.
from langchain.llms import OpenAI
from langchain import PromptTemplate, LLMChain
rtdocs_stable/api.python.langchain.com/en/stable/integrations/ray_serve.html