.ipynb
.pdf
Evaluating an OpenAPI Chain
 Contents 
Load the API Chain
Optional: Generate Input Questions and Request Ground Truth Queries
Run the API Chain
Evaluate the requests chain
Evaluate the Response Chain
Generating Test Datasets
Evaluating an OpenAPI Chain#
This notebook goes over ways to semantically evaluate an OpenAPI Chain, which calls an endpoint defined by the OpenAPI specification using purely natural language.
from langchain.tools import OpenAPISpec, APIOperation
from langchain.chains import OpenAPIEndpointChain, LLMChain
from langchain.requests import Requests
from langchain.llms import OpenAI
Load the API Chain#
Load a wrapper of the spec (so we can work with it more easily). You can load from a url or from a local file.
# Load and parse the OpenAPI Spec
spec = OpenAPISpec.from_url("https://www.klarna.com/us/shopping/public/openai/v0/api-docs/")
# Load a single endpoint operation
operation = APIOperation.from_openapi_spec(spec, '/public/openai/v0/products', "get")
verbose = False
# Select any LangChain LLM
llm = OpenAI(temperature=0, max_tokens=1000)
# Create the endpoint chain
api_chain = OpenAPIEndpointChain.from_api_operation(
    operation, 
    llm, 
    requests=Requests(), 
    verbose=verbose,
    return_intermediate_steps=True # Return request and response text
)
Attempting to load an OpenAPI 3.0.1 spec.  This may result in degraded performance. Convert your OpenAPI spec to 3.1.* spec for better support.
Optional: Generate Input Questions and Request Ground Truth Queries#
See Generating Test Datasets at the end of this notebook for more details.
# import re
rtdocs_stable/api.python.langchain.com/en/stable/use_cases/evaluation/openapi_eval.html