{'question': 'How many employees are there?', 'answer': '8'}
Setting up a chain#
This uses the example Chinook database.
To set it up follow the instructions on https://database.guide/2-sample-databases-sqlite/, placing the .db file in a notebooks folder at the root of this repository.
Note that here we load a simple chain. If you want to experiment with more complex chains, or an agent, just create the chain object in a different way.
from langchain import OpenAI, SQLDatabase, SQLDatabaseChain
db = SQLDatabase.from_uri("sqlite:///../../../notebooks/Chinook.db")
llm = OpenAI(temperature=0)
Now we can create a SQL database chain.
chain = SQLDatabaseChain.from_llm(llm, db, input_key="question")
Make a prediction#
First, we can make predictions one datapoint at a time. Doing it at this level of granularity allows use to explore the outputs in detail, and also is a lot cheaper than running over multiple datapoints
chain(dataset[0])
{'question': 'How many employees are there?',
 'answer': '8',
 'result': ' There are 8 employees.'}
Make many predictions#
Now we can make predictions. Note that we add a try-except because this chain can sometimes error (if SQL is written incorrectly, etc)
predictions = []
predicted_dataset = []
error_dataset = []
for data in dataset:
    try:
        predictions.append(chain(data))
        predicted_dataset.append(data)
    except:
        error_dataset.append(data)
Evaluate performance#
Now we can evaluate the predictions. We can use a language model to score them programatically
from langchain.evaluation.qa import QAEvalChain
llm = OpenAI(temperature=0)
rtdocs_stable/api.python.langchain.com/en/stable/use_cases/evaluation/sql_qa_benchmarking_chinook.html