|         20|       1|     3|Masselmani, Mrs. ...|female|null|    0|    0|            2649|  7.225| null|       C|
+-----------+--------+------+--------------------+------+----+-----+-----+----------------+-------+-----+--------+
only showing top 20 rows
from langchain.agents import create_spark_dataframe_agent
from langchain.llms import OpenAI
import os
os.environ["OPENAI_API_KEY"] = "...input your openai api key here..."
agent = create_spark_dataframe_agent(llm=OpenAI(temperature=0), df=df, verbose=True)
agent.run("""
who bought the most expensive ticket?
You can find all supported function types in https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/dataframe.html
""")
> Entering new AgentExecutor chain...
Thought: I need to find the row with the highest fare
Action: python_repl_ast
Action Input: df.sort(df.Fare.desc()).first()
Observation: Row(PassengerId=259, Survived=1, Pclass=1, Name='Ward, Miss. Anna', Sex='female', Age=35.0, SibSp=0, Parch=0, Ticket='PC 17755', Fare=512.3292, Cabin=None, Embarked='C')
Thought: I now know the name of the person who bought the most expensive ticket
Final Answer: Miss. Anna Ward
> Finished chain.
'Miss. Anna Ward'
spark.stop()
previous
Python Agent
next
Spark SQL Agent
 Contents
  
Spark Connect Example
By Harrison Chase
    
      © Copyright 2023, Harrison Chase.
      
  Last updated on Jun 16, 2023.
rtdocs_stable/api.python.langchain.com/en/stable/modules/agents/toolkits/examples/spark.html