Thought: I now have the math library imported, I can get the square root
Action: python_repl_ast
Action Input: math.sqrt(29.69911764705882)
Observation: 5.449689683556195
Thought: I now know the final answer
Final Answer: 5.449689683556195
> Finished chain.
'5.449689683556195'
spark.stop()
Spark Connect Example#
# in apache-spark root directory. (tested here with "spark-3.4.0-bin-hadoop3 and later")
# To launch Spark with support for Spark Connect sessions, run the start-connect-server.sh script.
!./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_2.12:3.4.0
from pyspark.sql import SparkSession
# Now that the Spark server is running, we can connect to it remotely using Spark Connect. We do this by 
# creating a remote Spark session on the client where our application runs. Before we can do that, we need 
# to make sure to stop the existing regular Spark session because it cannot coexist with the remote 
# Spark Connect session we are about to create.
SparkSession.builder.master("local[*]").getOrCreate().stop()
23/05/08 10:06:09 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
# The command we used above to launch the server configured Spark to run as localhost:15002. 
# So now we can create a remote Spark session on the client using the following command.
spark = SparkSession.builder.remote("sc://localhost:15002").getOrCreate()
csv_file_path = "titanic.csv"
df = spark.read.csv(csv_file_path, header=True, inferSchema=True)
df.show()
rtdocs_stable/api.python.langchain.com/en/stable/modules/agents/toolkits/examples/spark.html