<jupyter_start><jupyter_text>Playing with data frames in a DataBricks cluster This notebook needs a databricks cluster to execute. Cannot run on local Jupyter notebook<jupyter_code>mytable = spark.table("summary2015")
display(mytable.select("*"))
dbutils.fs.mount(
  source = "wasbs://flight-data<EMAIL>/",
  mount_point = "/mnt/flight-data",
  extra_configs = {"fs.azure.account.key.datasets4ml.blob.core.windows.net": "<KEY>})

flightData2015=spark\
  .read\
  .option("inferSchema","true")\
  .option("header","true")\
  .csv("/mnt/flight-data/2015-summary.csv")
flightData2015.take(3)
flightData2015.sort("count").explain()
spark.conf.set("spark.sql.shuffle.partitions","5")
flightData2015.sort("count").take(2)
flightData2015.createOrReplaceTempView("flight_data_2015")
sqlWay=spark.sql("""
              SELECT DEST_COUNTRY_NAME, count(1) 
              FROM flight_data_2015
              GROUP BY DEST_COUNTRY_NAME
              """)

dataFrameWay=flightData2015\
            .groupBy("DEST_COUNTRY_NAME")\
            .count()
#display(sqlWay)
#display(dataFrameWay)

sqlWay.explain()
dataFrameWay.explain()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup