<jupyter_start><jupyter_text>Using the Spark BigQuery connector in JupyterUse a Python kernel (not PySpark) to allow configuring the SparkSession inside the notebook.<jupyter_code>from pyspark.sql import SparkSession
spark = SparkSession.builder \
  .appName('Top Shakepeare words')\
  .config('spark.jars', 'gs://spark-lib/bigquery/spark-bigquery-latest.jar') \
  .getOrCreate()<jupyter_output><empty_output><jupyter_text>Read the data in from BigQuery<jupyter_code>df = spark.read \
  .format('bigquery') \
  .load('bigquery-public-data.samples.shakespeare')<jupyter_output><empty_output><jupyter_text>Convert words to lower case and filter out stop words<jupyter_code>from pyspark.ml.feature import StopWordsRemover
from pyspark.sql import functions as F

df = df.withColumn('lowered', F.array(F.lower(df.word)))

remover = StopWordsRemover(inputCol='lowered', outputCol='filtered')

df = remover.transform(df)<jupyter_output><empty_output><jupyter_text>Create `(count, word)` struct and take the max of that in each corpus<jupyter_code>df.select(df.corpus, F.struct(df.word_count, df.filtered.getItem(0).alias('word')).alias('count_word')) \
  .where(F.col('count_word').getItem('word').isNotNull()) \
  .groupby('corpus') \
  .agg({'count_word': 'max'}) \
  .orderBy('corpus') \
  .select(
     'corpus',
     F.col('max(count_word)').getItem('word').alias('word'),
     F.col('max(count_word)').getItem('word_count').alias('count')) \
  .toPandas()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup