<jupyter_start><jupyter_text>Identify 100 users with highest number of ratings/fans.  - User dataset- You can decide/justify this ranking and decide the importance of ratings/fans. - Focusing on fans is better.   Step 1: - Load the data:  - We can use SQL: SQlContext<jupyter_code># import SQLContext and create a sqlContext
from pyspark.sql import SQLContext
sqlContext = SQLContext(sc)
# get SQL table
df_json = sqlContext.read.json("../data/yelp/yelp_academic_dataset_user.json.gz")
df_json.printSchema()
df_json.registerTempTable("tbl_json")<jupyter_output><empty_output><jupyter_text>- PySpark way<jupyter_code>df_data = sc.textFile("../data/yelp/yelp_academic_dataset_user.json.gz")
print(df_data.count())
df_data.take(1)<jupyter_output>1968703<jupyter_text>Step 2:  - Get the information using SQL Statement<jupyter_code>df_json.printSchema()
# 100 users with highest number of ratings/fans.
highest_fan = sqlContext.sql("SELECT user_id, review_count, fans FROM tbl_json ORDER BY fans DESC LIMIT 100")
highest_fan.show()<jupyter_output>+--------------------+------------+-----+
|             user_id|review_count| fans|
+--------------------+------------+-----+
|37cpUoM8hlkSQfReI...|        1787|11568|
|hizGc5W1tBHPghM5Y...|        1629| 3315|
|eKUGKQRE-Ywi5dY55...|        2430| 2916|
|Hi10sGSZNxQH3NLyW...|       11112| 2718|
|j14WgRoU_-2ZE1aw1...|        3566| 2634|
|iLjMdZi0Tm7DQxX1C...|        2431| 2516|
|JjXuiru1_ONzDkYVr...|        1207| 2316|
|ITa3vh5ERI90G_WP4...|        3079| 2280|
|UsXqCXRZwSCSw0AT7...|        2919| 2263|
|VHdY6oG2JPVNjihWh...|        1979| 2140|
|fgwI3rYHOv1ipfVfC...|        1847| 2113|
|nkN_do3fJ9xekchVC...|        1363| 2046|
|m07sy7eLtOjVdZ8oN...|        4913| 2034|
|peuxbSQwXed-81cSq...|        1592| 1916|
|AHRrG3T1gJpHvtpZ-...|        2109| 1747|
|WG3w_73scm_JUWJ_3...|        5013| 1733|
|NfU0zDaTMEQ4-X9db...|        1137| 1726|
|lt7bNHl-TXziny4FE...|         802| 1708|
|3zxy3LVBV3ttxoYbY...|        1894| 1696|
|ysCBsXWPB-LAiewVS...|        3687| 1672|
+--------------------+------------[...]<jupyter_text>- Get the information using PySpark Statement<jupyter_code>df_data.take(1)
import json

df_top_users = df_data.map(lambda x: json.loads(x)).map(lambda x: (x["user_id"], x["review_count"], x["fans"]))
df_top_users.take(1)
def extract_user(x):
    x_json = json.loads(x)
    return (x_json["user_id"], x_json["review_count"], x_json["fans"])

tmp = df_data.take(1)
extract_user(tmp[0])
df_top_users = df_data.map(extract_user).takeOrdered(100, lambda x: -x[2])
df_top_users
highest_fan.show()<jupyter_output>+--------------------+------------+-----+
|             user_id|review_count| fans|
+--------------------+------------+-----+
|37cpUoM8hlkSQfReI...|        1787|11568|
|hizGc5W1tBHPghM5Y...|        1629| 3315|
|eKUGKQRE-Ywi5dY55...|        2430| 2916|
|Hi10sGSZNxQH3NLyW...|       11112| 2718|
|j14WgRoU_-2ZE1aw1...|        3566| 2634|
|iLjMdZi0Tm7DQxX1C...|        2431| 2516|
|JjXuiru1_ONzDkYVr...|        1207| 2316|
|ITa3vh5ERI90G_WP4...|        3079| 2280|
|UsXqCXRZwSCSw0AT7...|        2919| 2263|
|VHdY6oG2JPVNjihWh...|        1979| 2140|
|fgwI3rYHOv1ipfVfC...|        1847| 2113|
|nkN_do3fJ9xekchVC...|        1363| 2046|
|m07sy7eLtOjVdZ8oN...|        4913| 2034|
|peuxbSQwXed-81cSq...|        1592| 1916|
|AHRrG3T1gJpHvtpZ-...|        2109| 1747|
|WG3w_73scm_JUWJ_3...|        5013| 1733|
|NfU0zDaTMEQ4-X9db...|        1137| 1726|
|lt7bNHl-TXziny4FE...|         802| 1708|
|3zxy3LVBV3ttxoYbY...|        1894| 1696|
|ysCBsXWPB-LAiewVS...|        3687| 1672|
+--------------------+------------[...]
StarcoderdataJupyterStructuredCleanDedup