<jupyter_start><jupyter_text>Run all cells at once , in top menu select Kernel > Restart and Run All Code Submission for CMAP Assignment.By <NAME> (<EMAIL>)For this assignment, the entire process has been done in Python on Jupiter Notebook. The required Shapefile and Table (.csv) could be exported by running the entire pipeline. These files are also enclosed in the main directory for this notebook.This notebook comprises four parts. 1) Data Import and Processing (Most requirements of the assignment are covered here)2) Data Analysis - Prelimnary Analysis of findings.3) Data Validation - Validating the spatial join and process used.4) Data Visualization- Sample elements for dashboard.(Note: Some extra steps in the process have been extended over multiple cells, for easy understanding.)Date- 3/2/2021Data Credits-1) DIVVY Chicago&nbsp; &nbsp; &nbsp;2) CMAP&nbsp; &nbsp; &nbsp; &nbsp; 3) U.S Census BureauPhoto Credit: Patch.com Link to Summary Document- https://docs.google.com/document/d/1XWj_Wyq5NKJFvwGFLnYybXveObSOEx_JW9IwfREGAYg/edit?usp=sharingLink to PDF version of Jupyter Notebook - https://drive.google.com/file/d/13e4aqILYq8ZA-pXzyGqKyhTXLgk0vqHk/view?usp=sharingLink to Resume- https://docs.google.com/document/d/1_munGYl_xZVTrLQxekoZFIBKCfn3lM9VLXp6Ff56OmY/edit?usp=sharing  1) Data Import and Processing<jupyter_code>%%capture
!pip install rtree
!pip install descartes
# import the required Python Libraries
import requests
import matplotlib.pyplot as plt
import pandas as pd
import warnings
import geopandas as gpd
warnings.filterwarnings('ignore')
import ipywidgets as widgets
from ipywidgets import interact, interactive, fixed, interact_manual
import mapclassify
import rtree
print("All Good!")<jupyter_output>All Good!<jupyter_text>The JSON file comprising of the Dock information from the DIVVY website comprises of data in Dictionary form. Prelimnary visual analysis for identification of data structure and types is being done below to organize the data in a pandas dataframe for easy analysis and processing.<jupyter_code># Loading the JSON file
url = 'https://gbfs.divvybikes.com/gbfs/en/station_information.json'
data = requests.get(url).json()

#print(data)
print(type(data))
# Selecting the sub-classes of the Python dictionary using keys.
data_list = data['data']['stations']
# Convert the dictionary to Pandas Dataframe.
df_full = pd.DataFrame(data_list)
# Dataframe for complete dataset including unwanted columns.
df_full.head(5)
#New Dataframe comprising of only desired columns. Being stored as df.
df = df_full[['station_id', 'name', 'capacity', 'lat', 'lon']]
df.rename(columns = {'capacity':'DOCKS'}, inplace = True)
#Checking the new dataframe structure.
df.head()
# verifying for any NULL or NaN values.
df.info()
# Extracting the lat, lon from the dataframe to form the Geometry

from shapely.geometry import Point

# combine lat and lon column to a shapely Point() object
df['Geometry'] = df.apply(lambda x: Point((float(x.lon), float(x.lat))), axis=1)
# Loading the dataframe into geopandas along with the Geometry
df_shape = gpd.GeoDataFrame(df, geometry='Geometry')

# Inititalizing the Coordinate System to WGS 84 using identifier 4326
df_shape.crs = {'init' :'epsg:4326'}
# Exporting the geopandas dataframe as Shapefile.
df_shape.to_file('divvy_stations.shp', driver='ESRI Shapefile')
# Visualizing the exported Shapefile using geopandas.

gdf = gpd.read_file('divvy_stations.shp')

%matplotlib inline
fig, ax = plt.subplots(figsize=(15,10))

gdf.plot(
    column='DOCKS', ax=ax, legend=True,  scheme='quantiles', cmap='OrRd')
plt.title(('DIVVY Station Capacity Across Chicago, IL'), color ='black', fontsize='15');<jupyter_output><empty_output><jupyter_text>For this assignment we will directly import the desired shapefiles into Python environment from the provided zip file ie Datasets.zip<jupyter_code># Importing the Shapefiles using Geopandas.

CCA = gpd.read_file('zip://Datasets.zip!Datasets/ChicagoCommunityAreas.shp')
CCB= gpd.read_file('zip://Datasets.zip!Datasets/CensusBlocks_Population2010.shp')
DIVVY_ST = gpd.read_file('divvy_stations.shp')
# Verifying the Attributes of the ChicagoCommunityAreas using GeoPandas dataframe.
CCA.head(5)
# Verifying the Attributes of the CensusBlocks_Population2010 using GeoPandas dataframe.
CCB.head(5)
# Checking the Cordinate System for the two shapefiles.
CCA.crs, CCB.crs

# these are different from the Divvy Dock file we prepared previously (epsg 4326)
# Re-projecting coordinate reference systems of the dataframes to epsg:4326 (WGS94)
CCA=CCA.to_crs({'init':'epsg:4326'})
CCB= CCB.to_crs({'init':'epsg:4326'})
DIVVY_ST = DIVVY_ST.to_crs({'init':'epsg:4326'}) # not needed though
CCA.crs, CCB.crs, DIVVY_ST.crs<jupyter_output><empty_output><jupyter_text>We now have all our data loaded into python environment and all coordinate reference systems set up for any spatial analysis to be done correctly.First we will calculate the total population for each of the Chicago Community Area using the Spatial Join feature of GeoPandas library.<jupyter_code>#get a spatial join of ccb and cca for population cstatistics

CCA_CB=gpd.sjoin(CCA, CCB, how='left', op='contains')
CCA.shape,CCB.shape, CCA_CB.shape
# Census tracts now have a defined Community Name and Community Number.
CCA_CB.head(10)<jupyter_output><empty_output><jupyter_text>Using the above dataframe we will regroup the data and aggregare on the POP2010 column for each Community Number and Community name.<jupyter_code>CCA_pop=CCA_CB.groupby(['CCA_NUM','COMMUNITY']).agg({'POP2010': 'sum'}).reset_index() 
CCA_pop.head()
# Check the meta data for the data. Check for any None Values as possible errors.
CCA_pop.info()
# Lets take a look at out DIVVY dataframe from the shapefile we exported.
DIVVY_ST.head()<jupyter_output><empty_output><jupyter_text>We would need to do a second Spatial Join for the CCA_Pop dataframe that includes total population counts for each Community and the DIVVY_ST dataframe which has the individual dock location and information.<jupyter_code>#Perform Spatial Join
CCA_DIVVY=gpd.sjoin(CCA, DIVVY_ST, how='left', op='contains')
CCA_DIVVY.shape, CCA.shape
# We now have Community Name and Number assigned for each of the DIVVY Station.
CCA_DIVVY.head()
# Group and aggregate for capacity.
CCA_DIVVY_AGG=CCA_DIVVY.groupby(['CCA_NUM','COMMUNITY']).agg({'DOCKS': 'sum'}).reset_index()
CCA_DIVVY_AGG.head()
CCA_DIVVY_AGG.info() # Checking for any Null values.<jupyter_output><class 'pandas.core.frame.DataFrame'>
RangeIndex: 77 entries, 0 to 76
Data columns (total 3 columns):
 #   Column     Non-Null Count  Dtype  
---  ------     --------------  -----  
 0   CCA_NUM    77 non-null     int64  
 1   COMMUNITY  77 non-null     object 
 2   DOCKS      77 non-null     float64
dtypes: float64(1), int64(1), object(1)
memory usage: 1.9+ KB<jupyter_text>Since we have all data prepped, we can now merge the data and prepare a final table for export.<jupyter_code>#merging CCA_pop (Total Population for each Community Area) and CCA_DIVVY_AGG (Total dock capacity of Each Community Area) 
CCA_DIVVY_POP=pd.merge(CCA_pop,CCA_DIVVY_AGG[['DOCKS', 'CCA_NUM']], on= 'CCA_NUM',how='left') 
CCA_DIVVY_POP.shape # Checking the structure of table.
CCA_DIVVY_POP.head()
#Check for Null values
CCA_DIVVY_POP.info()
#Compute Population Per Dock.
CCA_DIVVY_POP['POP/DOCK']=CCA_DIVVY_POP['POP2010']/CCA_DIVVY_POP['DOCKS']
CCA_DIVVY_POP.head()
CCA_DIVVY_POP.info()
# Export as CSV file.
CCA_DIVVY_POP.to_csv('Chicao_Divvy_Pop_Per_Dock.csv', index=False)
# we can also download the file directly from Jupyter
import base64
import pandas as pd
from IPython.display import HTML

def create_download_link( df, title = "Download CSV file", filename = "data.csv"):
    csv = df.to_csv()
    b64 = base64.b64encode(csv.encode())
    payload = b64.decode()
    html = '<a download="{filename}" href="data:text/csv;base64,{payload}" target="_blank">{title}</a>'
    html = html.format(payload=payload,title=title,filename=filename)
    return HTML(html)

df = pd.DataFrame(data = [[1,2],[3,4]], columns=['Col 1', 'Col 2'])
create_download_link(CCA_DIVVY_POP)<jupyter_output><empty_output><jupyter_text>2) Data Analysis<jupyter_code>CCA.head()
#Combining the evaluated table with CCA data that includes spatial information.
CCA_new = pd.merge(CCA, CCA_DIVVY_POP, on=['COMMUNITY','CCA_NUM'])
CCA_new.head()
CCA_new.info()
#sample plot with the new data
%matplotlib inline
fig, ax = plt.subplots(figsize=(20,15))
df=CCA_new
df.plot(
    column='POP/DOCK', ax=ax, legend=True,  scheme='quantiles', cmap='coolwarm',
)
plt.title(('Population Per Dock By CCA'), color ='black', fontsize='20')
# In order to get labels for each Community lets extract cordinates and assign a representative point.
df['coords'] = df['geometry'].apply(lambda x: x.representative_point().coords[:])
df['coords'] = [coords[0] for coords in df['coords']]
for idx, row in df.iterrows():
    plt.annotate(s=row['CCA_NUM'], xy=row['coords'],horizontalalignment='center', color='w')
# Export as a Shapefile.
CCA_new = pd.merge(CCA, CCA_DIVVY_POP, on=['COMMUNITY','CCA_NUM'])
CCA_new.to_file('cc_divvy_pop_per_dock.shp', driver='ESRI Shapefile')<jupyter_output><empty_output><jupyter_text>3) Data ValidationTo validate the performance of spatial joins we will use the provided sample DIVVY dock stations shapefile and compare it for the same region in the dataframe evalauated above for any difference in total sum of Docks.<jupyter_code>#import the sample file
DIVVY_sample = gpd.read_file('zip://Datasets.zip!Datasets/DivvyStations_Docks20210219_Sample.shp')
DIVVY_sample=DIVVY_sample.to_crs({'init':'epsg:4326'}) #ensuring projection as wgs84
DIVVY_sample.head()
#Spatial Join Sample with CCA boundaries
CCA_DIVVY_SAM=gpd.sjoin(CCA, DIVVY_sample, how='left', op='contains')

# Group and aggregate for capacity.
CCA_DIVVY_AGG_SAM=CCA_DIVVY_SAM.groupby(['CCA_NUM','COMMUNITY']).agg({'DOCKS': 'sum'}).reset_index()
CCA_DIVVY_AGG_SAM.head(10)<jupyter_output><empty_output><jupyter_text>The sample dataset only comprised of data for 8 Communities for which the total number of DOCKS were evaluated.  Now the total of docking stations in 8 communities from Sample file and for same 8 communities from prepared dataset has been compared.<jupyter_code>#merging CCA_pop (Total Population for each Community Area) and CCA_DIVVY_AGG (Total dock capacity of Each Community Area) 
CCA_DIVVY_POP_SAM=pd.merge(CCA_pop,CCA_DIVVY_AGG_SAM[['DOCKS', 'CCA_NUM']], on= 'CCA_NUM',how='left') 

CCA_DIVVY_POP_SAMv2= CCA_DIVVY_POP_SAM.iloc[0:8] #Selecting CCA NUM from 1 to 8
TotalDocks=CCA_DIVVY_POP_SAMv2['DOCKS'].sum() #Sum of All the Docs in CCA-NUM 1 to 8
print("Total Docks based on Provided DIVVY Station Sample", TotalDocks)
CCA_DIVVY_POP_SAMv2.head(10)
#Extracting these 8 Rows of data from our evaluated dataframe CCA_DIVVY_POP. ie CCA_NUM 1-8
CCA_DIVVY_POPv2= CCA_DIVVY_POP.iloc[0:8]
TotalDocks=CCA_DIVVY_POPv2['DOCKS'].sum()
print("Total Docks based on DIVVY Station from JSON feed", TotalDocks)
CCA_DIVVY_POPv2.head(10)<jupyter_output>Total Docks based on DIVVY Station from JSON feed 3637.0<jupyter_text>Hence validated!  4) Data Visualization  In this section an attempt is made to visualize the information and make it interactive by creating dashboard elements using the data prepared.<jupyter_code>def f(Type='DOCKS'):
    df=CCA_new
    %matplotlib inline
    fig, ax = plt.subplots(figsize=(20,15))
    
    if Type=='DOCKS':
        cmap='Oranges'
        x=800
    elif Type=='POP/DOCK':
        cmap = 'coolwarm'
        x=500
    else:
        cmap= 'Reds'
        x=80000
    
    df.plot(
        column=Type, ax=ax, legend=True,  scheme='quantiles', cmap=cmap)
    plt.title((Type+ ' in 77 Chicago Community Areas (CCAs)'), color ='black', fontsize='20')
   
    # In order to get labels for each Community lets extract cordinates and assign a representative point.
    df['coords'] = df['geometry'].apply(lambda x: x.representative_point().coords[:])
    df['coords'] = [coords[0] for coords in df['coords']]
    for idx, row in df.iterrows():
        plt.annotate(s=row['CCA_NUM'], xy=row['coords'],horizontalalignment='center',color='w')
    plt.show()
print("Please select a type from the dropdown")
interactive_plot = interactive(f, Type=['POP/DOCK','POP2010', 'DOCKS'])
output = interactive_plot.children[-1]
interactive_plot<jupyter_output>Please select a type from the dropdown<jupyter_text>Mini DashBoard<jupyter_code>def f(Community='LOOP'):
    df =CCA_new.loc[CCA_new['COMMUNITY'] == Community]
    print('Name of Community:', Community)
    print('CCA Number:',df['CCA_NUM'].iloc[0])
    print('Total Population as per 2010 Census:',df['POP2010'].iloc[0])
    print('Total Docking Capacity:',df['DOCKS'].iloc[0])
    print('Population Per Dock Ratio:',df['POP/DOCK'].iloc[0])
    
    if df['DOCKS'].iloc[0] == 0:
        print("This Community has No Dock Station and hence Can't be plotted. Please select another.")
    else:
        %matplotlib inline
        fig, ax = plt.subplots(figsize=(12,10))
        ax= df.plot(column='DOCKS',ax=ax, scheme='quantiles', cmap='GnBu', label = Community)
        ax.margins(tight=False)


        df2 =CCA_DIVVY.loc[CCA_DIVVY['COMMUNITY'] == Community]
        plt.xlim(df2['lon'].min()-0.01,df2['lon'].max()+0.01)
        plt.ylim(df2['lat'].min()-0.01,df2['lat'].max()+0.01)
        plt.scatter(x=df2['lon'], y=df2['lat'], label = 'DIVVY Stations', edgecolors='k')
        ax.legend()
        plt.title(('DIVVY Stations in '+Community+' Community'), color ='black', fontsize='20')
        print('Total Stations in Community:',df2['lat'].count())
print("Please select a Community from the dropdown")
CC_names=CCA_new['COMMUNITY'].tolist()
interactive_plot = interactive(f, Community=CC_names)
output = interactive_plot.children[-1]
interactive_plot<jupyter_output>Please select a Community from the dropdown
StarcoderdataJupyterStructuredCleanDedup