<jupyter_start><jupyter_text>Data Science Blog post  The Dataset: Immigration to Canada from 1980 to 2013  Dataset Source: [International migration flows to and from selected countries - The 2015 revision](http://www.un.org/en/development/desa/population/migration/data/empirical2/migrationflows.shtml).The dataset contains annual data on the flows of international immigrants as recorded by the countries of destination. The data presents both inflows and outflows according to the place of birth, citizenship or place of previous / next residence both for foreigners and nationals. The current version presents data pertaining to 45 countries.In this notebook, we will focus on the Canadian immigration data.Data can be fetched the data from [here](https://ibm.box.com/shared/static/lw190pt9zpy5bd1ptyg2aw15awomz9pu.xlsx).---  Questions1. What are the top 5 countries those contributed the most immigrants to Canada?2. How was the contribution from ASIA?3. What effects on the number?  Understanding data Data acquisition and preparation<jupyter_code>import numpy as np  # useful for many scientific computing in Python
import pandas as pd # primary data structure library
#!conda install -c anaconda xlrd --yes
df_can = pd.read_excel('https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/DV0101EN/labs/Data_Files/Canada.xlsx',
                       sheet_name='Canada by Citizenship',
                       skiprows=range(20),
                       skipfooter=2)

print ('Data read into a pandas dataframe!')
df_can.head()
df_can.shape<jupyter_output><empty_output><jupyter_text>1. Our dataset has a total of 195 rows and 43 columns2. Some of the columns are containing categorical values, some are numerical3. We don't need to process the categorical value as we don't those to process4. Our result will be visually dependent 5. We won't need data modeling part for this notebook  Preparing data  Cleaning up the dataset to remove columns that are not informative to us for visualization (eg. Type, AREA, REG).<jupyter_code>df_can.drop(['AREA', 'REG', 'DEV', 'Type', 'Coverage'], axis=1, inplace=True)

# let's view the first five elements and see how the dataframe was changed
df_can.head()
#Renaming some columns for better understanding 
df_can.rename(columns={'OdName':'Country', 'AreaName':'Continent','RegName':'Region'}, inplace=True)

# let's view the first five elements and see how the dataframe was changed
df_can.head()<jupyter_output><empty_output><jupyter_text>For consistency, ensure that all column labels of type string.<jupyter_code># let's examine the types of the column labels
all(isinstance(column, str) for column in df_can.columns)
df_can.columns = list(map(str, df_can.columns))

# let's check the column labels types now
all(isinstance(column, str) for column in df_can.columns)<jupyter_output><empty_output><jupyter_text>Set the country name as index - useful for quickly looking up countries using .loc method.<jupyter_code>df_can.set_index('Country', inplace=True)

# let's view the first five elements and see how the dataframe was changed
df_can.head()<jupyter_output><empty_output><jupyter_text>Add a column called Total<jupyter_code>df_can['Total'] = df_can.sum(axis=1)

# let's view the first five elements and see how the dataframe was changed
df_can.head(5)
print ('data dimensions:', df_can.shape)<jupyter_output>data dimensions: (195, 38)<jupyter_text>Checking if there's any missing value<jupyter_code>df_can.isnull().mean()<jupyter_output><empty_output><jupyter_text>There is no missing value in the datasetWe don't need to handle categorical values<jupyter_code># finally, let's create a list of years from 1980 - 2013
# this will come in handy when we start plotting the data
years = list(map(str, range(1980, 2014)))

years
# use the inline backend to generate the plots within the browser
%matplotlib inline 

import matplotlib as mpl
import matplotlib.pyplot as plt

mpl.style.use('ggplot') #visualizefor ggplot-like style

# check for latest version of Matplotlib
print ('Matplotlib version: ', mpl.__version__) # >= 2.0.0
def area_plot(df, num):
    df_top5.plot(kind='area', 
             stacked=True,
             figsize=(20, 10), # pass a tuple (x, y) size
             )

    plt.title(f'Immigration Trend of Top {num} Countries')
    plt.ylabel('Number of Immigrants')
    plt.xlabel('Years')
    plt.show()
    
def bar_plot(df, num):
    df.plot(kind='barh', figsize=(10, 6), color='steelblue') # rotate the bars by 90 degrees

    plt.xlabel('Number of Immigrants')
    #plt.ylabel('Number of Immigrants')
    plt.title(f'Top {num} Conuntries Contributing to the Immigration to Canada between 1980 - 2013')

    # Annotate arrow
    for index, value in enumerate(df): 
        #print(index," ",value)
        label = format(int(value), ',') # format int with commas
        #print(label)
    
        # place text at the end of bar (subtracting 47000 from x, and 0.1 from y to make it fit within the bar)
        plt.annotate(label,xy=(value - 54000, index-0.05), color='white')
    plt.show()<jupyter_output><empty_output><jupyter_text>Q1: What are the top 5 countries those contributed the most immigrants to Canada?> __we will now create an area plot that visualized the top 5 countries that contribued the most immigrants to Canada from 1980 to 2013.__<jupyter_code>df_can.sort_values(['Total'], ascending=False, axis=0, inplace=True)

# get the top 5 entries
df_top5 = df_can.head()

# transpose the dataframe
df_top5 = df_top5[years].transpose() 

df_top5
df_top5.index = df_top5.index.map(int) # let's change the index values of df_top5 to type integer for plotting
area_plot(df_top5, 5)
#df_can.sort_values(by='Total', ascending=False, inplace=True)
df_top5_bar= df_can['Total'].head(5)
df_top5_bar

bar_plot(df_top5_bar, 5)<jupyter_output><empty_output><jupyter_text>1. We are seeing contribution of top five countries to the Canada2. India has the highest contribution over the years3. 4 among the top 5 countries are from Asian subcontinent  Q2: How was the contribution from ASIA?> __Now, we will visualize the contributions from the top 10 countries of ASIA those contribued the immigrants to Canada from 1980 to 2013__<jupyter_code>df_asia = df_can[(df_can['Continent']=='Asia')]
df_asia

df_asia.sort_values(['Total'], ascending=False, axis=0, inplace=True)

# get the top 5 entries
df_top10 = df_asia.head(10)

# transpose the dataframe
df_top10 = df_top10[years].transpose() 
df_top10
df_top10.index = df_top5.index.map(int)
df_top10.df_top10.plot(kind='line', figsize=(14, 8))
plt.title('Immigration Trend of Top 5 Countries from ASIA')
plt.ylabel('Number of Immigrants')
plt.xlabel('Years')
plt.show()plot(kind='line', figsize=(14, 8))
plt.title('Immigration Trend of Top 5 Countries from ASIA')
plt.ylabel('Number of Immigrants')
plt.xlabel('Years')
plt.show()

df_top10_bar= df_asia['Total'].head(10)
#df_top10_bar

bar_plot(df_top10_bar, 10)<jupyter_output><empty_output><jupyter_text>Q3: What affects on the immigrants? **Let's start off by analyzing the effect of Iceland's Financial Crisis:**> The 2008 - 2011 Icelandic Financial Crisis was a major economic and political event in Iceland. Relative to the size of its economy, Iceland's systemic banking collapse was the largest experienced by any country in economic history. The crisis led to a severe economic depression in 2008 - 2011 and significant political unrest.<jupyter_code>#step 1: Prepare the data 
df_iceland = df_can.loc['Iceland', years]
df_iceland
# step 2: plot data
df_iceland.plot(kind='bar', figsize=(10, 6))

plt.xlabel('Year') # add to x-label to the plot
plt.ylabel('Number of immigrants') # add y-label to the plot
plt.title('Icelandic immigrants to Canada from 1980 to 2013') # add title to the plot

plt.show()
df_iceland.plot(kind='bar', figsize=(10, 6), rot=90) 

plt.xlabel('Year')
plt.ylabel('Number of Immigrants')
plt.title('Icelandic Immigrants to Canada from 1980 to 2013')

# Annotate arrow
plt.annotate('',                      # s: str. will leave it blank for no text
             xy=(32, 70),             # place head of the arrow at point (year 2012 , pop 70)
             xytext=(28, 20),         # place base of the arrow at point (year 2008 , pop 20)
             xycoords='data',         # will use the coordinate system of the object being annotated 
             arrowprops=dict(arrowstyle='->', connectionstyle='arc3', color='blue', lw=2)
            )

# Annotate Text
plt.annotate('2008 - 2011 Financial Crisis', # text to display
             xy=(28, 30),                    # start the text at at point (year 2008 , pop 30)
             rotation=72.5,                  # based on trial and error to match the arrow
             va='bottom',                    # want the text to be vertically 'bottom' aligned
             ha='left',                      # want the text to be horizontally 'left' algned.
            )

plt.show()<jupyter_output><empty_output><jupyter_text>> In 2010, Haiti suffered a catastrophic magnitude 7.0 earthquake. The quake caused widespread devastation and loss of life and aout three million people were affected by this natural disaster. As part of Canada's humanitarian effort, the Government of Canada stepped up its effort in accepting refugees from Haiti. We can quickly visualize this effort using a `Line` plot:<jupyter_code>haiti = df_can.loc['Haiti', years] # passing in years 1980 - 2013 to exclude the 'total' column
haiti.head(10)
#df_can.loc['Haiti']

haiti.index = haiti.index.map(int) # let's change the index values of Haiti to type integer for plotting
haiti.plot(kind='line')

plt.title('Immigration from Haiti')
plt.ylabel('Number of immigrants')
plt.xlabel('Years')

plt.show() # need this line to show the updates made to the figure<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup