<jupyter_start><jupyter_text>![Thles_logo.PNG](attachment:Thles_logo.PNG)  Lecture 3: Introduciton to `numpy`, `pandas` and `matplotlib`  Objectives - Learn how to use Pandas and Numpy to easily and efficiently manipulate data - Review some simple statistical operations - Complete the bootcamp exercises Background reading- Python for Data Analysis, <NAME>- Chapter 6, Python for Finance, <NAME>- More examples on plotly and its documentation can be found here: https://plot.ly/- More on visualization in seaborn can be found here: https://seaborn.pydata.org/- Pandas dataframe docs: https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html  Introduction to PandasPandas is a python library for data analysis. At the core of Pandas, lies the concept of a 'DataFrame' - this can be thought of as powerful type of dictionary designed for data analysis.The DataFrame is designed to manage indexed and labeled data, not too different from a SQL database table or a worksheet in a spreadsheet application. Consider the following creation of a DataFrame object:<jupyter_code>#All libs used in the tutorial
%matplotlib inline
import numpy as np
import pandas as pd
import pandas_datareader.data as web
import datetime 
%pylab inline
import matplotlib.pyplot as plt
import seaborn as sns
from matplotlib.pylab import rcParams
import plotly.offline as py
import plotly.graph_objs as go
%matplotlib inline
import numpy as np
import pandas as pd

df = pd.DataFrame([10, 20, 30, 40], columns=['numbers'],
                  index=['a', 'b', 'c', 'd'])
df<jupyter_output><empty_output><jupyter_text>We observe a number of important proporties of the dataframe:- Data itself can be provided in different shapes and types (list, tuple, ndarray, and dict objects are candidates). - Labels: Data is organized in columns, which can have custom names. - Index: There is an index that can take on different formats (e.g., numbers, strings, time information).  The following examples demonstrate several of the most import methods and properties of the Pandas dataframe object.<jupyter_code>df.index  # the index values
df.columns  # the column names
df.ix['a'] # selection via index
df.ix[['a', 'd']]  # selection of multiple indices
df.ix[-1]  # selection via Index object
df.numbers.std()  # sum per column
df.apply(lambda x: x ** 2)  # square of every element
df2 = pd.DataFrame(np.arange(1,1000),columns=['integers'])
df2.apply(lambda x: np.sin(x))
df ** 2  # again square, this time NumPy-like
df['floats'] = (1.5, 2.5, 3.5, 4.5)
  # new column is generated
df

df['names'] = pd.DataFrame(['Yves', 'Guido', 'Felix', 'Francesc'],
                           index=['d', 'a', 'b', 'c'])
df
df = df.append(pd.DataFrame({'numbers': 100, 'floats': 5.75,
                             'names': 'Henry'}, index=['z',]))
df
df.join(pd.DataFrame([1, 4, 9, 16, 25],
            index=['a', 'b', 'c', 'd', 'y'],
            columns=['squares',]))<jupyter_output><empty_output><jupyter_text>Statistical operations on dataframes We shall now turn to collecting and performing basic summary statisical information on some time series data. This is data that is commonly used in finance, more time series data exist everywhere, for example, monthly product sales data or the number of users subscribed to a website on any day. Once we understood how to apply key statistical operations to the dataframe, we shall then briefly review more advanced time series functions. There are a number of data sources that Pandas can collect historical financial data from. These include:- Google Finance (google)- The investor exchange (iex)- Robinhood (robinhood)- World Bank (via `pandas.io.wb`) We shall see, later in this class, that Quandl is another important source of daily and quarterly historical financial data. Let's collect historical daily closes for a collection of symbols and populate a dataframe.<jupyter_code>import pandas_datareader.data as web
import datetime

prices = pd.DataFrame([])

symbols = ['AAPL', 'GOOG', 'MSFT','ZNGA', 'VIXY']
cols = ['r','b','g','y', 'c']

start = datetime.datetime(2015, 5, 4)
end = datetime.datetime(2018, 8, 25)
for symbol in symbols:
  prices[symbol] = web.DataReader(symbol,'iex',start,end)['close']

prices.tail(10)<jupyter_output><empty_output><jupyter_text>The info() member function of the dataframe provides the meta-data.<jupyter_code>prices.info()<jupyter_output><class 'pandas.core.frame.DataFrame'>
Index: 836 entries, 2015-05-04 to 2018-08-24
Data columns (total 5 columns):
AAPL    836 non-null float64
GOOG    836 non-null float64
MSFT    836 non-null float64
ZNGA    836 non-null float64
VIXY    836 non-null float64
dtypes: float64(5)
memory usage: 39.2+ KB<jupyter_text>The describe() member function of the dataframe summarizes the statistical properities of the data.<jupyter_code>prices.describe()<jupyter_output><empty_output><jupyter_text>Plotting DataFramesIn the following example, we shall demonstrate how to plot the columns of the dataframe against the time index using the matplotlib module.<jupyter_code>%matplotlib inline  
%pylab inline
import matplotlib.pyplot as plt
prices.plot(style=cols, lw=2., grid=True)
plt.xlabel('date')
plt.ylabel('Close Prices')<jupyter_output>Populating the interactive namespace from numpy and matplotlib<jupyter_text>We observe that the different instruments, i.e. the columns, are scaled differently. It is therefore convenient to normalize the data by calculating the returns. We shall first perform a quick visual check that the normalized data looks 'reasonable'. We use the tail() member function of the dataframe to view the 5 most recent observations of the data.<jupyter_code>rets = pd.DataFrame([])

for symbol in symbols:
  rets[symbol] = np.log(prices[symbol] / prices[symbol].shift(1))

rets.tail()
%matplotlib inline  
%pylab inline
rets.plot(style=cols, lw=2., grid=True)
plt.xlabel('date')
plt.ylabel('daily returns')
rets.describe()<jupyter_output><empty_output><jupyter_text>We measure the correlation between time series. Here we shall explore VIX returns and its relation to single named equity returns in order to observe the "leverage effect". We shall print the pair-wise correlations in addition to producing scatter plots of the equity returns against the VIX returns.<jupyter_code>rets[['AAPL','VIXY']].corr()
for s in symbols : print('Returns scatterplots for {}'.format(s))
for symbol in symbols:
  xdat = rets[symbol]
  ydat = rets['VIXY']
  plt.plot(xdat, ydat, 'b.')
  ax = plt.axis()  # grab axis values
  plt.grid(True)
  plt.xlabel(symbol + ' returns')
  plt.ylabel('VIX returns')
  plt.show()<jupyter_output>Returns scatterplots for AAPL
Returns scatterplots for GOOG
Returns scatterplots for MSFT
Returns scatterplots for ZNGA
Returns scatterplots for VIXY<jupyter_text>Pandas provides a number of useful time series filters such as moving averages and moving standard deviations. We plot here, the 30-day moving average for APPL price.<jupyter_code>prices['AAPL'].rolling(window=30).mean().plot(grid=True, style=cols[1])
prices['AAPL'].rolling(window=30).std().plot(grid=True, style=cols[4])
(prices['AAPL'].rolling(window=30).mean()+prices['AAPL'].rolling(window=30).std()).plot(grid=True, style=cols[0])
(prices['AAPL'].rolling(window=30).mean()-prices['AAPL'].rolling(window=30).std()).plot(grid=True, style=cols[3])
prices['AAPL'].plot(grid=True, style=cols[2])
rets['index_stock'] = (rets['AAPL'] + rets['GOOG']) / 2 #linear operation with two timeseries
rets['index_stock'].plot(style='r', lw=2., grid=True)
plt.xlabel('date')
plt.ylabel('daily returns index')
prices['AAPL_diff'] = prices['AAPL'].diff() #calculate difference with previous elements
prices['AAPL_diff_2'] = prices['AAPL'].diff(2) # calculate difference with previous second elements
prices.head(10)
prices['AAPL'].apply(np.log).diff() # easy way to calculate log differences
prices['AAPL'].apply(np.log).diff().dropna() # can also drop NaN values
#We can create bolean vectors 
rets['index_stock'].head(10) > 0
#and we can use the vectors to chose the elements in the dataframe
rets[rets['index_stock'] > 0].head()
#we can also work in a similar manner with a specific column of the dataframe
rets['GOOG'][rets['index_stock'] > 0].head()
#Suppose we want to use a very specific location in a dataframe
t = [3,4]
#we can easily do that with the locator
rets.iloc[t]
#Plotting histogram
num_bins = 10
plt.hist(rets['VIXY'].dropna(), num_bins, facecolor = 'red')
plt.title("Histogram of VIX returns ")
plt.xlabel("Value")
plt.ylabel("Frequency")
#Plotting histogram
num_bins = 100
plt.hist(rets['AAPL'].dropna(), num_bins, facecolor = 'blue')
plt.title("Histogram of AAPL stock returns ")
plt.xlabel("Value")
plt.ylabel("Frequency")
#Using visualization library seaborn
import seaborn as sns
from matplotlib.pylab import rcParams
rcParams['figure.figsize'] = 12, 10
corr = rets.corr()
sns.heatmap(corr, 
            xticklabels=corr.columns.values,
            yticklabels=corr.columns.values)
sns.jointplot(x="VIXY", y="AAPL", data=rets, kind="kde")
#We can also plot different plots on one figure
from matplotlib.pylab import rcParams
rcParams['figure.figsize'] = 12, 10
plt.figure(1)
plt.subplot(311)

plt.title('Google',fontsize=20)
plt.plot(range(len(prices.index)),prices["GOOG"], linestyle = '-', color='green', lw='1')
plt.legend(loc = 'lower right', frameon = True)
plt.grid()
plt.ylabel('Price')

plt.subplot(312)

plt.title('Apple',fontsize=20)
plt.plot(range(len(prices.index)),prices.AAPL, linestyle = '-', color='blue', lw='1')
plt.legend(loc = 'lower right', frameon = True)
plt.grid()
plt.ylabel('Price')

plt.subplot(313)

plt.title('VIX',fontsize=20)
plt.plot(range(len(prices.index)),prices.VIXY, linestyle = '-', color='red', lw='1')
plt.legend(loc = 'lower right', frameon = True)
plt.grid()
plt.xlabel('Date');plt.ylabel('Price')<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup