\section{Basics of Reinforcement Learning}
\label{sec:basics_reinforcement_learning}

\emph{Reinforcement Learning} (RL) is a general class of algorithms in the field of \emph{Machine Learning} (ML) that allows an agent to learn how to behave in a stochastic and possibly unknown environment, where the only feedback consists of a scalar reward signal \cite{sutton1998introduction}. The goal of the agent is to learn by trial-and-error which actions maximize his long-run rewards. However, since the environment evolves stochastically and may be influenced by the actions chosen, the agent must balance his desire to obtain a large immediate reward by acting greedily and the opportunities that will be available in the future. Thus, RL algorithms can be seen as computational methods to solve sequential decision problems by directly interacting with the environment.\\

\subsection{Markov Decision Processes}
\label{sec:markov_decision_processes}
Sequential decision problems are typically formalized using \emph{Markov Decision Processes} (MDP). An MDP is a stochastic dynamical system specified by the tuple $<\S, \A, \calP, \calR, \gamma>$, where $(\S, \calS)$ is a measurable state space, $(\A, \calA)$ is a measurable action space, $\calP: \S \times \A \times \calS \to \R$ is a Markov transition kernel, $\calR: \S \times \A \to \R$ is a reward function and $0 < \gamma < 1$ is the discount factor. Suppose that at time $t$ the system is in state $S_t = s$ and that the agent takes action $A_t = a$, then, regardless of the previous history of the system, the probability to find the system in a state belonging to $B\in\calS$ at time $t+1$ is given by
\begin{equation}
	\calP(s, a, B) = \P{S_{t+1} \in B | S_t = s, A_t = a}
\end{equation}
Following this random transition, the agent receives a stochastic reward
$R_{t+1}$. The reward function $\calR(s, a)$ gives the expected reward
obtained when action $a$ is taken in state $s$, i.e.
\begin{equation}
	\calR(s, a) = \E{R_{t+1} | S_t = s, A_t = a}
\end{equation}
This feedback mechanism between the environment and the agent is illustrated in Figure \ref{fig:sequential_decision_problem}. At any time step, the agent selects his actions according to a certain policy $\pi: \S \times \calA \to \R$ such that for every $s \in \S$, $C \mapsto \pi(s,C)$ is a probability distribution over $(\A, \calA)$. Hence, a policy $\pi$ and an initial state $s_0 \in \S$ determine a random state-action-reward sequence ${\{(S_t, A_t, R_{t+1})\}}_{t\geq 0}$ with values on $\S \times \A \times \R$.
\begin{figure}[t]
	\centering
	\begin{tikzpicture}[node distance = 6em, auto, thick]
		\node [block] (Agent) {Agent};
		\node [block, below of=Agent] (Environment) {Environment};
		\path [line] (Agent.0) --++ (4em,0em) |- node [near start]{Action $a_t$} (Environment.0);
		\path [line] (Environment.190) --++ (-6em,0em) |- node [near start]{State  $s_{t}$} (Agent.170);
		\path [line] (Environment.170) --++ (-4.25em,0em) |- node [near start, right] {Reward $r_{t+1}$} (Agent.190);
	\end{tikzpicture}
	\caption{Agent-environment interaction in sequential decision problems.}
	\label{fig:sequential_decision_problem}
\end{figure}
In an infinite horizon task, the agent's performance is typically measured as the total discounted reward obtained following a specific policy
\begin{equation}
	G_t = \sum^{\infty}_{t=0} \gamma^t R_{t+k+1}
\end{equation}
Since this gain is stochastic, the agent considers its expected value, which is typically called \emph{state-value function}
\begin{equation}
	V_\pi(s) = \E[\pi]{G_t|S_t = s}
\end{equation}
where the subscript in $\mathbb{E}_{\pi}$ indicates that all the actions are selected according to policy $\pi$. The state-value function measures how good it is for the agent to be in a given state and follow a certain policy. Similarly, we introduce the \emph{action-value function}
\begin{equation}
	Q_\pi(s,a) = \E[\pi]{G_t|S_t = s, A_t = a}
\end{equation}
We have the following relationship between $V_\pi$ and $Q_\pi$
\begin{equation}
	V_\pi(s) = \int_\A \pi(s,a) Q_\pi(s,a) da
\end{equation}
Almost all reinforcement learning algorithms are designed to estimate these
value functions and are typically based on the Bellman equations.
\begin{equation}
	V_\pi(s) = \calR_\pi(s) + \gamma T_\pi V_\pi(s)
	\label{eq:bellman_expectation_eq_V}
\end{equation}
\begin{equation}
		Q_\pi(s,a) = \calR(s,a) + \gamma T_a V_\pi(s)
		\label{eq:bellman_expectation_eq_Q}
\end{equation}
where we denoted by $T_a$ (resp. $T_\pi$) the transition operator for action
$a$ (resp. for policy $\pi$)
\begin{equation}
	T_a F(s) = \E{F(S_{t+1})|S_t = s, A_t = a} = \int_\S \calP(s, a, s') F(s') ds'
\end{equation}
\begin{equation}
	T_\pi F(s) = \E[\pi]{F(S_{t+1})|S_t = s} = \int_\A \pi(s,a) \int_\S \calP(s,a,s') F(s') ds'	da
\end{equation}
These equations can be rewritten as fixed-point equations which, under some formal assumptions on the reward functions, admit a unique solution by the contraction mapping theorem. The agent's goal is to select a policy $\pi_*$ that maximizes his expected return in all possible states. Such a policy is called \emph{optimal} and the corrisponding value functions are called \emph{Optimal State-Value Function}
\begin{equation}
	V_*(s) = \sup_\pi V_\pi(s)
\end{equation}
and \emph{Optimal Action-Value Function}
\begin{equation}
	Q_*(s,a) = \sup_\pi Q_\pi(s,a)
\end{equation}
The optimal value functions satisfy the following Bellman equations.
\begin{equation}
	V_*(s) = \sup_a Q_*(s,a) = \sup_a \left\{\calR(s,a) + \gamma T_a V_*(s)\right\}
\end{equation}
\begin{equation}
	\begin{split}
		Q_*(s,a) &= \calR(s,a) + \gamma T_a V_*(s)\\
			 &= \calR(s,a) + \gamma \int_\S \calP(s,a,s') \sup_{a'} Q_*(s', a') ds'
	\end{split}
\end{equation}
Again, these are fixed-point equations for which the existence and uniqueness of a solution is guaranteed by the contraction mapping theorem. Given the optimal action-value function $Q_*$, an optimal policy is obtained by selecting in each state the action with maximizes $Q_*$
\begin{equation}
	a_* = \argsup_a Q_*(s,a)
\end{equation}
This greedy policy is deterministic and only depends on the current state of the system.

\subsection{Policy Gradient Methods}
The standard way to solve MDPs is through dynamic programming, which simply consists in solving the Bellman fixed-point equations discussed in the previous chapter. Following this approach, the problem of finding the optimal policy is transformed into the problem of finding the optimal value function. However, apart from the simplest cases where the MDP has a limited number of states and actions, dynamic programming becomes computationally infeasible. Moreover, this approach requires complete knowledge of the Markov transition kernel and of the reward function, which in many real-world applications might be unknown or too complex to use. \emph{Reinforcement Learning} (RL) is a subfield of Machine Learning which aims to turn the infeasible dynamic programming methods into practical algorithms that can be applied to large-scale problems. RL algorithms are based on two key ideas: the first is to use samples to compactly represent the unknown dynamics of the controlled system. The second idea is to use powerful function approximation methods to compactly estimate value functions and policies in high-dimensional state and action spaces. In this section we will only focus on a particular class of algorithms called \emph{Policy Gradient Methods}, which have proved successful in many applications. For a more complete introduction to RL, the reader may consult \cite{sutton1998introduction}, \cite{szepesvari2010algorithms} or \cite{wiering2012reinforcement}.\\
In \emph{policy gradient methods} \cite{peters2008reinforcement}, the optimal policy is approximated using a parametrized policy $\pi: \S \times \calA \times \Theta \to \R$ such that, given a parameter vector $\theta \in \Theta \subseteq \R^{D_\theta}$, $\pi(s, B; \theta) = \pi_\theta(s, B)$ gives the probability of selecting an action in $B \in \calA$ when the system is in state $s \in \S$.
The general goal of policy optimization in reinforcement learning is to
optimize the policy parameters $\theta \in \Theta$ so as to maximize a certain
objective function $J: \Theta \to \R$
\begin{equation}
	\theta^* = \argmax_{\theta \in \Theta} J(\theta)
\end{equation}
In the following, we will focus on gradient-based and model-free methods that exploit
the sequential structure of the the reinforcement learning problem. The idea of
policy gradient algorithms is to update the policy parameters using the gradient ascent direction of the objective function
\begin{equation}
	\theta_{k+1} = \theta_k + \alpha_k \nabla_\theta J\left(\theta_k\right)
\end{equation}
where $\{\alpha_k\}_{k\geq 0}$ is a sequence of learning rates. Typically, the
gradient of the objective function is not known and its approximation is the key component of every policy gradient algorithm. It is a well-know result from stochastic optimization \cite{kushner2003stochastic} that, if the gradient estimate is unbiased and the learning rates satisfy the \emph{Robbins-Monro conditions}
\begin{equation}
	\sum_{k=0}^\infty \alpha_k = \infty \;\;\;\;\;\; \sum^{\infty}_{k=0}
	\alpha_k^2 < \infty
\end{equation}
the learning process is guaranteed to converge at least to a local optimum of
the objective function. In an episodic environment where the system always starts from an initial state $s_0$, the typical objective function is the start value.
\begin{equation}
	J_{\text{start}}(\theta) = V_{\pi_\theta}(s_0) = \E[\pi_\theta]{G_0 |
 S_0 = s_0}
\end{equation}
In a continuing environment, where no terminal state exists and the task might go on forever, it is common to use either the average value
\begin{equation}
	J_{\text{avV}}(\theta) = \E[S \sim d^{\theta}]{V_{\pi_\theta}(S)} = \int_\S
	d^{\theta}(s) V_{\pi_\theta}(s) ds
\end{equation}
where $d^\theta$ is the stationary distribution of the Markov chain induced by $\pi_\theta$. Alternatively, one may use the average reward per time step
\begin{equation}
	J_{\text{avR}}(\theta) = \rho(\theta) = \E[\substack{S \sim d^{\theta}\\A \sim \pi_\theta}]{\calR(S,A)}
	= \int_\S d^{\theta}(s) \int_\A \pi_\theta(s,a) \calR(s,a) da ds
\end{equation}
Luckily, the same methods apply with minor changes to the three objective functions.

\subsubsection{Policy Gradient Theorem}
The \emph{policy gradient theorem} \cite{sutton1999policy} shows that the gradient can be rewritten in a form suitable for estimation from experience aided by an approximate action-value or advantage function.
\begin{theorem}[Policy Gradient]
\label{thm:risk_neutral_policy_gradient}
	Let $\pi_\theta$ be a differentiable policy. The policy gradient for the average reward formulation is given by
	\begin{equation}
		\nabla_\theta \rho(\theta) =
		\E[\substack{S \sim d^\theta\\A \sim \pi_\theta}]{\nabla_\theta\log
		\pi_\theta(S,A) Q_{\theta}(S, A)}
	\end{equation}
	where $d^\theta$ is the stationary distribution of the Markov chain induced by $\pi_\theta$. The policy gradient for the start value formulation is given by
	\begin{equation}
		\nabla_\theta J_{\text{start}}(\theta) =
		\E[\substack{S \sim d_\gamma^\theta(s_0, \cdot)\\A \sim \pi_\theta}]{\nabla_\theta\log
		\pi_\theta(S,A) Q_{\theta}(S, A)}
	\end{equation}
	where $d_\gamma^\theta(s_0, \cdot)$ is the $\gamma$-discounted visiting distribution over states starting from the initial state $s_0$ and following policy $\pi_\theta$
		\begin{equation}
			d_\gamma^\theta(s, x) = \sum_{k=0}^{\infty} \gamma^k \calP_\theta^{(k)}(s, x)
		\end{equation}
\end{theorem}
Let us notice that we can subtract a state-dependent baseline from the action-value function without changing the value of the expectation, indeed
\begin{equation*}
	\begin{split}
	\E[\substack{S \sim d^\theta\\A \sim \pi_\theta}]{\nabla_\theta\log
			\pi_\theta(S,A) B_\theta(S)}
	&= \int_\S d^\theta(s) \int_\A \pi_\theta(s,a) \nabla_\theta\log
				\pi_\theta(s,a) B_\theta(s) da ds\\
	&= \int_\S d^\theta(s)  B_\theta(s) \int_\A \nabla_\theta \pi_\theta(s,a) da ds\\
	&= \int_\S d^\theta(s)  B_\theta(s)  \nabla_\theta  \underbrace{\int_\A  \pi_\theta(s,a) da}_{= 1} ds = 0
	\end{split}
\end{equation*}
Hence, the policy gradient theorem can be rewritten as
\begin{equation}
\label{eq:pg_theorem_baseline}
	\nabla_\theta \rho(\theta) =
	\E[\substack{S \sim d^\theta\\A \sim \pi_\theta}]{\nabla_\theta\log
	\pi_\theta(S,A) \left(Q_{\pi_\theta}(S, A) - B_\theta(S)\right)}
\end{equation}
The baseline can be chosen so as to minimize the variance of the gradient estimate which can prove beneficial for the algorithm convergence \cite{peters2008reinforcement}. This result can be used as the starting point to derive several policy gradient methods that use different approximation of the action-value function, which is typically unknown. For instance, in an episodic MDP the action-value function can be estimated with the total return obtained on a sample trajectory
\begin{equation}
	Q_\theta(s_0,a_0) \approx \sum_{t=0}^{T^{(m)}} \gamma^t r_{t+1}^{(m)}
\end{equation}
Combining this remark with a Monte Carlo approximation of Eq. (\ref{eq:pg_theorem_baseline}), we obtain the \emph{Monte Carlo Policy Gradient} algorithm \cite{baxter2001infinite} (also known as GPOMDP) for which the pseudocode is reported in Algorithm \ref{algo:GPOMDP}.
\begin{algorithm}[t]
	\caption{GPOMDP}
	\label{algo:GPOMDP}
	\begin{algorithmic}[0]
		\Require{\\
			\begin{itemize}
				\item Initial policy parameters $\theta_0 = (\theta_0^1, \ldots, \theta_0^{D_\theta})^T$
				\item Learning rate $\{\alpha_k\}$
				\item Number of trajectories $M$
			\end{itemize}
		}
		\Ensure Approximation of the optimal policy $\pi_{\theta^*} \approx \pi_*$
		\begin{algorithmic}[1]
		\State Initialize $k = 0$
		\Repeat
			\State Sample $M$ trajectories $h^{(m)} = \{(s_t^{(m)}, a_t^{(m)}, r_{t+1}^{(m)}\}_{t = 0}^{T^{(m)}}$ of the MDP under policy $\pi_{\theta_k}$
			\State Compute the optimal baseline
			\begin{equation}
				\widehat{b}_k^n = \frac{\sum^{M}_{m=1} \left[ \sum_{i=0}^{T^{(m)}}
									\partial_{\theta_k} \log \pi_\theta\left(s_i^{(m)}, a_i^{(m)}\right) \right]^2
									\sum^{T^{(m)}}_{j=0} \gamma^j r_{j+1}^{(m)}}{\sum^{M}_{m=1} \left[ \sum_{i=0}^{T^{(m)}} \partial_{\theta_k} \log \pi_\theta\left(s_i^{(m)}, a_i^{(m)}\right) \right]^2}
			\end{equation}
			\State Approximate policy gradient
			\begin{equation}
				\frac{\partial}{\partial\theta^n} J_{\text{start}}(\theta_k) \approx \widehat{g}_k^n = \frac{1}{M} \sum^{M}_{m=1} \sum_{i=0}^{T^{(m)}}
					\frac{\partial}{\partial\theta^n} \log \pi_{\theta_k}\left(s_i^{(m)}, a_i^{(m)}\right) \left(
					\sum^{T^{(m)}}_{j=i} \gamma^j r_{j+1}^{(m)} - \widehat{b}_k^n \right)
			\end{equation}
			\State Update actor parameters $\theta_{k+1} = \theta_k + \alpha_k \widehat{g}_k $.

			\State $k \leftarrow k + 1$
		\Until{converged}
		\end{algorithmic}
	\end{algorithmic}
\end{algorithm}

\subsubsection{Parameter-Based Policy Gradient Methods}
In Monte Carlo Policy Gradient, trajectories are generated by sampling at each
time step an action according to a stochastic policy $\pi_\theta$ and the
objective function gradient is estimated by differentiating the policy with
respect to the parameters. However, sampling an action from the policy at each
time step leads to a large variance in the sampled histories and therefore in
the gradient estimate, which can in turn slow down the convergence of the
learning process. To address this issue, the \emph{policy gradient with parameter-based exploration} (PGPE) method \cite{sehnke2008policy} replaces the search in the policy space with a direct search in the model parameter space. Given an episodic MDP, PGPE considers a deterministic controller $F: \S \times \Theta \to \A$ that, given a set of parameters $\theta \in \Theta \subseteq \R^{D_\theta}$, maps a state $s \in \S$ to an action $a = F(s; \theta) = F_\theta(s) \in \A$. The policy parameters are drawn from a probability distribution $p_\xi$, with hyper-parameters $\xi \in \Xi \subseteq \R^{D_\xi}$. Combining these two hypotheses, the agent follows a stochastic policy $\pi_\xi$ defined by
\begin{equation}
	\forall B \in \calA ,\ \pi_\xi(s,B) = \pi(s, B; \xi) = \int_\Theta p_\xi(\theta)
	\ind{F_{\theta}(s)\in B} d\theta
\end{equation}
In this setting, the policy gradient theorem can be reformulated in the following way
\begin{theorem}[Parameter-Based Policy Gradient]
	Let $p_\xi$ be differentiable with respect to $\xi$, then the gradient of the average reward is given by
	\begin{equation}
		\nabla_\xi J(\xi) = \E[\substack{S \sim d^\xi\\\theta \sim p_\xi}]{\nabla_\xi \log p_\xi(\theta) Q_{\pi_\xi}(S, \theta)}
	\end{equation}
	where we denoted $Q_\xi(S, \theta) = Q_\xi(S, F_\theta(S))$.
\end{theorem}
This expression is very similar to the original policy gradient theorem, but
the expectation is taken over the controller parameters instead of the action space and we have the likelihood score of the controller parameters distribution instead of that of the stochastic policy. Thus, we might interpret this result as if the agent directly selected the parameters $\theta$ according to a policy $p_\xi$, which then lead to an action through the deterministic mapping $F_\theta$. Therefore, it is as if the agent's policy was in the parameters space and not in the control space. As in the standard policy gradient methods, we can subtract a state-dependent baseline $B_\xi(S)$ to the gradient without increasing the bias
\begin{equation}
	\nabla_\xi J(\xi) = \E{\nabla_\xi \log p_\xi(\theta) \left(Q_{\pi_\xi}(S,
						\theta) - B_\xi(S)\right)}
\end{equation}
The PGPE algorithm, which is outlined in Algorithm \ref{algo:PGPE}, employs a Monte Carlo approximation of this gradient, where the action-value function is estimated using the returns on a sampled trajectory of the MDP. The benefit of this approach is that the controller is deterministic and therefore the actions do not need to be sampled at each time step, with a consequent reduction of the gradient estimate variance. Indeed, It is sufficient to sample the parameters $\theta$ once at the beginning of the episode and then generate an entire trajectory following the deterministic policy $F_\theta$. As an additional benefit, the parameter gradient is
estimated by direct parameter perturbations, without having to backpropagate
any derivatives, which allows to use non-differentiable controllers. Again the baseline can be chosen so as to minimize the gradient estimate variance \cite{zhao2011analysis}.
\begin{algorithm}[t!]
	\caption{Episodic PGPE algorithm}
	\label{algo:PGPE}
	\begin{algorithmic}[0]
		\Require{\\
			\begin{itemize}
				\item Initial hyper-parameters $\xi_0 = (\xi_0^1, \ldots, \xi_0^{D_\xi})^T$
				\item Learning rate $\{\alpha_k\}$
				\item Number of trajectories $M$
			\end{itemize}
		}
		\Ensure Approximation of the optimal policy $F_{\xi^*} \approx \pi_*$
		\begin{algorithmic}[1]
			\State Initialize $k = 0$
			\Repeat
				\For {$m = 1, \ldots, M$}
					\State Sample controller parameters $\theta^{(m)} \sim p_{\xi_k}$
					\State Sample trajectory $h^{(m)} = \{(s_t^{(m)}, a_t^{(m)}, r_{t+1}^{(m)}\}_{t = 0}^{T^{(m)}}$ under policy $F_{\theta^{(m)}}$
				\EndFor
				\State Compute optimal baseline
				\begin{equation}
					\widehat{b}_k^n = \frac{\sum^{M}_{m=1} \left[\partial_{\xi^n} \log p_{\xi_k} \left(\theta^{(m)}\right)\right]^2 \sum^{T^{(m)}}_{j=0} \gamma^j r_{j+1}^{(m)}}{\sum^{M}_{m=1} \left[\partial_{\xi^n} \log p_{\xi_k} \left(\theta^{(m)}\right)\right]^2}
				\end{equation}
				\State Approximate policy gradient
				\begin{equation}
					\frac{\partial}{\partial\xi^n} J_{\text{start}}(\xi_k) \approx \widehat{g}_k^n = \frac{1}{M} \sum^{M}_{m=1}
						\frac{\partial}{\partial\xi^n} \log p_{\xi_k}\left(\theta^{(m)}\right) \left(
						\sum^{T^{(m)}}_{j=i} \gamma^j r_{j+1}^{(m)} - \widehat{b}_k^n \right)
				\end{equation}
				\State Update hyperparameters using gradient ascent $\xi_{k+1} = \xi_k + \alpha_k \widehat{g}_k^n$
				\State $k \leftarrow k + 1$
			\Until{converged}
		\end{algorithmic}
	\end{algorithmic}
\end{algorithm}


\subsection{Example: Four-bar linkage Mechanism}

\begin{frame}
	\begin{block}{Example 2: Four-bar linkage Mechanism}
		\begin{table}
			\begin{minipage}{0.5\linewidth}
				\begin{tabular}{l|l}
					& $l_{AB}=l_1=0.15m$\\
					& $l_{BC}=l_2=0.35m$\\
					& $l_{CD}=l_3=0.3m$\\
					Given & $l_{CE}=l_4=0.15m$\\
					& $x_D=0.3m$\\
					& $y_D=0.3m$\\
					& $\theta_1=45^ {\circ}$\\ \hline
					Find & $\vb{r}{B}$, $\vb{r}{C}$, $\vb{r}{E}$\\
				\end{tabular}
			\end{minipage}\hfill
			\begin{minipage}{0.5\linewidth}
				\begin{figure}
					\includegraphics[width=50mm]{images/R-RRR.png}
				\end{figure}
			\end{minipage}
		\end{table}
	\end{block}
\end{frame}

\begin{frame}
\emph{Solution}\vskip .25cm
Position of joint $B$:  $\displaystyle \vb{r}{B} = x_B\ih + y_B\jh = l_1\cos{\theta_1}\ih + l_1\sin{\theta_1}\jh$\\
Position of joint $C$:  $\displaystyle \vb{r}{C} = x_C\ih + y_C\jh = 0.1\jh$\\
Position of joint $D$: $\displaystyle \vb{r}{D} = x_D\ih + y_D\jh$\\
Position of joint $E$: $\displaystyle \vb{r}{E} = x_E\ih + y_E\jh$\\
\[
\Rightarrow\begin{cases}
(x_B-x_C)^2+(y_B-y_C)^2=l_1^2\\
(x_D-x_C)^2+(y_D-y_C)^2=l_3^ 2
\end{cases}
\]
Solving the system of equations yields $x_{C_1}$ and $x_{C_2}$. Notice that in the mechanism, $0>x_C$ is the condition to obtain correct solution.
\[
\Rightarrow \begin{cases}
(x_C-x_E)^2 + (y_C-y_E)^2 = l_4^2\\
\displaystyle \frac{y_C-y_E}{x_C-x_E}=\frac{y_D-y_E}{x_D-x_E}
\end{cases}
\]
Solving the system of equations yields $x_{E_1}$ and $x_{E_2}$. Notice that in the mechanism, $x_C>x_E$ is the condition to obtain correct solution.
\end{frame}

\begin{frame}{MATLAB R2019a code}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/RRRR-position1.m}
\end{frame}
\begin{frame}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/RRRR-position2.m}
\end{frame}
\begin{frame}{Plotting using MATLAB R2019a}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/RRRR-plot.m}
\end{frame}
\begin{frame}{Output figure}
\centering
\includegraphics[width=100mm]{images/RRRR-plot.png}
\end{frame}
\begin{frame}{Trajectory plotting using MATLAB R2019a}
\lstinputlisting[style=Matlab-editor, basicstyle=\mlttfamily]{codes/RRRR-trajectory.m}
\end{frame}
\begin{frame}{Output figure}
\centering
\includegraphics[width=100mm]{images/RRRR-trajectory.png}
\end{frame}


%! TEX root = master.tex

\lecture{5}{Wed 03 Nov 2021 10:45}{Determinants}

\subsection*{Definitions}

Let \matri{A} be a square matrix. The scalar $\det(A)$ or $\mod{A}$  is called the \textbf{determinant} of \matri{A}.

For a  $2 \times 2$ square matrix, the determinant is given by $a_{11}a_{22} - a_{12}a_{21}$. The plot thickens very quickly as we try to determine the determinant of larger $n \times n$ matrices.\\

The process of evaluating the determinant of an $n \times n$ matrix is described by the following steps:
\begin{enumerate}
  \item The \textbf{minor} $\matr{M_{jk}}$ is the matrix obtained from a square matrix when omitting the $j^{th}$ row and the $k^{th}$ column.
  \item The scalar quantity referred to as the \textbf{cofactor} $\alpha_{jk} = (-1)^{j+k} \det(\matr{M}_{jk})$
\end{enumerate}
The determinant of an $n \times n$ matrix \matri{A}, when choosing the first row as the starting point is therefore given by
    \begin{align*}
  \det(\matr{A}) &= a_{11}\matr A_{11} + a_{12}\matr A_{12} + \cdots + a_{1n}\matr A_{1n} \\
                 &= \sum_{i=1}^n a_{1r}\matr A_{1r}
\end{align*}
The same value of the determinant is obtained if the first point of reference is chosen to be any other row or column. To minimize computation complexity, the row or column with the most entries equal to 0 should be chosen with the result of reducing the maximum number of terms in the sequence to 0.

\subsection*{Properties of Determinants}

\begin{enumerate}
  \item [P1.] $\det(\matr{A}) = \det(\matr{A}^T)$
  \item [P2.] If \matri{A} is an $n \times n$ triangular matrix, then $\det(\matr{A}) = a_{11}a_{22}\cdots a_{nn}$.
  \item [P3.] The interchanging of any two consecutive rows only alters the sign of of the determinant without affecting its magnitude.
  \item [P4.] If any two rows consecutive rows or columns of an $n \times n$ matrix \matri{A} are equal, then $\det(\matr{A}) = 0$.
\end{enumerate}
\newpage
  \begin{theorem}\label{thm:scalar-out}
    If all elements of one row are multiplied by a constant $k$, then the value of the determinant is also multiplied by k.
  \end{theorem}

\begin{proof}
    Let \matri{K} be the general $n \times n$ matrix \matri{A} but with all entries in the first row multiplied by $k$. Then,
    \begin{align*}
      \mod{K} &=
    \begin{vmatrix}
      ka_{11} & ka_{12} & \cdots & ka_{1n}\\
      a_{21} & a_{22} & \cdots & a_{2n}\\
      \vdots & \vdots & \ddots & \vdots\\
      a_{n1} & a_{n2} & \cdots & a_{nn}
    \end{vmatrix}\\
              &= ka_{11}A_{11} + ka_{12}A_{12} + \cdots + ka_{1n}A_{1n}\\
              &= k(a_{11}A_{11} + a_{12}A_{12} + \cdots + a_{1n}A_{1n})\\
              &= k\mod{A}
    \end{align*}
\end{proof}

\corollary{If any two rows of an $n \times n$ matrix \matri A are multiples of the other (\emph{are linearlly dependent}), then the $\mod{A} = 0$.}
\corollary{If \matri{A} is an $n \times n$ matrix, then $\det{(k\matr A)} = k^n \det(\matr A)$.}
\vspace{2em}
\begin{theorem}
 If a distinct scalar $\alpha_{ij}$ is added to all the entries of a row of an $n \times n$ matrix \matri{A}, then $\det(\matr A)$ is equal to
 \[  \begin{vmatrix}
      a_{11} & a_{12} & \cdots & a_{1n}\\
      a_{21} & a_{22} & \cdots & a_{2n}\\
      \vdots & \vdots & \ddots & \ldots\\
      a_{n_1} & a_{n_2} & \cdots & a_{nn}\\
    \end{vmatrix}
    +
    \begin{vmatrix}
      \alpha_{11} & \alpha_{12} & \cdots & \alpha_{1n}\\
      a_{21} & a_{22} & \cdots & a_{2n}\\
      \vdots & \vdots & \ddots & \vdots\\
      a_{n_1} & a_{n_2} &\cdots& a_{nn}
      \end{vmatrix}.\]
\end{theorem}
\begin{proof}
  \[
    \text{Consider\ \ \ }
    \begin{vmatrix}
      a_{11}+\alpha_{11} & a_{12}+\alpha_{12} & \cdots & a_{1n}+\alpha_{1n}\\
      a_{21} & a_{22} & \cdots & a_{2n}\\
      \vdots & \vdots & \ddots & \ldots\\
      a_{n_1} & a_{n_2} & \cdots & a_{nn}\\
    \end{vmatrix}\\
  \]
  \begin{align*}
    &= \left(a_{11} + \alpha_{11}\right)\matr A_{11} + \left(a_{12} + \alpha_{12}\right)A_{12} + \cdots + \left( a_{1n} + \alpha_{1n} \right)\matr A_{1n}\\
    &= (a_{11}\matr A_{11} + a_{12}\matr A_{12} + \cdots + a_{1n}\matr A_{1n}) + \left(\alpha_{11}\matr A_{11} + \alpha_{12}\matr A_{12} + \cdots + \alpha_{1n}\matr A_{1n}\right)\\
  \end{align*}
\end{proof}
\newpage
\begin{theorem}\label{thm:row-distr}
  The value of a determinant is unchanged if we add to the entries of any row the same multiple of the corresponding entries of another row.

\begin{center}
  If $\mod{A} =
  \begin{vmatrix}
    a_{11} & a_{12} & \cdots & a_{1n}\\
    a_{21} & a_{22} & \cdots & a_{2n}\\
    \vdots & \vdots & \ddots & \vdots\\
    a_{n1} & a_{n2} & \cdots & a_{nn}
  \end{vmatrix}$, then $
  \begin{vmatrix}
    a_{11} + ka_{21} & a_{12}+ka_{22} & \cdots & a_{1n} + ka_{2n}\\
    a_{21} & a_{22} & \cdots & a_{2n}\\
    \vdots & \vdots & \ddots & \vdots\\
    a_{n1} & a_{n2} & \cdots & a_{nn}
  \end{vmatrix} = \mod{A}$
\end{center}

\end{theorem}
\begin{proof}
Let  $\matr K =\begin{vmatrix}
    a_{11} + ka_{21} & a_{12}+ka_{22} & \cdots & a_{1n} + ka_{2n}\\
    a_{21} & a_{22} & \cdots & a_{2n}\\
    \vdots & \vdots & \ddots & \vdots\\
    a_{n1} & a_{n2} & \cdots & a_{nn}
  \end{vmatrix}$. Then, by Theorem ~\ref{thm:row-distr}, \[\mod{K} = \mod{A} +
\begin{vmatrix}
    ka_{21} & ka_{22} & \cdots & ka_{2n}\\
    a_{21} & a_{22} & \cdots & a_{2n}\\
    \vdots & \vdots & \ddots & \vdots\\
    a_{n1} & a_{n2} & \cdots & a_{nn}
  \end{vmatrix}.\]
  Then, by Theorem \ref{thm:scalar-out} and P4,
$
  \begin{vmatrix}
    ka_{21} & ka_{22} & \cdots & ka_{2n}\\
    a_{21} & a_{22} & \cdots & a_{2n}\\
    \vdots & \vdots & \ddots & \vdots\\
    a_{n1} & a_{n2} & \cdots & a_{nn}
  \end{vmatrix}
  =
  k\begin{vmatrix}
    a_{21} & a_{22} & \cdots & a_{2n}\\
    a_{21} & a_{22} & \cdots & a_{2n}\\
    \vdots & \vdots & \ddots & \vdots\\
    a_{n1} & a_{n2} & \cdots & a_{nn}
  \end{vmatrix}
  = 0
$
~\\~\\~\\Therefore, $\mod{K} = \mod{A}$.
\end{proof}
\begin{theorem}\label{thm:det-distr}
  The determinant of the product of two matrices is equal to the product of the two determinants.
  $$\det(\matr{AB}) = \det(\matr A) \det(\matr B)$$
\end{theorem}
\begin{proof}
  This proof is left as an exercise to the reader for the $2 \times 2$ matrix case.
\end{proof}

\begin{theorem}
  If $\matr A \inverse$ is invertible, then $\mod{A \inverse} = \frac{1}{\det{(\matr A)}}$
\end{theorem}

\begin{proof}
  \matri A is invertible. Thus
  \begin{align*}
    \exists\ \matr B \text{ s.t. } \matr{AB} &= I\\
    \det{(\matr{AB})} &= \det{(I)}\\
    \det(\matr A) \cdot \det(\matr B) &= 1\\
    \det(\matr B) &= \frac{1}{\det(\matr A)}
 \end{align*}
\end{proof}
\remark{An $n \times n$ matrix is invertible $\Leftrightarrow \det(\matr A) \neq 0$}


\section{(Linear) Regression \quad
{\normalfont\sffamily model: $\color{section-text-color} \hat{\mathbf y} = \bm X \beta$}}

% ===

Assuming $\bm X^\top \bm X$ non-singular. %$\hat\beta\ped{Linear}$ has smallest variance among linear \underline{unbiased} estimates.

Bayesian view:\;
$(Y \mid X,\beta) \sim \Gauss{\bm x^\top \beta, \sigma^2 \mathbb I}$

Distrib. of estimator
$\hat\beta\ped{LS} \sim \Gauss{\beta, (\bm X^\top \bm X)^{-1} \sigma^2}$

\emph{Ridge:}
$\epsilon\ped{RSS}(\beta,{\color{OrangeRed}{\lambda}}) = \paren{\bm y - \bm X^\top \!\beta}^\top \paren{\bm y - \bm X^\top \!\beta} \color{OrangeRed} {+} \lambda \beta^\top \!\beta$
\\
$\hat\beta = (\bm X^\top \bm X \begingroup \color{OrangeRed} + \lambda \mathbb I \endgroup)^{-1} \bm X^\top \bm y$,
\quad prior: $\beta \sim \Gauss{0, \frac{\sigma^2}{\lambda} \mathbb I}$

\emph{(Ridge) Shrinkage:}
Decompose $\bm X = \bm{UDV}^\top$\\
$\bm X \hat\beta = \bm{UD} (\bm D^2 {+} \begingroup \color{OrangeRed} \lambda \mathbb I \endgroup)^{-1} \bm{DU}^\top \bm y = \sum\limits_{j\leq d} \bm u_j \frac{d_j^2}{d_j^2 \begingroup \color{OrangeRed} + \lambda \endgroup} \bm u_j^\top \bm y$

\emph{Lasso:}
%$\epsilon\ped{RSS}(\beta,\lambda) = \sum_{i\leq n} (y_i - x_i^\top \beta)^2 + \lambda \norm{\beta}_1$
$\hat\beta = \arg\min_\beta \sum_{i\leq n} (y_i - x_i^\top \beta)^2 \color{OrangeRed} + \lambda \norm{\beta}_1$
\\
\textit{(no closed form)},
\hfill prior: $p(\beta_i) {=} \frac{\lambda}{4\sigma^2} \exp(-\abs{\beta_i} \frac{\lambda}{2\sigma^2})$

\emph{Bias-variance:}
$\E[D]{ \E[Y\mid X=x]{ (\hat f(x) - Y)^2 } }$
{\small ${=} \E[D]{ (\hat f(x) {-} \E[D]{\hat f(x)})^2 } + \paren*{ \E[D]{\hat f(x)} {-} \E{Y\vert X=x} }^2$}
$+ \E{ (Y - \E{Y\vert X=x})^2 }$
$= variance + bias^2 + noise$

\emph{Gauss-Markov Theorem:}\\
For any linear estimator $\widetilde\theta = c\!^\top \bm y = a\!^\top (\hat\beta {+} \bm{Dy})$ that is unbiased for $a^\top \beta$, it holds: $\V{a^\top \hat\beta} \leq \V{c^\top \bm y}$.

Among all linear \textbf{u}-estimators, $\hat\beta\ped{LS}$ minimises the gen. error!
\enskip
What about \textbf{biased} estimators? We \rotatebox[origin=c]{45}{$\to$} bias a bit in the hope that the variance \rotatebox[origin=c]{-45}{$\to$}.

\emph{Combining Regressors:} $\hat f(x) \coloneqq \frac1B \sum_{i\leq B} \hat f_i(x)$\\
$\mathrm{bias} [\hat f(x)] = \frac1B \sum \mathrm{bias} [\hat f_i(x)]$\\
$\V{\hat f} = \frac{1}{B^2} \sum \V[D]{\hat f_i} + \frac{1}{B^2} {\sum\sum}_{i\neq j} \Cov{\hat f_i, \hat f_j} \approx \frac{\sigma^2}{B}$

% ===


%!TEX root =  ../main.tex

\subsection{Zeros}

\objective{Graph, read, and produce rational functions}

A rational function is one defined by $\frac{f(x)}{g(x)}$ where $f(x)$ and $g(x)$ are both
polynomials.  Rational functions exhibit some common behaviors with polynomials,
such as $x$-intercepts and one $y$-intercept.  But they also tend to have vertical and
horizontal asymptotes.  The most basic rational function --- $\frac{1}{x}$ --- is, in fact,
a hyperbola, with two branches.  But rational functions can have many more than two
branches, or even just have one.

There are three things one could consider with regard to 0 and a given rational function.
First, the entire quotient could be 0, which would make $x$-intercepts.  Fortunately,
a quotient is only equation to zero when the numerator is equal to zero, meaning
we only need to find when top equation's zeros to find the whole equation's.

When the denominator is equal to zero, much more bizarre behavior emerges.  A vertical
asymptote will occur, though it is not immediately clear if the left and right limits will be
to opposite infinities, or to the same one.  Factoring will make it clear: if the term
has an even multiplicity, the asymptote will be followed to the same infinity.  Otherwise,
one sided-limit will be $\infty$ and the other $-\infty$.

Lastly, we should consider what happens when we make the argument 0.  A $y$-intercept
is always a coordinate with the appearance $(0,y_0)$, so plugging in 0 is an efficient way
to find it.

Each of these three techniques for various ``zero's'', assumes that it is not occurring when
function is in an indeterminate form.  Holes and asymptotes can prevent intercepts from
actually occurring, and obfuscate one another

\subsection{Horizontal Asymptotes}
If a rational function looks like a fraction, a ratio, a quotient, that's because it is!  Just as
a polynomial looks more and more like an integer power function the further out one zooms,
so too a rational function will look more like the results if one simply divides the numerator
by the denominator.

Consider the function $r(x) = \dfrac{2x^2-4x-6}{x^2+x-2}$.  In the big picture,
it is a simple division problem.  Ultimately, we want to know $lim_{x\rightarrow\infty}
r(x)$:

\polylongdiv{2x^2-4x-6}{x^2+x-2}

and so the question becomes equivalent to

$$
\lim_{x\rightarrow\infty} 2-\frac{6x+2}{x^2+x-2}
$$

which is two.

We might have spared ourselves the work of polynomial long division by contrasting the degree
of the numerator and denominator beforehand.  Which has greater degree?
\begin{itemize}
\item[\textbf{denom.}] the long term behavior can be modeled by a horizontal asymptote at $y=0$
\item[\textbf{neither}] the long tern behavior will still be a horizontal asymptote at $y=\frac{a}{b}$, where $a$ and $b$ are the leading terms of the numerator and denominator respectively
\item[\textbf{numer.}] polynomial division cannot be avoided, and the long-run behavior trends towards an asymptote at $y=$ the quotient, ignoring the remainder
\end{itemize}

\subsection{Signs}
Rational functions are among the hardest for graphers to correctly display.
Even large computers can display misleading results if the window is not correctly
pre-programmed.  If it is necessary to map out the sign of the function beforehand,
a tabular approach can be very helpful.  We will combine a number line and a table
into a new kind of grid, using the following function as an example:

$$Y_1=\frac{x+1}{(10x-1)^2}$$

\begin{figure}[h]
\begin{centering}
\begin{tikzpicture}

\matrix (m) [matrix of math nodes,
             column sep=0cm, row sep=0pt,
     nodes={text width=15mm, align=center,
            text height=3ex, text depth=1.5ex}]
{
(x+1)       &   -   &   +   &   +   \\
(10x-1)^2    &   +   &   +   &   +   \\
\textbf{Q}        &   \textbf{-}  &  \textbf{+}    &  \textbf{+}    \\
};
\draw  (m-1-1.north west) -- (m-1-4.north east);
\draw (m-2-1.north west) -- (m-2-4.north east);
\draw (m-3-1.north west) -- (m-3-4.north east);
\draw (m-3-1.south west) -- (m-3-4.south east);

\draw  (m-1-1.north east) -- (m-3-1.south east);
\draw (m-1-2.north east) -- (m-3-2.south east);
\draw (m-1-3.north east) -- (m-3-3.south east);
\draw (m-1-4.north east) -- (m-3-4.south east);

\node[above] at (m-1-2.north east) {$-1$};
\node[above] at (m-1-3.north east) {$0.1$};

\node at (m-1-2.east) {$0$};
\node at (m-2-2.east) {$+$};
\node at (m-3-2.east) {\textbf{0}};

\node at (m-1-3.east) {$+$};
\node at (m-2-3.east) {$0$};
\node at (m-3-3.east) {\textbf{$\infty$}};
\end{tikzpicture}
\caption{A sign table and number-line combination}
\end{centering}
\end{figure}

The table is made by writing the factors of the numerator and denominator down the
left.  The bottom row is the function itself, which is the quotient of the rest.  Across
the top, you will need a division for each of the zeros of the factors and room
for all the spaces in-between and around.  On the lines, write the zero's if the factor
is zero there, or just the sign.  To complete the bottom, simply remember the
rules of  signs:

\begin{enumerate}
\item a positive multiplied or divided by a positive is a positive
\item a negative multiplied or divided by a negative is a negative
\item a negative multiplied or divided by a negative is positive
\end{enumerate}

The zero's require only a little more thought.  A zero in the numerator
makes the entire quotient zero.  If a zero in the numerator is surrounded by
like signs, then either side of the vertical asymptote will go to that sign infinity.
If the signs disagree, then both infinities will be approached, each from the
side matching their sign.

\begin{figure}
\begin{centering}
\includegraphics[scale=0.3]{\chapdir/pics/verticalasymptote}
\caption{An appropriately zoomed in graph of $\frac{x+1}{(10x-1)^2}$}
\end{centering}
\end{figure}

With this much more appropriate window, we can see that there is a local minimum,
perhaps in the vicinity of $x=-2$.  If it indeed occurs at a rational number, there is no
reason to find the derivative by hand with the TI-8* will do it for us!

\index{TI-8*!graph derivatives}
Set $Y_2$ to be \texttt{nDeriv(Y$_1$(X),X,X)} or $\left.\frac{d}{dx}(Y_1(X))\right|_{X=X}$,
depending upon your model of TI-8*.  (nDerive is choice 8 under
\Touche[style=function,principal=math].)  It seems have a
zero near 2.  The function is clearly concave and derivative is increasing, so it should be
a local minimum.  Using ZERO (under
\Touche[style=function,principal=trace,fontsize=7pt,position=0.9]
), we find it to be exactly -2.1.  Plugging
back into the original equation, we get $\frac{-1.1}{484}$, which $\triangleright$FRAC
converts to $-\frac{1}{440}$.


\chapter{Trigonometric Functions}
\section{Radians}
A radian is a measure of the central angle $\theta$ such that the arc $s$
intercepted by the angle is equal to the length of the radius, $r$.

One radian is the measure of a central angle $\theta$ that intercepts an arc $s$
equal to the length of the radis $r$ of the circle: ($\theta$ is measured in
radians)
\begin{equation}
  \theta=\frac{s}{r}
\end{equation}

There are a few key terms that are used when describing angles inside of a
circle:
\begin{description}
  \item[Initial side]{The starting position of the ray}
  \item[Terminal side]{The other side of the ray}
  \item[Vertex]{The intersection of the initial and terminal side}
\end{description}

\subsection{Converting beteen radians and degrees}
\begin{enumerate}
  \item{To convert degrees to radians, simply multiply degrees by
    $\frac{\pi\text{rad}}{180}$}
  \item{To convert radians to degrees, multiply radians by
    $\frac{180}{\pi\text{rad}}$}
\end{enumerate}

\subsection{Linear and Angular Speed}
Arc legth is defined for a length $s$ as the quantity of the length of the
radius, $r$ and the central angle that $s$ is intercepted by, $\theta$:
\begin{equation}
  s=r\theta
\end{equation}

Therefore, linear speed is defined as the quotient of the arc-length and the
time that has passed:
\begin{equation}
  \text{linear speed}=\frac{\text{arc length}}{\text{time}}=\frac{s}{t}
\end{equation}

And as follows, angular speed is the same formula, except we are looking at
radians per second, as in the following equation:
\begin{equation}
  \text{angular speed}=\frac{\text{central angle}}{\text{time}}=\frac{\theta}{t}
\end{equation}

\section{Unit Circle Trigonometry}
Consider a unit circle (a circle of radius $1$ centered around $(0,0)$) defined
by the following equation: $x^2+y^2=1$.

For any point \textit{on} the circle, the coordinate pair that will intersect
the edge of the circle is defined as $P(\theta)=(cos(\theta),sin(\theta))$.

\begin{itemize}
  \item{$sin(t)=y$}
  \item{$cos(t)=x$}
  \item{$tan(t)=\frac{y}{x}$}
  \item{$csc(t)=\frac{1}{y}$}
  \item{$sec(t)=\frac{1}{x}$}
  \item{$cot(t)=\frac{x}{y}$}
\end{itemize}

\subsection{Even and Odd Trigonometric Functions}
\begin{itemize}
  \item{Even:}
    \begin{itemize}
      \item{$cos(-t)=cos(t)$}
      \item{$sec(-t)=sec(t)$}
    \end{itemize}
  \item{Odd:}
    \begin{itemize}
      \item{$sin(-t)=-sin(t)$}
      \item{$csc(-t)=-csc(t)$}
      \item{$tan(-t)=-tan(t)$}
      \item{$cot(-t)=-cot(t)$}
    \end{itemize}
\end{itemize}

\section{Right Triangle Trigonmetry}
Consider a right triangle.  A geometric figure with three sides, two of which
meet at a $90\deg$ angle.  The other side is reffered to as the hypoteneuse, and
the length of that side is equal to $\sqrt{a^2+b^2}$.

There are six trigonometric functions that we will study that relate to right
triangles.  They are defined as follows:
\begin{itemize}
  \item{$sin(\theta)=\frac{\text{opp}}{\text{hyp}}$}
  \item{$cos(\theta)=\frac{\text{adj}}{\text{hyp}}$}
  \item{$tan(\theta)=\frac{\text{opp}}{\text{adj}}$}
  \item{$csc(\theta)=\frac{1}{sin(\theta)}$}
  \item{$sec(\theta)=\frac{1}{cos(\theta)}$}
  \item{$cot(\theta)=\frac{1}{tan(\theta)}$}
\end{itemize}

\subsection{Trigonometric Identities}

\subsubsection{Reciprocal Identities}
This set of identities defines the relationships between pairs of trigonometric
functions as inverses of each other.

\begin{itemize}
  \item{$sin(\theta)=\frac{1}{csc(\theta)}$}
  \item{$cos(\theta)=\frac{1}{sec(\theta)}$}
  \item{$tan(\theta)=\frac{1}{cot(\theta)}$}
  \item{$csc(\theta)=\frac{1}{sin(\theta)}$}
  \item{$sec(\theta)=\frac{1}{cos(\theta)}$}
  \item{$cot(\theta)=\frac{1}{tan(\theta)}$}
\end{itemize}

\subsubsection{Quotient Identites}
This set of identnties defines two trigonometric functions as quotients of other
pairs of trigonometric functions.

\begin{itemize}
  \item{$tan(\theta)=\frac{sin(\theta)}{cos(\theta)}$}
  \item{$cot(\theta)=\frac{cos(\theta)}{sin(\theta)}$}
\end{itemize}

\section{Graphs of Sine and Cosine Functions}
The sine function is defined for all $x\in(-\infty,\infty)$, and has a range of
$[-1,1]$.  It has a period of $2\pi$ and has $x$-intercepts at $(n\pi, 0)$ and
$y$-intercepts at the origin.  It is an odd-function with original symmetry.

On the other hand, the cosine is also defined for all $x\in(-\infty,\infty)$,
and has a range of $[-1,1]$.  It has a period of $2\pi$ and has $x$-intercepts
at $(\frac{\pi}{2}+n\pi,0)$.  It has a $y$-intercept at $(1,0)$ and is an even
function with $y$-axis symmetry.

\subsection{Amplitude}
The amplitude of a function is defined as one half of the distance between its
relative minimum and maximum points.  For any trigonmetric function of the form
$y=a\text{sin}(x)$ or $y=a\text{cos}(x)$, the amplitude is $|a|$.

\subsection{Period}
The period of a function is the time it takes the function to repeat itself, for
any point.  The period itself is not a function, it is a constant value for all
points \textit{across} a function.  For any function that is defined by the
pattern: $y=a\text{sin}(x)$ or $y=a\text{cos}(x)$, the period is defined by the
following equation:
\begin{equation}
  \text{Period}=\frac{2\pi}{b}
\end{equation}

\section{Graphs of other Trigonometric Functions}

\subsection{The Tangent and its Inverse}
The graph of the tangent function is defined for all real numbers $x$, such that
$x\neq\frac{\pi}{2}+n\pi$.  Its range is defined for all
$x\in(-\infty,\infty)$ and has $x$-intercepts at $(n\pi,0)$, and a
$y$-intercept at the origin.  It has vertical asymptotes at
$x=\frac{\pi}{2}+n\pi$ and is an odd function with origianl symmetry.

The function $f(x)=cot(x)$ is exactly the $tan(x)$ function, but shifted
$\frac{\pi}{2}$ units to the right.  That is to say that
$f(x)=cot(x)=tan(x)+\frac{\pi}{2}$.

\subsection{The Cosecant and Secant Functions}
These functions are very easy to reason about.  Take their inverse functions,
and draw plot them on a graph.  Define vertical asymptotes when the origianl
function is $f(x)=0$, and draw points at the relative minimum and maximum across
the original function.  Stretch those points to the asymptotes through
continuous curves, and you have the function that you are trying to sketch.

The same rules apply when applying transformations to these functions.

\section{Inverse Trignonometric Functions}
The inverse sine function, or $f(x)=arcsin(x)$ is defined over the interval
$[-\frac{\pi}{2},\frac{\pi}{2}]$.  Simply take the sine function at those two
endpoints and plot the inverse function.  Apply transformations as normal.

\subsection{Characteristics of Inverse Trigonometric Functions}
\begin{description}
  \item[$f(x)=arcsin(x)$]{Defined over all $x\in[-1,1]$; $-\frac{\pi}{2} \leq
    f(x) \leq \frac{pi}{2}$}
  \item[$f(x)=arccos(x)$]{Defined over all $x\in[-1,1]$; $0 \leq f(x) \leq \pi$}
  \item[$f(x)=arctan(x)$]{Defined over all $x\in(-\infty,\infty)$;
    $-\frac{\pi}{2} \leq f(x) \leq \frac{\pi}{2}$}
\end{description}


\chapter{Elliptic curve cryptography}
\label{chpr:ecc}
In this chapter we will analyse the cryptographic primitives and assumptions that will allow in the next chapter to introduce the digital signature schemes that are the core of the present work. The first argument will be the specialization of elliptic curves to finite fields, the mathematical structure underpinning the whole elliptic curve cryptography.

\bigskip

\section{Elliptic curves over finite fields}
\label{ecoverff}
Up to now we considered the field over which the curve is defined to be the set of real numbers: this is not at all the unique possibility. The typical choice in cryptography is $K = \mathbb{F}_p$, where $\mathbb{F}_p$ is the finite field with $p$ elements.
\\
Since there are only finitely many pairs $(x, y)$ with $x, y \in \mathbb{F}_p$, the group $E(\mathbb{F}_p)$ is finite.
In practice we consider:
$$\{(x, y) \in \mathbb{F}_p^2 \ | \ y^2 \equiv x^3 + ax + b \ (\text{mod} \ p), \ 4a^3 + 27b^2 \not\equiv 0 \ (\text{mod} \ p)\} \cup \{\infty\}.$$
It can be shown that the formulas for point addition are the same derived in Section \ref{grouplaw}, but where the calculations are done modulo $p$\footnote{Notice that reduction modulo $p$ can be executed much faster if the prime $p$ is a Mersenne or a pseudo-Mersenne prime, i.e. if $p = 2^d - 1$ for some $d$ in the first case or if $p \simeq 2^d$ in the second one.}.
\\
\begin{center}
	\includegraphics[width=0.9\linewidth]{Images/ec_over_ff.eps}
	\captionof{figure}{The curve $y^2 = x^3 - x$ over $\mathbb{F}_{61}$.}
	\label{fig:figure4}
	\source{\url{https://en.wikipedia.org/wiki/Elliptic_curve\#Elliptic_curves_over_finite_fields}.}
\end{center}
In Figure \ref{fig:figure4} the typical shape of an elliptic curve over a finite field is represented: we can see that we are not anymore dealing with a smooth curve, but with a finite set of points scattered along the plane. We can notice a certain degree of symmetry with respect to the line $y = \frac{p}{2}$, with the exception of the points with $y$ coordinate equal to zero, that correspond to the roots of the cubic $x^3 + ax + b$ in $\mathbb{F}_p$.
\\
It is still possible to define a geometric method for point addition: the general idea is that we draw the line passing through the points considered. Since we are working in modular arithmetic, this line ``repeats" itself along the plane. Once the line intersects a third point, we take the opposite one to be the result of the addition. The approach is presented in Figure \ref{fig:figure5}, however the method can be somewhat counterintuitive, especially when dealing with point doubling, so we stick with the algebraic notation.

\bigskip
\noindent
The number of points on an EC over $\mathbb{F}_q$ plays a central role in the cryptographic applications, so we state here a useful theorem that allows to give an estimate:
\begin{thm} [{\bf Hasse's theorem}] Let $\mathbb{F}_q$ be a finite field and let $E$ be an elliptic curve defined over $\mathbb{F}_q$. Then the order of $E(\mathbb{F}_q)$ (i.e. the number of points) satisfies: $$|q + 1 - \#E(\mathbb{F}_q)| \leq 2\sqrt{q}.$$
\end{thm}
\begin{center}
	\includegraphics[width=0.9\linewidth]{Images/sum_ec_over_ff.eps}
	\captionof{figure}{Geometric representation for the addition of points on the elliptic curve $y^2 = x^3 - x + 3$ over $\mathbb{F}_{127}$.}
	\label{fig:figure5}
	\source{\cite{RefWork:4}.}
\end{center}
\noindent
Loosely speaking, Hasse's theorem tells us that the number of points over $E(\mathbb{F}_q)$ increases linearly with the order of the field.
\\
Another observation we want to do is that if $\#E(\mathbb{F}_p)$ is a prime number, then $E(\mathbb{F}_p)$ is a cyclic group: this is a direct consequence of Lagrange's theorem. Since the order of each subgroup is a divisor of the order of the group, we have that, if the order of the group is a prime number, there is no subgroup (with the exception of the trivial one, comprising only the point at infinity). This means that it has to exist a generator of the whole group. Suppose this is not the case, i.e. $\nexists G \in E(\mathbb{F}_p) \ | \ \forall Q \in E(\mathbb{F}_p) \ \exists q \in [1, ..., \#E(\mathbb{F}_p)] \ \text{s.t.} \ Q = qG$: thus, $\forall G \in E(\mathbb{F}_p)$, there exists at least a $Q$ for which the relation $Q = qG$ does not hold. Fix $G$ and start adding it repeatedly to itself. Of course we would reach different points in $E(\mathbb{F}_p)$ due to the closure of the addition. However we won't be able to reach $Q$, by definition. But eventually we would reach the point at infinity (when $q$ gets equal to the order of $G$): then we would start again, meaning that we would have found a cyclic subgroup. But this is in contrast with Lagrange's theorem, so that we can conclude that the whole set of elliptic curve's points form a cyclic group if $\#E(\mathbb{F}_p)$ is a prime number.

\bigskip

\subsection{Elliptic curve domain parameters}
\label{ecparam}
In order to rely on ECC, the parties involved in a scheme need to agree on a set of parameters called elliptic curve domain parameters. A typical choice is to rely on standardized curves (e.g. those defined in \cite{RefWork:3}); however, it is still possible to choose other parameters, but this has to be done carefully, as we will explain in this section. We start defining EC parameters, following closely \cite{RefWork:2}.
\\
\\
Elliptic curve parameters are defined as a sextuple $T = (p, a, b, G, n, h)$, where:
\begin{itemize}
	\item The integer $p$ specifies the prime finite field $\mathbb{F}_p$;
	\item The two elements $a, b \in \mathbb{F}_p$ specifies the elliptic curve $E(\mathbb{F}_p)$ through the Weierstrass equation;
	\item $G$ is a point on $E(\mathbb{F}_p)$;
	\item $n$ is the order of $G$, i.e. the number of elements of the cyclic subgroup generated by $G$ (that can coincide with the whole $E(\mathbb{F}_p)$);
	\item $h = \frac{\#E(\mathbb{F}_p)}{n}$ is an integer\footnote{We can deduce that $h$ is an integer directly from Lagrange's theorem.} called the cofactor.
\end{itemize}
There are different ways in which we can choose these parameters. For example, they can be generated at random or not. Random generation is a conservative choice, since it offers a guarantee against future special purpose attacks. In this case it could be better to use verifiable random parameters, meaning that $(a, b)$ and/or $G$ are obtained as output from a secure hash function\footnote{An hash function in general is a function that takes as input strings of arbitrary length and outputs in a fixed length sequence of bytes. For a discussion on the security properties of an hash function we refer to the appendix of \cite{RefWork:2}.}, applied to some seed $S$. When choosing verifiable random parameters, the sextuple $T$ should be coupled with the seed value $S$ to allow parameters' validation. However, non random curves are typically built in particular ways to ensure efficient computations. Standardized curves, that comprises both types of EC, have the advantage of underpinning interoperability, plus an high degree of reliability, having been heavily tested.
\\
Now that we have explained what they are and how they can be chosen, we will show some constraints and the validation process for the sextuple $T$.

\bigskip
\bigskip
\bigskip
\bigskip
\bigskip
\noindent
{\bf Parameters' selection}:
\begin{enumerate}
	\item Select the approximate security level in bits, $t$\footnote{The security level should be chosen such that solving the Elliptic Curve Discrete Logarithm Problem (ECDLP, Section \ref{dlp}) requires $2^t$ curve's operations. The SEC standard that we are following restricts to $t \in \{80, 112, 128, 192, 256\}$.};
	\item Choose a prime $p$ such that $\lceil log_2p\rceil = 2t$ if $80 < t < 256$, such that $\lceil log_2p\rceil = 521$ if $t = 256$ and such that $\lceil log_2p\rceil = 192$ if $t = 80$;
	\item Select $a$ and $b$ in $\mathbb{F}_p$ such that:
	\begin{itemize}
		\item $4a^3 + 27b^2 \neq 0 \ (\text{mod} \ p)$;
		\item $\#E(\mathbb{F}_p) \neq p$;
	\end{itemize}
	\item Select $G \in E(\mathbb{F}_p)$ such that:
	\begin{itemize}
		\item $p^B \neq 1 \ (\text{mod} \ n), \ \forall 1 \leq B < 100$;
		\item $h \leq 2^{\frac{t}{8}}$;
		\item $n - 1$ and $n + 1$ should each have a large prime factor $r$, which is large in the sense that $log_nr > \frac{19}{20}$.
	\end{itemize}
\end{enumerate}
These requirements may look strange, but are all needed in order to avoid special attacks. In particular, the two requirements at point 3 are needed in order for the curve to be non singular and to avoid Smart's attack\footnote{The points on the curve are mapped to the elements of the additive group of $\mathbb{F}_p$, so that the discrete logarithm problem is solvable in polynomial time through the {\bf Extended Euclidean Algorithm}.}, respectively.
\\
The three requirements at point 4 instead are needed to ensure that the curve is robust against MOV and Cheon's attack\footnote{These attacks are complex and their explanation is out of the scope of the present work; we refer the interested reader to the appendix of \cite{RefWork:2} for an overview and for further bibliography.}, and to ensure that the order of $G$ is sufficiently high.
\\
Notice however that the choice of the parameters does not secure against all possible attacks. Indeed other kind of attacks may be possible, such as side channel attacks: timing and power analysis are an example. They exploit the differences in the point addition and doubling operations, that lead to different timings or power consumptions. Possible solutions are a change of coordinates or the use of Edwards curves, a special family of elliptic curves for which addition and doubling can be done with the same operation.

\bigskip
\noindent
{\bf Parameters' validation}: Given $T = (p, a, b, G, n, h)$ and $t$, the parameters are deemed valid if:
\begin{enumerate}
	\item $p$ is a prime such that $\lceil log_2p\rceil = 2t$ if $80 < t < 256$, or such that $\lceil log_2p\rceil = 521$ if $t = 256$, or such that $\lceil log_2p\rceil = 192$ if $t = 80$;
	\item $a, b, x_G$ and $y_G$ are integers in $[0, ..., p - 1]$;
	\item $4a^3 + 27b^2 \neq 0 \ (\text{mod} \ p)$;
	\item $y_G^2 = x_G^3 + ax_G + b \ (\text{mod} \ p)$;
	\item $n$ is a prime number;
	\item $h \leq 2^{\frac{t}{8}}$ and $h = \lfloor (\sqrt{p} + 1)^2 / n \rfloor$;
	\item $nG = \infty$;
	\item $p^B \neq 1 \ (\text{mod} \ n), \ \forall 1 \leq B < 100$ and $n \neq p$.
\end{enumerate}
If the parameters are verifiably random, it should also be checked that $(a, b)$ and/or $G$ have been correctly derived from the seed $S$.
\\
These checks are needed to verify that: the curve has the required difficulty level, it is defined over $\mathbb{F}_p$, it is non singular, $G$ is a point of the curve, the Discrete Logarithm (DL, Section \ref{dlp}) is difficult, $h$ is effectively the cofactor of $G$, $G$ has order $n$ and neither MOV nor Smart's attacks are possible.

\bigskip

\subsection{Elliptic curve key pairs}
\label{keypairs}
In this section we would like to briefly analyse the core of the public key cryptography based on elliptic curves: the concept of elliptic curve key pair. Shortly, public key cryptography is a cryptographic system that relies on pairs of keys, a public and a private key, with different roles. The asymmetry of the keys is the reason behind the fact that public key cryptography is sometimes called also asymmetric cryptography. We talk about asymmetry since public keys can be published, while private keys must be kept secret, as the names suggest. This accomplishes two functions: authentication, since everybody with the public key can verify that the holder of the paired private key sent a signed message, and encryption, where the public key is used for encryption, but only the owner of the private counterpart can decrypt.

\bigskip
\noindent
Given some elliptic curve domain parameters $T = (p, a, b, G, n, h)$, an elliptic curve key pair $\{\textcolor{red}{q}, \textcolor{green}{Q}\}$ associated with $T$ consists of an elliptic curve secret key $\textcolor{red}{q}$, which is an integer in $[1, n - 1]$, and an elliptic curve public key $\textcolor{green}{Q} = (x_Q, y_Q)$, which is the point $\textcolor{green}{Q} = \textcolor{red}{q}G$.
\\
The choice of the colours is intended to make clear what is secret and what can be made public.
\\
To detect transmission errors or to prevent the deliberate submit of an invalid key, it is desirable to validate the public key of the counterparties. This can be achieved following these steps:
\begin{enumerate}
	\item Check that $Q \neq \infty$;
	\item Check that $x_Q$ and $y_Q$ are integers in $[0, p - 1]$ and that $y_Q^2 = x_Q^3 + ax_Q + b \ (\text{mod} \ p)$;
	\item Check that $nQ = \infty$.
\end{enumerate}
Through this algorithm we are checking that $Q$ is effectively a point on the curve different from the point at infinity and that it belongs to the cyclic subgroups generated by $G$. Indeed, if the relation $Q = qG$ holds we have that $nQ = n(qG) = q(nG) = q\infty = \infty$. If this would not be the case, the third check would fail.

\bigskip

\subsection{Jacobian coordinates}
\label{jac}
We have already written that for EC over finite fields the same formulas previously seen hold, with the calculations done modulo $p$. What we have not dealt with is that this comes with a downside: in particular we have now to deal with modular inversion, an operation that, although can be done efficiently, is up to two orders of magnitude slower than field's multiplication.
\\
In the next chapter we will deal with digital signatures and we will see that ECDSA requires modular inversion. This traduces in poor performances when it comes to a system like Bitcoin in which signatures are (potentially) verified by each participant in the network to validate transactions. Therefore, it is sometimes advantageous to avoid modular inversion, either in the formulas for point addition or in the signature scheme. The first case can be solved through a change of coordinates: the approach consists in writing all the points as points in a projective space, the key idea being to defer the divisions by multiplying them into a denominator, represented then as a new coordinate, instead of performing every division immediately. Only at the very end we perform a single division to convert from projective coordinates back to affine coordinates.

\bigskip
\noindent
Jacobian coordinates are a modification of projective coordinates, typically used since they lead to a faster doubling procedure. In affine form, each elliptic curve point has two coordinates $(x, y)$. In projective form, each point will have three coordinates $(X : Y : Z)$, with the restriction that $Z$ is never zero for points different from the point at infinity\footnote{A projective point is denoted by $(X : Y : Z)$ since, in general, the two dimensional projective space \textbf{P}$_K^2$ over a field $K$ is given by equivalence classes of triples $(X, Y, Z)$, with $X, Y, Z \in K$ and at least one element different from zero. Here equivalence between $(X_1, Y_1, Z_1)$ and $(X_2, Y_2, Z_2)$ means that $\exists \lambda \in K \ | \ (X_1, Y_1, Z_1) = \lambda(X_2, Y_2, Z_2)$. Thus, the equivalence classes only depend on the ratios of $X$ to $Y$ to $Z$, from which the notation.}. The forward mapping is given by $(x, y) \mapsto (xz^2, yz^3, z)$, for any non zero $z$ (usually chosen to be 1 for convenience). The reverse mapping is given by $(X : Y : Z) \mapsto (X/Z^2, Y/Z^3)$, as long as $Z$ is non zero.
\\
The elliptic curve $y^2 = x^3 + ax + b$ becomes:
$$Y^2 = X^3 + aXZ^4 + bZ^6.$$
The point at infinity now has the coordinates $(1 : 1 : 0)$: indeed, substituting $Z = 0$ in the equation defining the curve we get $Y^2 = X^3$; then we simply normalize.
\\
Let $Q_i = (X_i : Y_i : Z_i), \ i = 1, 2$, be points on the elliptic curve $Y^2 = X^3 + aXZ^4 + bZ^6$. Then:
$$(X_3 : Y_3 : Z_3) = (X_1 : Y_1 : Z_1) + (X_2 : Y_2 : Z_2),$$
where $X_3,Y_3$ and $Z_3$ are computed as follows:
\begin{itemize}
	\item $Q_1 \neq \pm Q_2$: we have the affine test $x_1 = x_2$, that in jacobian coordinates correspond to the check $X_1/Z_1^2 = X_2/Z_2^2 \ \Longrightarrow \ X_1Z_2^2 = X_2Z_1^2$.
	\\
	We start again from the slope of the line passing through $Q_1$ and $Q_2$.
	$$m = \frac{y_2 - y_1}{x_2 - x_1} = \frac{Y_2/Z_2^3 - Y_1/Z_1^3}{X_2/Z_2^2 - X_1/Z_1^2} = \frac{Y_2/Z_2^3 - Y_1/Z_1^3}{X_2/Z_2^2 - X_1/Z_1^2} \cdot \frac{Z_1^3Z_2^3}{Z_1^3Z_2^3} =$$ $$= \frac{Y_2Z_1^3 - Y_1Z_2^3}{X_2Z_1^3Z_2 - X_1Z_1Z_2^3}.$$
	Defining $T = Y_1Z_2^3$, $U = Y_2Z_1^3$, $W = U - T$, $R = X_1Z_2^2$, $S = X_2Z_1^2$ and $V = S - R$ we can write:
	$$m = \frac{U - T}{SZ_1Z_2 - RZ_1Z_2} = \frac{W}{VZ_1Z_2}.$$
	Now consider:
	$$x_3 = m^2 - x_1 - x_2 = \frac{W^2}{V^2Z_1^2Z_2^2} - \frac{X_1}{Z_1^2} - \frac{X_2}{Z_2^2} = \frac{W^2}{V^2Z_1^2Z_2^2} - \frac{X_1}{Z_1^2}\cdot\frac{V^2Z_2^2}{V^2Z_2^2} - \frac{X_2}{Z_2^2}\cdot\frac{V^2Z_1^2}{V^2Z_1^2} = $$ $$= \frac{W^2 - V^2X_1Z_2^2 - V^2X_2Z_1^2}{V^2Z_1^2Z_2^2} = \frac{W^2 - V^2R - V^2S}{V^2Z_1^2Z_2^2} = \frac{W^2 - V^2(S + R)}{V^2Z_1^2Z_2^2} = $$ $$= \frac{W^2 - V^2(S - R + 2R)}{V^2Z_1^2Z_2^2} = \frac{W^2 - V^3 - 2RV^2}{V^2Z_1^2Z_2^2},$$
	$$y_3 = m(x_1 - x_3) - y_1 = \frac{W}{VZ_1Z_2}\left(\frac{X_1}{Z_1^2} - \frac{W^2 - V^3 - 2RV^2}{V^2Z_1^2Z_2^2}\right) - \frac{Y_1}{Z_1^3} = $$ $$= \frac{W}{VZ_1Z_2}\left(\frac{V^2X_1Z_2^2 - W^2 + V^3 + 2RV^2}{V^2Z_1^2Z_2^2}\right) - \frac{Y_1}{Z_1^3} = $$ $$= \frac{W(RV^2 - [W^2 - V^3 - 2RV^2]) - V^3Y_1Z_2^3}{V^3Z_1^3Z_2^3} = $$ $$\frac{W(RV^2 - [W^2 - V^3 - 2RV^2]) - TV^3}{V^3Z_1^3Z_2^3}.$$
	Thus, we can write:
	$$X_3 = W^2 - V^3 - 2RV^2, \ Y_3 = W(RV^2 - X_3) - TV^3, \ Z_3 = VZ_1Z_2,$$
	where
	$$T = Y_1Z_2^3, \ U = Y_2Z_1^3, \ R = X_1Z_2^2, \ S = X_2Z_1^2, \ V = S - R, \ W = U - T.$$
	\item $Q_1 = Q_2$:  now we have to check if $y = 0$. This is equivalent to check whether  $Y / Z^3 = 0 \ \Longrightarrow \ Y = 0$, since $Z \neq 0$.
	$$m = \frac{3x^2 + a}{2y} = \frac{3(X/Z^2)^2 + a}{2Y/Z^3} = \frac{3X^2/Z^4 + a}{2Y/Z^3} \cdot \frac{Z^4}{Z^4} = \frac{3X^2 + aZ^4}{2YZ} = \frac{W}{2YZ},$$
	if we define $W = 3X^2 + aZ^4$.
	$$x_3 = m^2 - 2x = \frac{W^2}{4Y^2Z^2} - 2\frac{X}{Z^2} = \frac{W^2 - 8XY^2}{4Y^2Z^2} = \frac{W^2 - 2V}{4Y^2Z^2},$$ once we define $V = 4XY^2$. Moreover:
	$$y_3 = m(x - x_3) - y = \frac{W}{2YZ}\left(\frac{X}{Z^2} - \frac{W^2 - 2V}{4Y^2Z^2}\right) - \frac{Y}{Z^3} = $$ $$= \frac{W(4XY^2 - [W^2 - 2V])}{8Y^3Z^3} - \frac{Y}{Z^3} = \frac{W(4XY^2 - [W^2 - 2V]) - 8Y^4}{8Y^3Z^3} =$$ $$= \frac{W(V - [W^2 - 2V]) - 8Y^4}{8Y^3Z^3}.$$
	Finally we can write:
	$$X_3 = W^2 - 2V, \ Y_3 = W(V - X_3) - 8Y^4, \ Z_3 = 2YZ,$$
	where
	$$W = 3X^2 + aZ^4, \ V = 4XY^2.$$
	\item Once again we can consider jointly the separate cases $Q_1 = Q_2 \wedge y = 0$ and $Q_1 = -Q_2$: indeed, by definition, we have $Q_1 + Q_2 = \infty$.
\end{itemize}
Looking at the formulas we can see that no inversion is involved in the calculations. Moreover, a further speed up is possible when doubling if we set $a = -3$\footnote{This is the reason why some standardized curves make this choice for the parameter $a$.}: we have $W = 3X^2 -3Z^4 = 3(X^2 - Z^4) = 3(X + Z^2)(X - Z^2)$, which can be computed via one squaring and one multiplication rather than via three squarings.

\bigskip
\noindent
We implemented this approach in the library that can be found at \url{https://github.com/dginst/BitcoinBlockchainTechnology} to improve the efficiency of the curve's operations. The speed up checks have been performed considering the curves secp192k1, secp192r1, secp224k1, secp224r1, secp256k1, secp256r1, secp384r1 and secp521r1 as described in \cite{RefWork:3}, resulting in a scalar multiplication that is from six to seven times faster in all the cases.

\bigskip

\subsection{The Bitcoin curve: secp256k1}
\label{btccurve}
Since this work is mainly focused on applications that can be deployed in Bitcoin, we would like to present the elliptic curve over a finite field used there. This could also suggest the shape of the curves used in practice.
\\
\\
The curve is named secp256k1: the naming is not casual, and its analysis could help a better understanding of the curve's properties. It begins with {\bf sec} to denote "Standards for Efficient Cryptography", the documentation from which it is taken; then it follows a {\bf p}, denoting the use of parameters over a prime field $\mathbb{F}_p$, in contrast with the so called binary fields $\mathbb{F}_{2^m}$, denoted by the letter {\bf t}; the {\bf p} is followed by a number, 256, denoting the length in bits of the field size $p$, that suggests the difficulty of solving the DL on the curve; then it comes a {\bf k} to denote parameters associated with a Koblitz curve\footnote{The name Koblitz curve used here refers to the definition given in \cite{RefWork:3}.}, to be distinguished from an {\bf r}, that would denote the use of verifiably random parameters; at last we find the sequence number {\bf 1}, meaning that this curve is the first, actually the unique, with all these characteristics.
\\
We have already discussed the benefits of using a random curve; however, secp256k1 was constructed in a special non random way to ensure efficient computations.
\\
Here follows the sextuple $T$ defining the secp256k1 curve:
\begin{itemize}
	\item The finite field $\mathbb{F}_p$ is defined by the pseudo-Mersenne prime number:
	$$p = \text{FFFFFFFF} \ \ \text{FFFFFFFF} \ \ \text{FFFFFFFF} \ \ \text{FFFFFFFF} \ \ \text{FFFFFFFF}$$
	$\text{FFFFFFFF} \ \text{FFFFFFFE} \ \text{FFFFFC2F} =$
	\\
	\\
	$= 2^{256} - 2^{32} - 2^9 - 2^8 - 2^7 - 2^6 - 2^4 - 1.$
	\item The defining equation $E$: $y^2 = x^3 + ax + b$ is determined by:
	$$a = \text{00000000} \ \ \text{00000000} \ \ \text{00000000} \ \ \text{00000000} \ \ \text{00000000} \ \ \text{00000000} \ \ \text{00000000}$$
	$ \text{00000000} =$
	\\
	\\
	$= 0;$
	$$b = \text{00000000} \ \ \text{00000000} \ \ \text{00000000} \ \ \text{00000000} \ \ \text{00000000} \ \ \text{00000000} \ \ \text{00000000}$$
	$ \text{00000007} =$
	\\
	\\
	$= 7.$
	\\
	\\
	Hence $E$:	$y^2 = x^3 + 7$.
	\item The point $G$ in compressed form is\footnote{The starting bytes 0x02 and 0x04 mean exactly that the first expression of $G$ is in compressed form, while the second one is uncompressed. For what concerns the compressed form we need to have informations about the $y$ coordinate. 0x02 is used when it is even, while 0x03 is used when it is odd: if $y$ is a square root in $\mathbb{F}_p$, we know that also $-y \ (\text{mod} \ p) = p - y$ is a square root. Since $p$ is an odd prime, we are sure that one of the roots is odd while the other is even.}:
	$$G = \text{02} \ \ \text{79BE667E} \ \ \text{F9DCBBAC} \ \ \text{55A06295} \ \ \text{CE870B07} \ \ \text{029BFCDB}$$
	$\text{2DCE28D9} \ \ \text{59F2815B} \ \ \text{16F81798},$
	\\
	\\
	and in uncompressed form:
	$$G = \text{04} \ \ \text{79BE667E} \ \ \text{F9DCBBAC} \ \ \text{55A06295} \ \ \text{CE870B07} \ \ \text{029BFCDB}$$
	$\text{2DCE28D9} \ \ \text{59F2815B} \ \ \text{16F81798} \ \ \text{483ADA77} \ \ \text{26A3C465} \ \ \text{5DA4FBFC}$
	\\
	\\
	$\text{0E1108A8} \ \ \text{FD17B448} \ \ \text{A6855419} \ \ \text{9C47D08F} \ \ \text{FB10D4B8}$.
	\item Finally, the order $n$ of $G$ and the cofactor are:
	$$n = \text{FFFFFFFF} \ \ \text{FFFFFFFF} \ \ \text{FFFFFFFF} \ \ \text{FFFFFFFE} \ \ \text{BAAEDCE6}$$
	$\text{AF48A03B} \ \ \text{BFD25E8C} \ \ \text{D0364141},$
	\\
	\\
	$h = \text{01}$.
\end{itemize}

\bigskip

\bigskip

\section{The discrete logarithm problem}
\label{dlp}
In this section we will deal more in depth with the obscure concept of Discrete Logarithm Problem (DLP).
\\
In general the DLP is defined over a group $\mathbb{G}$. For the moment write $\mathbb{G}$ in multiplicative notation and consider $x, y \in \mathbb{G}$ such that $y$ is in the cyclic subgroup generated by $x$. The DLP is the problem of determining an integer $k \geq 1$ such that $x^k = y$: this notation justifies the name, since it would mean to find $k = log_xy$.
\\
Typically, the cryptosystems whose security is based on ECC, depend on the difficulty of solving the DLP defined over the EC (ECDLP). One way of attacking the DLP is simple brute force: try all possible values of $k$ until one works. This is impractical when the answer $k$ is an integer of several hundred digits, which is a typical size used in cryptography (indeed this approach is fully exponential in the number of bits representing $k$\footnote{Assume for example to work with secp256k1, the Bitcoin curve. This means that $p \simeq 2^{256}$, and the same holds true for $n$. By brute force we try all possible values of $k$, that ranges from $1$ to $n-1$. Trying all possible $k$ would lead to a number of steps of the order $O(2^{256})$, exponential in the number of bits representing the order of the group.}). Therefore, better techniques are needed.
\\
More formally, the DLP can be defined as follows:
\begin{mydef}
	Let $\mathbb{G}$ be a cyclic group of order $n$ and let $g$ be a generator. An algorithm $\mathcal{A}$ is said to $(t, \epsilon)$-solve the DLP in $\mathbb{G}$ if on input a random $h \in \mathbb{G}$, it runs in time at most $t$ and returns $k \in \{0, ..., n - 1\}$ such that $h = g^k$ with probability at least $\epsilon$.
\end{mydef}
\noindent
The most general algorithms that works in any group has a time upper bound of $O(|\mathbb{G}|^{\frac{1}{2}})$, where with $|\mathbb{G}|$ we denote the cardinality of the set $\mathbb{G}$. But it turns out that the DLP is significantly easier in some groups than it is in others. In order of difficulty:
\begin{enumerate}
	\item The additive group of $\mathbb{F}_q$: the problem here can be stated as finding $k$ such that $kx = y$, i.e. we only need to find the inverse of $x$, and we have already stated that this can be done efficiently through the extended euclidean algorithm (in particular this requires $O(log_2(q))$ elementary operations, meaning that the problem can be solved in polynomial time);
	\item The multiplicative group $\mathbb{F}_q^{\times}$: the problem here is finding $k$ such that $x^k = y$. It can be shown that there exist some algorithms that work in sub-exponential time (the so called index calculus methods);
	\item Elliptic curves over finite fields: the fastest known procedure to solve the ECDLP on general curves are collision algorithms. Those algorithms are fully exponential: this explains the widespread adoption of ECC in recent years
\end{enumerate}
\noindent
The last point is linked to the primary benefit of ECC, that is a smaller key size: since the problem is harder, in order to have the same level of security we can rely on smaller keys, reducing storage and transmission requirements.

\bigskip
\noindent
The two most general algorithms are known as {\bf Baby Step, Gian Step Algorithm} and as the {\bf Pollard's $\rho$ Method} with its variants\footnote{For a detailed but still easily understandable presentation of the algorithms we refer to \cite{RefWork:4}.}: as explained above both runs in time $O(|\mathbb{G}|^{\frac{1}{2}})$, but the first is exponential also in the memory required to run, making it absolutely impractical. But let's focus on the run time of the algorithms on the Bitcoin's curve: assuming that the scalar multiplication requires constant time on the order of milliseconds, solving the ECDLP would require time proportional to $10^{-3} * 2^{128} \simeq 10^{35}$ seconds or around $10^{28}$ years. It is easy to see that even relying on optimizations, super computers or parallelization, nowadays it is computationally infeasible to solve the ECDLP.


\documentclass[11pt, a4paper]{article}

\usepackage{tikz}
\usetikzlibrary{arrows,automata, shapes, petri}
\usepackage{amsmath}

\begin{document}

\title{BACKPROPAGATION}
\date{}
\maketitle

Backpropagation is a computer algorithm used for supervised training of Artificial Neural Networks. It is based on two ideas - the Chain Rule of derivatives in Calculus and Dynamic Programming.

\section{Notation}

\begin{center}
	\begin{tikzpicture}[->,>=stealth',shorten >=1pt,auto,
		semithick]

		\node[state] (I1)                    {$\ \ a^1_1$};
		\node[state] (I2) [below of=I1, node distance=5em] {$\ \ a^1_2$};
		\node[state] (I3) [below of=I2, node distance=5em] {$\ \ a^1_3$};

		\node[state] (J1) [above right of=I1, node distance=8em]  {$z^2_1|a^2_1$};
		\node[state] (J2) [below of=J1, node distance=7em] {$z^2_2|a^2_2$};
		\node[state] (J3) [below of=J2, node distance=7em] {$z^2_3|a^2_3$};
		\node[state] (J4) [below of=J3, node distance=7em] {$z^2_4|a^2_4$};

		\node[state] (K1) [below right of=J1, node distance=10em]  {$z^3_1|a^3_1$};
		\node[state] (K2) [below of=K1, node distance=7em] {$z^3_2|a^3_2$};

		\node [rectangle, node distance=2cm] (I) [above of=I1] {1};
		\node [rectangle, node distance=2cm] (J) [above of=J1] {2};
		\node [rectangle, node distance=2cm] (K) [above of=K1] {3};

		\path (I2) edge  node {$w^2_{32}$} (J3);
		\path (J4) edge  node {$w^3_{24}$} (K2);

	\end{tikzpicture}
\end{center}

The layers are of the Neural Network are denoted by positive integers with input layer denoted by $1$, the first hidden layer by $2$ and so on till the output layer. The inputs in the input layer are denoted by $a^1_{i}$ as shown above. For all the other neurons in other layers, $z^l_i$ denotes the weighted sum of the neuron activations in the previous layer and $a^l_i$ denotes that neuron's activation. The weights are superscripted by the index of the layer of the neuron at the end, and subscripted first by the layer index of the end neuron and then by that of the start neuron.

\section{Forward Pass}

For any layer $l$ which is apart from the input layer, it can be written,

\begin{align*}
	z^l_i = \left( \sum_j w^l_{ij}a^{l-1}_j \right)  + b^l_i
\end{align*}

Taking examples from the figure,

\begin{align*}
	z^2_1 = (w^2_{11}a^1_1  + w^2_{12}a^1_2  +  w^2_{13}a^1_3) + b^2_1 \\
	z^2_2 = (w^2_{21}a^1_1  + w^2_{22}a^1_2  +  w^2_{23}a^1_3) + b^2_2 \\
	z^2_3 = (w^2_{31}a^1_1  + w^2_{32}a^1_2  +  w^2_{33}a^1_3) + b^2_3 \\
	z^2_4 = (w^2_{41}a^1_1  + w^2_{42}a^1_2  +  w^2_{43}a^1_3) + b^2_4
\end{align*}

These equations can be rewritten in matrix form as follows,

\begin{align*}
	\begin{pmatrix} z^2_1 \\ z^2_2 \\ z^2_3 \\z^2_4 \end{pmatrix} = \begin{pmatrix}  w^2_{11} & w^2_{12} & w^2_{13} \\ w^2_{21} & w^2_{22} & w^2_{23} \\ w^2_{31} & w^2_{32} & w^2_{33} \\w^2_{41} & w^2_{42} & w^2_{43}  \end{pmatrix} \begin{pmatrix} a^1_1 \\ a^1_2 \\ a^1_3 \\ \end{pmatrix} + \begin{pmatrix} b^2_1 \\ b^2_2 \\ b^2_3 \\b^2_4 \end{pmatrix}
\end{align*}

or

\begin{align*}
	\boldsymbol{Z}^2 = \boldsymbol{W}^2 \boldsymbol{A}^1 + \boldsymbol{B}^2
\end{align*}

Generalizing for all layers,

\begin{align}
	\boldsymbol{Z}^l = \boldsymbol{W}^l \boldsymbol{A}^{l-1} + \boldsymbol{B}^l
\end{align}

Activation function $\sigma$ is applied to $z^l_i$ to yield $a^l_i$.

\begin{align*}
	a^2_1 = \sigma(z^2_1) \\
	a^2_2 = \sigma(z^2_2) \\
	a^2_3 = \sigma(z^2_3) \\
	a^2_4 = \sigma(z^2_4) \\
\end{align*}

which can be written in matrix form as,

\begin{align*}
	\boldsymbol{Z}^2 = \sigma(\boldsymbol{A}^2)
\end{align*}

or more generally as,

\begin{align}
	\boldsymbol{Z}^l = \sigma(\boldsymbol{A}^l)
\end{align}

Given $\boldsymbol{A}^1$, any $\boldsymbol{Z}^l$ or $\boldsymbol{A}^l$ can be calculated using the two equations (1) and (2). This completes the analysis of the forward pass of backpropagation algorithm.

\section{Backward Pass}

Imagine a point in time where the weights and biases of a neural network are fixed. Now, one training example is taken and its inputs are fed into the neural network. Using the the forward pass equations, the neural network's output is computed. Usually, this output is different from the training example's expected output and thus the goal of backpropagation is to calucate the partial derivates of the cost function w.r.t. each weight and bias for them to be tweaked optimally.

\subsection{Cost Function}

The notion of the difference between the expected output of a training example and the actual output from neural network is formalized as follows,

\begin{align*}
	C & = \frac{1}{2} ||\boldsymbol{Y} - \boldsymbol{A}^L||^2 \\
	  & = \frac{1}{2}\sum_j (y_j - a^L_j)^2
\end{align*}

\subsection {Backpropagation}

\begin{align*}
	\frac{\partial C}{\partial z^L_j} & = (a^L_j - y_j) \frac{\partial a^L_j}{\partial z^L_j} \\
	                                  & = (a^L_j - y_j) \times \sigma^{'} (z^L_j)
\end{align*}

Denoting $\frac{\partial C}{\partial z^L_j}$ by $\delta^L_j$ and $(\delta^L_1, \delta^L_2, .. )^T$ by $\boldsymbol{\Delta^L}$, it can be written

\begin{align}
	\boldsymbol{\Delta^L} = (\boldsymbol{A^L} - \boldsymbol{Y}) \odot \sigma^{'}(\boldsymbol{Z^L})
\end{align}

where $\odot$ denotes element-wise matrix product.

Can $\boldsymbol{\Delta^L}$ somehow be related to $\boldsymbol{\Delta^{L-1}}$?

\begin{align*}
	\delta^{L-1}_j & = \frac{\partial C}{\partial z^{L-1}_j}                                                                                \\
	               & = \frac{\partial C}{\partial a^{L-1}_j} \frac{\partial a^{L-1}_j}{\partial z^{L-1}_j}                                                \\
	               & = \frac{\partial C}{\partial a^{L-1}_j} \times \sigma^{'}(z^{L-1}_j)                                                                 \\
	               & = \left( \sum_i \frac{\partial z^{L}_i}{\partial a^{L-1}_j} \frac{\partial C}{\partial z^{L}_i} \right) \times \sigma^{'}(z^{L-1}_j) \\
	               & = \left( \sum_i w^L_{ij} \delta^L_i \right) \times \sigma^{'}(z^{L-1}_j)                                                             \\
\end{align*}

For the above example, the equations become

\begin{align*}
	\begin{pmatrix} \delta^{L-1}_1 \\ \delta^{L-1}_2 \\ \delta^{L-1}_3 \\ \delta^{L-1}_4 \end{pmatrix} &= \begin{pmatrix}  w^L_{11} & w^L_{21} \\ w^L_{12} & w^L_{22} \\ w^L_{13} & w^L_{23} \\w^L_{14} & w^L_{24}  \end{pmatrix} \begin{pmatrix} \delta^L_1 \\ \delta^L_2 \end{pmatrix} \odot \begin{pmatrix} \sigma^{'}(z^{L-1}_1) \\ \sigma^{'}(z^{L-1}_2) \\ \sigma^{'}(z^{L-1}_3) \\ \sigma^{'}(z^{L-1}_4) \end{pmatrix} \\
	\boldsymbol{\Delta^{L-1}} &= \boldsymbol{W^L}^T \boldsymbol{\Delta^L} \odot \sigma^{'}(\boldsymbol{Z^{L-1}})
\end{align*}

The equation can be generalized to,

\begin{align}
	\boldsymbol{\Delta^{l}} & = \boldsymbol{W^{l+1}}^T \boldsymbol{\Delta^{l+1}} \odot \sigma^{'}(\boldsymbol{Z^{l}})
\end{align}

Using equation (3) and (4) $\boldsymbol{\Delta^{l}}$ can be calculated for any $l$.

\begin{align}
	\frac{\partial C}{\partial w^l_{jk}} & = \frac{\partial z^l_{j}}{\partial w^l_{jk}} \frac{\partial C}{\partial z^l_{j}}
	= a^{l-1}_k \delta^l_j \\
	\frac{\partial C}{\partial b^l_{j}}  & = \frac{\partial z^l_{j}}{\partial b^l_{j}} \frac{\partial C}{\partial z^l_{j}} = \delta^l_j
\end{align}

Hence, equations (3) and (4) ``propagate" the error backwards and then with help of equations (5) and (6), partial derivative of the error w.r.t. any neural network weight or bias can be calculated.

\section{Conclusion}

Backpropagation is famous because it can quickly calculate partial derivatives to optimally adjust the parameters of a neural network. In the description above, backpropagation calculates the derivatives w.r.t only one training example. In practice training examples are bunched into mini-batches and partials calculated for each example in a mini batch are averaged for network parameter adjustment.

\end{document}