\subsection{The Boyer-Moore Majority-Vote Algorithm}
\label{subsec:boyer_moore}

In this section we demonstrate how to use Haskell as an advanced macro language
on top of Copilot by implementing an algorithm for solving the voting problem
in Copilot.

Reliability in mission critical software is often improved by replicating
the same computations on separate hardware and by doing a vote in the end
based on the output of each system. The majority vote problem consists of
determining if in a given list of votes there is a candidate that has more
than half of the votes, and if so, of finding this candidate.

The Boyer-Moore Majority Vote Algorithm \cite{MooreBoyer82,Hesselink2005} solves
the problem in linear time and constant memory. It does so in two passes: The
first pass chooses a candidate; and the second pass asserts that the
found candidate indeed holds a majority.

The algorithm for the first pass involves the sequence of elements we are interpreting,
a single element that represents the current majority, and a counter,
which is initially set to zero. The algorithm is as follows:
\begin{itemize}
\item Initialize an element $m$ and a counter $i$ where $i=0$
\item For each element x of the input sequence:
	\begin{itemize}
	\item If $i=0$ then let $m=x$ and $i=1$
	\item else if $m=x$ then increment $i$
	\item else let $i = i-1$
	\end{itemize}
\item Return $m$
\end{itemize}

This algorithm will produce an output even if there is no majority, which is why
the second pass is needed to verify that the output of the first pass is valid.

\begin{figure*}[!htb]
\begin{lstlisting}[language = Copilot, frame = none]
majorityPure :: Eq a => [a] -> a
majorityPure []     = error "majorityPure: empty list!"
majorityPure (x:xs) = majorityPure' xs x 1

majorityPure' []     can _   = can
majorityPure' (x:xs) can cnt =
  let
    can' = if cnt == 0 then x else can
    cnt' = if cnt == 0 || x == can then succ cnt else pred cnt
  in
    majorityPure' xs can' cnt'
\end{lstlisting}
\caption{The first pass of the majority vote algorithm in Haskell.}
\label{fig:majority_pure}
\end{figure*}

\begin{figure*}[!htb]
\begin{lstlisting}[language = Copilot, frame = none]
aMajorityPure :: Eq a => [a] -> a -> Bool
aMajorityPure xs can = aMajorityPure' 0 xs can > length xs `div` 2

aMajorityPure' cnt []     _   = cnt
aMajorityPure' cnt (x:xs) can =
  let
    cnt' = if x == can then cnt+1 else cnt
  in
    aMajorityPure' cnt' xs can
\end{lstlisting}
\caption{The second pass of the majority vote algorithm in Haskell.}
\label{fig:amajority_pure}
\end{figure*}

The first pass can be implemented
in Haskell as shown in Figure \ref{fig:majority_pure}. The second pass, which
simply checks that a candidate has more than half of the votes, is
straightforward to implement and is shown in Figure \ref{fig:amajority_pure}.
E.g. applying {\tt majorityPure} on the string {\tt AAACCBBCCCBCC} yields {\tt
  C}, which {\tt aMajorityPure} can confirm is in fact a majority.

\begin{figure*}[!htb]
\begin{lstlisting}[language = Copilot, frame = none]
majority :: (Eq a, Typed a) => [Stream a] -> Stream a
majority []     = error "majority: empty list!"
majority (x:xs) = majority' xs x 1

majority' []     can _   = can
majority' (x:xs) can cnt =
  local
    (if cnt == 0 then x else can) $
      \ can' ->
        local (if cnt == 0 || x == can then cnt+1 else cnt-1) $
          \ cnt' ->
            majority' xs can' cnt'
\end{lstlisting}
\caption{The first pass of the majority vote algorithm in Copilot.}
\label{fig:majority}
\end{figure*}

\begin{figure*}[!htb]
\begin{lstlisting}[language = Copilot, frame = none]
aMajority :: (Eq a, Typed a) => [Stream a] -> Stream a -> Stream Bool
aMajority xs can = aMajority' 0 xs can > (fromIntegral (length xs) `div` 2)

aMajority' cnt []     _   = cnt
aMajority' cnt (x:xs) can =
  local
    (if x == can then cnt+1 else cnt) $
      \ cnt' ->
        aMajority' cnt' xs can
\end{lstlisting}
\caption{The second pass of the majority vote algorithm in Copilot.}
\label{fig:amajority}
\end{figure*}
% $

When implementing the majority vote algorithm for Copilot, we can use reuse
almost all of the code from the Haskell implementation. However, as functions
in Copilot are macros that are expanded at compile time, care must
be taken in order to avoid an explosion in the code size. Hence, instead of
using Haskell's built-in \emph{let}-blocks, we use explicit sharing, as
described in Section~\ref{sec:explicit_sharing}. The Copilot implementations
of the first and the second pass are given in Figure \ref{fig:majority} and
Figure \ref{fig:amajority} respectively. Comparing the Haskell implementation
with the Copilot implementation, we see that the code is almost identical,
except for the type signatures and the explicit sharing annotations.


\documentclass{article}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{tikz}
\usetikzlibrary{positioning}

\definecolor {processblue}{cmyk}{0.96, 0, 0, 0}

\begin{document}
	\section{Regular languages}
	\subsection{Deterministic Finite Automaton}
	$\Sigma$ a finite non-empty set called an \textbf{alphabet}. Typically $\Sigma = {0,1}$. \\
	A \textbf{string} $w$ is a finite sequence of symbols -aka. alphabets) in $\Sigma$. \\
	We have:
	\[\epsilon := \text{zero symbols} \]
	\[|w| = \text{number of symbosl in } w \]
	\[\Sigma^{k} := \{ w : w \text{ string over $\Sigma$ } | \; |w| = k \} \]
	\[\Sigma^{\star} := \bigcup_{k \geq 0}{\Sigma^{k}} \text{, } \Sigma^{} := \bigcup_{k \geq 1}{\Sigma^{k}} \]
	A \textbf{language} $L$ over $\Sigma$ is a subset of $\Sigma^{\star}$. The concatenation of $x=x_{1} \dots x_{n}$ and $y=y_{1} \dots y_{m}$ is
	\[ xy = x_{1} \dots x_{n} y_{1} \dots y_{m} \]
	Note : A language is called a \textbf{problem} when strings are given some interpretation (e.g. view $w \in \Sigma^{\star}$ as the binary representation of an integer).
	\par
	An \textbf{automaton} is an abstract model of computation :
	\begin{center}
	\begin{tikzpicture}[-latex ,auto ,node distance =2 cm and 3cm ,on grid ,
	semithick ,
	state/.style ={ circle ,top color =white , bottom color = processblue!20 ,
		draw,processblue , text=blue , minimum width = 1 cm}]
	\node[state] (A) {$q_{1}$};
	\node[state] (B) [right = of A]{$q_{2}$};
	\node[state] (C) [right = of B]{$q_{3}$};
	\draw[->, >=latex] (-2, 0) to (A);
	\draw (B) circle(0.6);
	\path (A) edge [loop above] node[above] {$0$} (A);
	\path (A) edge node[above] {$1$} (B);
	\path (B) edge [loop above] node(above) {$1$} (B);
	\path (B) edge [bend left = 15] node(above) {$0$} (C);
	\path (C) edge [bend left = 20] node(below) {$0,1$} (B);
	\end{tikzpicture}
	\end{center}

	Terminology:
	\begin{itemize}
		\item Transition diagram
		\item Start arrow
		\item Accept state
	\end{itemize}

	It reads an input $w=w_{1} \dots w_{n}$ from left to right. It follows the arcs according to the $w_{i}$ starting at the start state. It accepts IFF it ends up in an accepting state after reading the whole input. It rejects otherwise.
	\par
	Formally, a \textbf{Deterministic Finite Automaton (DFA)} is a 5-tuple $\mathcal{D}=(Q, \sigma, \delta, q_{0}, F)$ where:
	\begin{itemize}
		\item $Q$ is a finite set of states
		\item $\Sigma$ is an alphabet
		\item $\delta : Q \times \Sigma \rightarrow Q$ is a transition function
		\item $q_{0} \in Q$ is the start state
		\item $F \subseteq Q$ is the set of the accepting states
	\end{itemize}
	\par
	We define the \textbf{Extended Transition Function} $\hat{\delta}$ of $D$ as follows :
	\[ \hat{\delta}(q,\epsilon) := q \text{, } \hat{\delta}(q,xa) := \delta (\hat{\delta}(q,x), a) \]
	We define the \textbf{language of $D$} as follows :
	\[\mathcal{L}(\mathcal{D}) := \{ w \in \Sigma^{\star} \; | \; \hat{\delta}(q_{0}, w) \in F\}\]
	We say that $D$ recognizes a language $L$ if $L = \mathcal{L}(\mathcal{D})$
	We say $L \subseteq \Sigma^{\star}$ is \textbf{regular} if $L=\mathcal{L}(\mathcal{D}) \text{ for some } D$

	\subsection{Non-Deterministic Finite Automaton}
	\textbf{Nondeterminism :} A generally useful concept in studying computation. It gives the power of being in several states at once.
	Example:
	\begin{center}
		\begin{tikzpicture}[-latex ,auto ,node distance =2 cm and 3cm ,on grid ,
		semithick ,
		state/.style ={ circle ,top color =white , bottom color = processblue!20 ,
			draw,processblue , text=blue , minimum width = 1 cm}]
		\node[state] (A) {$q_{0}$};
		\node[state] (B) [right = of A] {$q_{1}$};
		\node[state] (c) [right = of B] {$q_{2}$};
		\draw (C) circle(0.6);
		\path (A) edge node[above]{$0$} (B);
		\path (B) edge node[above]{$1$} (C);
		\path (A) [loop above] edge node[above]{$0, 1$} (A);
		\end{tikzpicture}
	\end{center}
	This automaton accepts any string ending in $01$.
	\par
	Formally : A \textbf{NFA} is a 5-tuple $\mathcal{N} = (Q, \Sigma, \delta, q_{0}, F)$ where $Q, \Sigma,  q_{0}, F$ are defined as previously and $\delta : Q \times \Sigma \rightarrow \mathcal{P}(Q)$ is the transition function.
	\par
	The extended transition function is:
	\[ \hat{\delta}(q, \epsilon) = \{ q \} \text{, } \hat{\delta}(q, xa) = \bigcup_{p \in \hat{\delta}(q, x)}{\delta(p, a)} \]
	We let:
	\[ \mathcal{L}(\mathcal{N}) = \{ w \in \Sigma^{\star} \; | \; \hat{\delta}(q, xa) \cap F \neq \emptyset \} \]

	\subsection{Equivalence of DFAs and NFAs}
	Now, we want to find a way to convert an \textbf{NFA} $\mathcal{N}$ into a \textbf{DFA}, ie. find some $\mathcal{D}$ DFA such that $\mathcal{L}(\mathcal{N}) = \mathcal{L}(\mathcal{D})$ \\
	\textbf{Construction :} \\
	Let $\mathcal{N} = (Q_{\mathcal{N}}, \Sigma, \delta_{\mathcal{N}}, q_{0}, F )$. Consider $\mathcal{D} = (Q_{\mathcal{D}}, \Sigma, \delta_{\mathcal{D}}, q_{0}, F )$ as follows :
	\begin{itemize}
		\item $Q_{\mathcal{D}} = \mathcal{P}(Q_{\mathcal{N}})$
		\item $\delta_{\mathcal{D}}$ so that : $\forall a \in \Sigma, \forall s \in Q_{\mathcal{D}}$, we have :
			\[ \delta_{\mathcal{D}}(s, a) := \bigcup_{p \in S}{\delta_{\mathcal{N}}(p,a)} \]
		\item $F_{\mathcal{D}} = \{ s \subseteq Q_{\mathcal{N}} \; | \; s \cap F_{\mathcal{N}} \neq \emptyset \}$
	\end{itemize}
	\textbf{Theorem :} These automaton denote the same language. $\blacksquare$ \newpage
	\textbf{Note :} When converting, the number of states grows exponentially.
	\subsection{$\mathcal{E}$-NFA}
	\textbf{Idea :} Allow transition without reading any input symbol (ie. reading $\epsilon$). \\
	\textbf{Example :} One can simply build an automaton denoting the union of two regular languages.
	\begin{center}
		\begin{tikzpicture}[-latex ,auto ,node distance =2 cm and 3cm ,on grid ,
		semithick ,
		state/.style ={ circle ,top color =white , bottom color = processblue!20 ,
			draw,processblue , text=blue , minimum width = 1 cm}]
		\node[state] (A) {$q_{0}$};
		\node[state] (B) [above right = of A] {};
		\node[state] (C) [right = of B] {};
		\node[state] (D) [below right = of A] {};
		\node[state] (E) [right = of D] {};
		\draw (C) circle(0.6);
		\draw (E) circle(0.6);
		\path (A) edge node[above]{$\epsilon$} (B);
		\path (A) edge node[above]{$\epsilon$} (D);
		\draw[dashed] (B) -- (C);
		\draw[dashed] (D) -- (E);
		\end{tikzpicture}
	\end{center}
	\textbf{Definition :} $\mathcal{E}-Closure(q)= \{ q' \in Q \; | \; q'$ can be reached from $q$ by following only $\epsilon$-transitions over multiple transitions\}. We can hence extend the definition of the extended transition function to $\epsilon$-NFAs by having:
	\[ \hat{\delta}(q, \epsilon) := \mathcal{E}-Closure(q) \]
\end{document}


\chapter{Meromorphic functions}
\section{The second nicest functions on earth}
If holomorphic functions are like polynomials,
then \emph{meromorphic} functions are like rational functions.
Basically, a meromorphic function is a function of the form
$ \frac{A(z)}{B(z)} $
where $A , B:  U \to \CC$ are holomorphic and $B$ is not zero.
The most important example of a meromorphic function is $\frac 1z$.

We are going to see that meromorphic functions behave
like ``almost-holomorphic'' functions.
Specifically, a meromorphic function $A/B$ will be holomorphic at all points
except the zeros of $B$ (called \emph{poles}).
By the identity theorem, there cannot be too many zeros of $B$!
So meromorphic functions can be thought of as ``almost holomorphic''
(like $\frac 1z$, which is holomorphic everywhere but the origin).
We saw that
\[ \frac{1}{2\pi i} \oint_{\gamma} \frac 1z \; dz =  1 \]
for $\gamma(t) = e^{it}$ the unit circle.
We will extend our results on contours to such situations.

It turns out that, instead of just getting $\oint_\gamma f(z) \; dz = 0$
like we did in the holomorphic case,
the contour integrals will actually be used to
\emph{count the number of poles} inside the loop $\gamma$.
It's ridiculous, I know.

\section{Meromorphic functions}
\prototype{$\frac 1z$, with a pole of order $1$ and residue $1$ at $z=0$.}

Let $U$ be an open subset of $\CC$ again.
\begin{definition}
	A function $f : U \to \CC$ is \vocab{meromorphic}
	if there exists holomorphic functions $A, B \colon U \to \CC$
	with $B$ not identically zero in any open neighborhood,
	and $f(z) = A(z)/B(z)$ whenever $B(z) \ne 0$.
\end{definition}
Let's see how this function $f$ behaves.
If $z \in U$ has $B(z) \neq 0$,
then in some small open neighborhood the function $B$ isn't zero
at all, and thus $A/B$ is in fact \emph{holomorphic};
thus $f$ is holomorphic at $z$.
(Concrete example: $\frac 1z$ is holomorphic
in any disk not containing $0$.)

On the other hand, suppose $p \in U$ has $B(p) = 0$: without loss of generality, $p=0$
to ease notation.
By using the Taylor series at $p=0$ we can put
\[ B(z) = c_k z^k + c_{k+1} z^{k+1} + \dots \]
with $c_k \neq 0$
(certainly some coefficient is nonzero since $B$ is not identically zero!).
Then we can write
\[ \frac{1}{B(z)} = \frac{1}{z^k} \cdot \frac{1}{c_k + c_{k+1}z + \dots}. \]
But the fraction on the right is a
holomorphic function in this open neighborhood!
So all that's happened is that we have an extra $z^{-k}$ kicking around.

%We want to consider functions $f$ defined on all points in $U$
%except for a set of ``isolated'' singularities;
%for example, something like \[ \frac{1}{z(z+1)(z^2+1)} \] which is defined
%for all $z$ other than $z=0$, $z=-1$ and $z=i$.
%Or even \[ \frac{1}{\sin(2\pi z)}, \] which is defined at every $z$ which is \emph{not} an integer.
%% Even though there's infinitely many points, they are not really that close together.

This gives us an equivalent way of viewing meromorphic functions:

\begin{definition}
	Let $f : U \to \CC$ as usual.
	A \vocab{meromorphic} function is a
	function which is holomorphic on $U$
	except at an isolated set $S$ of points
	(meaning it is holomorphic as a function $U \setminus S \to \CC$).
	For each $p \in S$, called a \vocab{pole} of $f$,
	the function $f$ must admit a \vocab{Laurent series},
	meaning that
	\[
		f(z) =
		\frac{c_{-m}}{(z-p)^m}
		+ \frac{c_{-m+1}}{(z-p)^{m-1}}
		+ \dots
		+ \frac{c_{-1}}{z-p} + c_0 + c_1 (z-p) + \dots
	\]
	for all $z$ in some open neighborhood of $p$,
	other than $z = p$.
	Here $m$ is a positive integer (and $c_{-m} \neq 0$).
\end{definition}
Note that the trailing end \emph{must} terminate.
By ``isolated set'', I mean that we can draw
open neighborhoods around each pole in $S$,
in such a way that no two open neighborhoods intersect.

\begin{example}
	[Example of a meromorphic function]
	Consider the function \[ \frac{z+1}{\sin z}. \]
	It is meromorphic, because it is holomorphic everywhere except at the zeros of $\sin z$.
	At each of these points we can put a Laurent series: for example at $z=0$ we have
	\begin{align*}
		\frac{z+1}{\sin z}
		&= (z+1) \cdot \frac{1}{z - \frac{z^3}{3!} + \frac{z^5}{5!} - \dots} \\
		&= \frac 1z \cdot \frac{z+1}{1 - \left(%
			\frac{z^2}{3!} - \frac{z^4}{5!} + \frac{z^6}{7!} - \dots \right)} \\
		&= \frac 1z \cdot (z+1) \sum_{k \ge 0} \left( %
			\frac{z^2}{3!}-\frac{z^4}{5!}+\frac{z^6}{7!}-\dots \right)^k.
	\end{align*}
	If we expand out the horrible sum (which I won't do),
	then you get $\frac 1z$ times a perfectly
	fine Taylor series, i.e.\ a Laurent series.
\end{example}

\begin{abuse}
	We'll often say something like
	``consider the function $f : \CC \to \CC$
	by $z \mapsto \frac 1z$''.
	Of course this isn't completely correct,
	because $f$ doesn't have a value at $z=0$.
	If I was going to be completely rigorous
	I would just set $f(0) = 2015$ or something and move on
	with life, but for all intents
	let's just think of it as ``undefined at $z=0$''.

	Why don't I just write $g : \CC \setminus \{0\} \to \CC$?
	The reason I have to do this is that it's still important
	for $f$ to remember it's ``trying'' to be holomorphic on $\CC$,
	even if isn't assigned a value at $z=0$.
	As a function $\CC \setminus \{0\} \to \CC$ the function $\frac 1z$ is actually holomorphic.
\end{abuse}

\begin{remark}
	I have shown that any function $A(z)/B(z)$
	has this characterization with poles,
	but an important result is
	that the converse is true too:
	if $f : U \setminus S \to \CC$ is holomorphic for some isolated set $S$,
	and moreover $f$ admits a Laurent series at each point in $S$,
	then $f$ can be written as a rational quotient of holomorphic functions.
	I won't prove this here, but it is good to be aware of.
\end{remark}

\begin{definition}
	Let $p$ be a pole of a meromorphic function $f$, with Laurent series
	\[
		f(z) =
		\frac{c_{-m}}{(z-p)^m}
		+ \frac{c_{-m+1}}{(z-p)^{m-1}}
		+ \dots
		+ \frac{c_{-1}}{z-p} + c_0 + c_1 (z-p) + \dots.
	\]
	The integer $m$ is called the \vocab{order} of the pole.
	A pole of order $1$ is called a \vocab{simple pole}.

	We also give the coefficient $c_{-1}$ a name, the \vocab{residue} of $f$ at $p$,
	which we write $\Res(f;p)$.
\end{definition}

The order of a pole tells you how ``bad'' the pole is.
The order of a pole is the ``opposite'' concept of the \vocab{multiplicity} of a \vocab{zero}.
If $f$ has a pole at zero, then its Taylor series near $z=0$ might look something like
\[ f(z) = \frac{1}{z^5} + \frac{8}{z^3} - \frac{2}{z^2} + \frac{4}{z} + 9 - 3z + 8z^2 + \dots \]
and so $f$ has a pole of order five.
By analogy, if $g$ has a zero at $z=0$, it might look something like
\[ g(z) = 3z^3 + 2z^4 + 9z^5 + \dots \]
and so $g$ has a zero of multiplicity three.
These orders are additive: $f(z) g(z)$ still has a pole of order $5-3=2$,
but $f(z)g(z)^2$ is completely patched now, and in fact has a \vocab{simple zero} now
(that is, a zero of degree $1$).

\begin{exercise}
	Convince yourself that orders are additive as described above.
	(This is obvious once you understand that you
	are multiplying Taylor/Laurent series.)
\end{exercise}

Metaphorically, poles can be thought of as ``negative zeros''.

We can now give many more examples.
\begin{example}
	[Examples of meromorphic functions]
	\listhack
	\begin{enumerate}[(a)]
		\ii Any holomorphic function is a meromorphic function which happens to have no poles.
		Stupid, yes.
		\ii The function $\CC \to \CC$ by $z \mapsto 100z\inv$ for $z \neq 0$
		but undefined at zero is a meromorphic function.
		Its only pole is at zero, which has order $1$ and residue $100$.
		\ii The function $\CC \to \CC$ by $z \mapsto z^{-3} + z^2 + z^9$ is also a meromorphic function.
		Its only pole is at zero, and it has order $3$, and residue $0$.
		\ii The function $\CC \to \CC$ by $z \mapsto \frac{e^z}{z^2}$ is meromorphic,
		with the Laurent series at $z=0$ given by
		\[
			\frac{e^z}{z^2}
			= \frac{1}{z^2} + \frac{1}{z} + \frac{1}{2} + \frac{z}{6} + \frac{z^2}{24} + \frac{z^3}{120}
			+ \dots.
		\]
		Hence the pole $z=0$ has order $2$ and residue $1$.
	\end{enumerate}
\end{example}

\begin{example}
	[A rational meromorphic function]
	Consider the function $\CC \to \CC$ given by
	\begin{align*}
		z &\mapsto \frac{z^4+1}{z^2-1} = z^2 + 1 + \frac{2}{(z-1)(z+1)} \\
		&= z^2 + 1 + \frac1{z-1} \cdot \frac{1}{1+\frac{z-1}{2}} \\
		&= \frac{2}{z-1} + \frac32 + \frac94(z-1) + \frac{7}{8}(z-1)^2 - \dots
	\end{align*}
	It has a pole of order $1$ and residue $2$ at $z=1$.
	(It also has a pole of order $1$ at $z=-1$; you are invited to compute the residue.)
\end{example}
\begin{example}
	[Function with infinitely many poles]
	The function $\CC \to \CC$ by \[ z \mapsto \frac{1}{\sin(z)} \]
	has infinitely many poles: the numbers $z = 2\pi k$, where $k$ is an integer.
	Let's compute the Laurent series at just $z=0$:
	\begin{align*}
		\frac{1}{\sin(2\pi z)}
		&= \frac{1}{\frac{z}{1!} - \frac{z^3}{3!} + \frac{z^5}{5!} - \dots} \\
		% &= \frac{1/z}{\frac{1}{1!} - \frac{z^2}{3!} + \frac{z^4}{5!} - \dots} \\
		&= \frac 1z \cdot \frac{1}{1 - \left( \frac{z^2}{3!} - \frac{z^4}{5!} + \dots \right)} \\
		&= \frac 1z \sum_{k \ge 0} \left( \frac{z^2}{3!} - \frac{z^4}{5!} + \dots \right)^k.
	\end{align*}
	which is a Laurent series, though I have no clue what the coefficients are.
	You can at least see the residue; the constant term of that huge sum is $1$,
	so the residue is $1$.
	Also, the pole has order $1$.
\end{example}

The Laurent series, if it exists, is unique (as you might have guessed),
and by our result on holomorphic functions it is actually valid for \emph{any}
disk centered at $p$ (minus the point $p$).
The part $\frac{c_{-1}}{z-p} + \dots + \frac{c_{-m}}{(z-p)^m}$ is called the \vocab{principal part},
and the rest of the series $c_0 + c_1(z-p) + \dots$ is called the \vocab{analytic part}.

\section{Winding numbers and the residue theorem}
Recall that for a counterclockwise circle $\gamma$ and a point $p$ inside it, we had
\[
	\oint_{\gamma} (z-p)^m \; dz =
	\begin{cases}
		0 & m \neq -1 \\
		2\pi i & m = -1
	\end{cases}
\]
where $m$ is an integer.
One can extend this result to in fact show that $\oint_\gamma (z-p)^m \; dz = 0$
for \emph{any} loop $\gamma$, where $m \neq -1$.
So we associate a special name for the nonzero value at $m=-1$.
\begin{definition}
	For a point $p \in \CC$ and a loop $\gamma$ not passing through it,
	we define the \vocab{winding number}, denoted $\Wind(p, \gamma)$, by
	\[
		\Wind(\gamma, p) = \frac{1}{2\pi i} \oint_{\gamma} \frac{1}{z-p} \; dz
	\]
\end{definition}
For example, by our previous results we see that if $\gamma$ is a circle, we have
\[
	\Wind(\text{circle}, p)
	=
	\begin{cases}
		1 & \text{$p$ inside the circle} \\
		0 & \text{$p$ outside the circle}.
	\end{cases}
\]
If you've read the chapter on fundamental groups, then this is just the fundamental group
associated to $\CC \setminus \{p\}$.
In particular, the winding number is always an integer (the proof of this requires the complex logarithm,
so we omit it here).
In the simplest case the winding numbers are either $0$ or $1$.
\begin{definition}
	We say a loop $\gamma$ is \vocab{regular} if $\Wind(p, \gamma) = 1$
	for all points $p$ in the interior of $\gamma$ (for example,
	if $\gamma$ is a counterclockwise circle).
\end{definition}

With all these ingredients we get a stunning generalization of the Cauchy-Goursat theorem:
\begin{theorem}
	[Cauchy's residue theorem]
	Let $f : \Omega \to \CC$ be meromorphic, where $\Omega$ is simply connected.
	Then for any loop $\gamma$ not passing through any of its poles, we have
	\[
		\frac{1}{2\pi i} \oint_{\gamma} f(z) \; dz
		= \sum_{\text{pole $p$}} \Wind(\gamma, p) \Res(f; p).
	\]
	In particular, if $\gamma$ is regular then the contour integral
	is the sum of all the residues, in the form
	\[
		\frac{1}{2\pi i} \oint_{\gamma} f(z) \; dz
		= \sum_{\substack{\text{pole $p$} \\ \text{inside $\gamma$}}}  \Res(f; p).
	\]
\end{theorem}
\begin{ques}
	Verify that this result coincides
	with what you expect when you integrate $\oint_\gamma cz\inv \; dz$
	for $\gamma$ a counter-clockwise circle.
\end{ques}

The proof from here is not really too impressive -- the ``work'' was already
done in our statements about the winding number.
\begin{proof}
	Let the poles with nonzero winding number be $p_1, \dots, p_k$ (the others do not affect the sum).\footnote{
		To show that there must be finitely many such poles: recall that all our contours $\gamma : [a,b] \to \CC$
		are in fact bounded, so there is some big closed disk $D$ which contains all of $\gamma$.
		The poles outside $D$ thus have winding number zero.
		Now we cannot have infinitely many poles inside the disk $D$, for $D$ is compact and the
		set of poles is a closed and isolated set!}
	Then we can write $f$ in the form
	\[
		f(z) = g(z) + \sum_{i=1}^k P_i\left( \frac{1}{z-p_i} \right)
	\]
	where $P_i\left( \frac{1}{z-p_i} \right)$ is the principal part of the pole $p_i$.
	(For example, if $f(z) = \frac{z^3-z+1}{z(z+1)}$ we would write $f(z) = (z-1) + \frac1z - \frac1{1+z}$.)

	The point of doing so is that the function $g$ is holomorphic (we've removed all the ``bad'' parts), so
	\[ \oint_{\gamma} g(z) \; dz = 0 \]
	by Cauchy-Goursat.

	On the other hand, if $P_i(x) = c_1x + c_2x^2 + \dots + c_d x^d$ then
	\begin{align*}
		\oint_{\gamma} P_i\left( \frac{1}{z-p_i} \right) \; dz
		&=
		\oint_{\gamma} c_1 \cdot \left( \frac{1}{z-p_i} \right) \; dz
		+ \oint_{\gamma} c_2 \cdot \left( \frac{1}{z-p_i} \right)^2 \; dz
		+ \dots \\
		&= c_1 \cdot \Wind(\gamma, p_i) + 0 + 0 + \dots \\
		&= \Wind(\gamma, p_i) \Res(f; p_i).
	\end{align*}
	which gives the conclusion.
\end{proof}

\section{Argument principle}
One tricky application is as follows.
Given a polynomial $P(x) = (x-a_1)^{e_1}(x-a_2)^{e_2}\dots(x-a_n)^{e_n}$, you might know that we have
\[ \frac{P'(x)}{P(x)} = \frac{e_1}{x-a_1} + \frac{e_2}{x-a_2} + \dots + \frac{e_n}{x-a_n}. \]
The quantity $P'/P$ is called the \vocab{logarithmic derivative}, as it is the derivative of $\log P$.
This trick allows us to convert zeros of $P$ into poles of $P'/P$ with order $1$;
moreover the residues of these poles are the multiplicities of the roots.

In an analogous fashion, we can obtain a similar result for any meromorphic function $f$.
\begin{proposition}
	[The logarithmic derivative]
	Let $f : U \to \CC$ be a meromorphic function.
	Then the logarithmic derivative $f'/f$ is meromorphic as a function from $U$ to $\CC$;
	its zeros and poles are:
	\begin{enumerate}[(i)]
		\ii A pole at each zero of $f$ whose residue is the multiplicity, and
		\ii A pole at each pole of $f$ whose residue is the negative of the pole's order.
	\end{enumerate}
\end{proposition}
Again, you can almost think of a pole as a zero of negative multiplicity.
This spirit is exemplified below.
\begin{proof}
	Dead easy with Taylor series.
	Let $a$ be a zero/pole of $f$, and WLOG set $a=0$ for convenience.
	We take the Taylor series at zero to get
	\[ f(z) = c_k z^k + c_{k+1} z^{k+1} + \dots \] % chktex 25
	where $k < 0$ if $0$ is a pole and $k > 0$ if $0$ is a zero.
	Taking the derivative gives
	\[ f'(z) = kc_k z^{k-1} + (k+1)c_{k+1}z^{k} + \dots. \]
	Now look at $f'/f$; with some computation, it equals
	\[
		\frac{f'(z)}{f(z)}
		= \frac 1z \frac{kc_k + (k+1)c_{k+1}z + \dots}{c_k + c_{k+1}z + \dots}.
	\]
	So we get a simple pole at $z=0$, with residue $k$.
\end{proof}

Using this trick you can determine the number of zeros and poles inside a regular closed curve,
using the so-called Argument Principle.

\begin{theorem}
	[Argument principle]
	\label{thm:arg_principle}
	Let $\gamma$ be a regular curve.
	Suppose $f : U \to \CC$ is meromorphic inside and on $\gamma$, and
	none of its zeros or poles lie on $\gamma$.
	Then
	\[
		\frac{1}{2\pi i} \oint_\gamma \frac{f'}{f} \; dz
		= Z - P
	\]
	where $Z$ is the number of zeros inside $\gamma$ (counted with multiplicity)
	and $P$ is the number of poles inside $\gamma$ (again with multiplicity).
\end{theorem}
\begin{proof}
	Immediate by applying Cauchy's residue theorem alongside the preceding proposition.
	In fact you can generalize to any curve $\gamma$ via the winding number:
	the integral is
	\[ \frac{1}{2\pi i} \oint_\gamma \frac{f'}{f} \; dz
		= \sum_{\text{zero $z$}} \Wind(\gamma,z)
		- \sum_{\text{pole $p$}} \Wind(\gamma,p) \]
	where the sums are with multiplicity.
\end{proof}

Thus the Argument Principle allows one to count zeros and poles inside any region of choice.

Computers can use this to get information on functions whose values can be computed but whose behavior as a whole
is hard to understand.
Suppose you have a holomorphic function $f$, and you want to understand where its zeros are.
Then just start picking various circles $\gamma$.
Even with machine rounding error, the integral will be close enough to the true integer value that
we can decide how many zeros are in any given circle.
Numerical evidence for the Riemann Hypothesis (concerning the zeros of the Riemann zeta function)
can be obtained in this way.

\section{Philosophy: why are holomorphic functions so nice?}
All the fun we've had with holomorphic and meromorphic functions comes down to the fact that
complex differentiability is such a strong requirement.
It's a small miracle that $\CC$, which \emph{a priori} looks only like $\RR^2$,
is in fact a field.
Moreover, $\RR^2$ has the nice property that one can draw nontrivial loops
(it's also true for real functions that $\int_a^a f \; dx = 0$, but this is not so interesting!),
and this makes the theory much more interesting.

As another piece of intuition from Siu\footnote{Harvard professor.}:
If you try to get (left) differentiable functions over \emph{quaternions},
you find yourself with just linear functions.

\section\problemhead
% Looman-Menchoff theorem?

\begin{problem}
	[Fundamental theorem of algebra]
	Prove that if $f$ is a nonzero polynomial of degree $n$
	then it has $n$ roots.
\end{problem}

\begin{dproblem}
	[Rouch\'e's theorem]
	Let $f, g \colon U \to \CC$ be holomorphic functions,
	where $U$ contains the unit disk.
	Suppose that $\left\lvert f(z) \right\rvert > \left\lvert g(z) \right\rvert$
	for all $z$ on the unit circle.
	Prove that $f$ and $f+g$ have the same number of zeros
	which lie strictly inside the unit circle (counting multiplicities).
\end{dproblem}

\begin{problem}
	[Wedge contour]
	\gim
	For each odd integer $n$, evaluate the improper integral
	\[ \int_0^\infty \frac{1}{1+x^{n}} \; dx. \]
	\begin{hint}
		This is called a ``wedge contour''.
		Try to integrate over a wedge shape
		consisting of a sector of a circle of radius $r$,
		with central angle $\frac{2\pi}{n}$.
		Take the limit as $r \to \infty$ then.
	\end{hint}
	\begin{sol}
		See \url{https://math.stackexchange.com/q/242514/229197},
		which does it with $2019$ replaced by $3$.
	\end{sol}
\end{problem}

\begin{problem}
	[Another contour]
	\yod
	Prove that the integral
	\[ \int_{-\infty}^{\infty} \frac{\cos x}{x^2+1} \; dx \]
	converges and determine its value.
	\begin{hint}
		It's $\lim_{a \to \infty} \int_{-a}^{a} \frac{\cos x}{x^2+1} \; dx$.
		For each $a$, construct a semicircle.
	\end{hint}
	% semicircle integral
	% \quad \text{ and } \quad
	% \int_{-\infty}^{\infty} \frac{\sin x}{x^2+1} \; dx
\end{problem}

\begin{sproblem}
	\gim
	Let $f \colon U \to \CC$ be a nonconstant holomorphic function.
	\begin{enumerate}[(a)]
		\ii (Open mapping theorem)
		Prove that $f\im(U)$ is open in $\CC$.\footnote{Thus
			the image of \emph{any}
			open set $V \subseteq U$ is open in $\CC$
			(by repeating the proof for $f \restrict{V}$).}
		\ii (Maximum modulus principle)
		Show that $\left\lvert f \right\rvert$
		cannot have a maximum over $U$.
		That is, show that for any $z \in U$,
		there is some $z' \in U$ such that
		$\left\lvert f(z) \right\rvert < \left\lvert f(z') \right\rvert$.
	\end{enumerate}
	% http://en.wikipedia.org/wiki/Open_mapping_theorem_(complex_analysis)
\end{sproblem}


%!TEX root = ceres-solver.tex
\chapter{Fitting a Curve to Data}
\label{chapter:tutorial:curvefitting}
The examples we have seen until now are simple optimization problems with no data. The original purpose of least squares and non-linear least squares analysis was fitting curves to data. It is only appropriate that we now consider an example of such a problem\footnote{The full code and data for this example can be found in
\texttt{examples/data\_fitting.cc}. It contains data generated by sampling the curve $y = e^{0.3x + 0.1}$ and adding Gaussian noise with standard deviation $\sigma = 0.2$.}. Let us fit some data to the curve
\begin{equation}
	y = e^{mx + c}.
\end{equation}

We begin by defining a templated object to evaluate the residual. There will be a residual for each observation.
\begin{minted}[mathescape]{c++}
class ExponentialResidual {
 public:
  ExponentialResidual(double x, double y)
      : x_(x), y_(y) {}

  template <typename T> bool operator()(const T* const m,
                                        const T* const c,
                                        T* residual) const {
    residual[0] = T(y_) - exp(m[0] * T(x_) + c[0]);  // $y - e^{mx + c}$
    return true;
  }

 private:
  // Observations for a sample.
  const double x_;
  const double y_;
};
\end{minted}
%\caption{Templated functor to compute the residual for the exponential model fitting problem. Note that one instance of the functor is responsible for computing the residual for one observation.}
%\label{listing:exponentialresidual}
%\end{listing}
Assuming the observations are in a $2n$ sized array called \texttt{data}, the problem construction is a simple matter of creating a \texttt{CostFunction} for every observation.
\clearpage
\begin{minted}{c++}
double m = 0.0;
double c = 0.0;

Problem problem;
for (int i = 0; i < kNumObservations; ++i) {
  problem.AddResidualBlock(
      new AutoDiffCostFunction<ExponentialResidual, 1, 1, 1>(
          new ExponentialResidual(data[2 * i], data[2 * i + 1])),
      NULL,
      &m, &c);
}
\end{minted}
Compiling and running \texttt{data\_fitting.cc} gives us
\begin{minted}{bash}
 0: f: 1.211734e+02 d: 0.00e+00 g: 3.61e+02 h: 0.00e+00 rho: 0.00e+00 mu: 1.00e-04 li:  0
 1: f: 1.211734e+02 d:-2.21e+03 g: 3.61e+02 h: 7.52e-01 rho:-1.87e+01 mu: 2.00e-04 li:  1
 2: f: 1.211734e+02 d:-2.21e+03 g: 3.61e+02 h: 7.51e-01 rho:-1.86e+01 mu: 8.00e-04 li:  1
 3: f: 1.211734e+02 d:-2.19e+03 g: 3.61e+02 h: 7.48e-01 rho:-1.85e+01 mu: 6.40e-03 li:  1
 4: f: 1.211734e+02 d:-2.02e+03 g: 3.61e+02 h: 7.22e-01 rho:-1.70e+01 mu: 1.02e-01 li:  1
 5: f: 1.211734e+02 d:-7.34e+02 g: 3.61e+02 h: 5.78e-01 rho:-6.32e+00 mu: 3.28e+00 li:  1
 6: f: 3.306595e+01 d: 8.81e+01 g: 4.10e+02 h: 3.18e-01 rho: 1.37e+00 mu: 1.09e+00 li:  1
 7: f: 6.426770e+00 d: 2.66e+01 g: 1.81e+02 h: 1.29e-01 rho: 1.10e+00 mu: 3.64e-01 li:  1
 8: f: 3.344546e+00 d: 3.08e+00 g: 5.51e+01 h: 3.05e-02 rho: 1.03e+00 mu: 1.21e-01 li:  1
 9: f: 1.987485e+00 d: 1.36e+00 g: 2.33e+01 h: 8.87e-02 rho: 9.94e-01 mu: 4.05e-02 li:  1
10: f: 1.211585e+00 d: 7.76e-01 g: 8.22e+00 h: 1.05e-01 rho: 9.89e-01 mu: 1.35e-02 li:  1
11: f: 1.063265e+00 d: 1.48e-01 g: 1.44e+00 h: 6.06e-02 rho: 9.97e-01 mu: 4.49e-03 li:  1
12: f: 1.056795e+00 d: 6.47e-03 g: 1.18e-01 h: 1.47e-02 rho: 1.00e+00 mu: 1.50e-03 li:  1
13: f: 1.056751e+00 d: 4.39e-05 g: 3.79e-03 h: 1.28e-03 rho: 1.00e+00 mu: 4.99e-04 li:  1
Ceres Solver Report: Iterations: 13, Initial cost: 1.211734e+02, \
Final cost: 1.056751e+00, Termination: FUNCTION_TOLERANCE.
Initial m: 0 c: 0
Final   m: 0.291861 c: 0.131439
\end{minted}

\begin{figure}[t]
	\begin{center}
	\includegraphics[width=\textwidth]{fit.pdf}
	\caption{Least squares data fitting to the curve $y = e^{0.3x + 0.1}$. Observations were generated by sampling this curve uniformly in the interval $x=(0,5)$ and adding Gaussian noise with $\sigma = 0.2$.\label{fig:exponential}}
\end{center}
\end{figure}

Starting from parameter values $m = 0, c=0$ with an initial objective function value of $121.173$ Ceres finds a solution $m= 0.291861, c = 0.131439$ with an objective function value of $1.05675$. These values are a a bit different than the parameters of the original model $m=0.3, c= 0.1$, but this is expected. When reconstructing a curve from noisy data, we expect to see such deviations. Indeed, if you were to evaluate the objective function for $m=0.3, c=0.1$, the fit is worse with an objective function value of 1.082425. Figure~\ref{fig:exponential} illustrates the fit.


\chapter{Forecasting}
\label{cha:forecasting}

\section{Prophet}

\subsection{Model overview}

Prophet is a statistical model developed by engineers at Facebook, which mainly fits its models in Stan and has been open-sourced with public APIs for python and R languages\cite{fb_prophet}. It takes a regression-fitting approach to learn the time series and then forecasts by extrapolating such models and adding uncertainty in a Bayesian framework.

It uses a decomposable time series models \cite{harvey1990estimation} with three components: trend, seasonality and holidays:

\begin{equation}
	y(t) = g(t) + s(t) + h(t) + \varepsilon_t
\end{equation}

Where $g(t)$ is a trend function that models non-periodic fluctuations, $s(t)$ capture different kinds of seasonalities, and $h(t)$ represents special situations that could add irregularities to the other components and a random error term $\varepsilon$.

The model is similar to a \ac{gam}\cite{hastie1987generalized}, which allows adding more and more components as needed. The seasonalities are modelled by an exponential smoothing approach\cite{gardner1985exponential}.

\subsection{Trend model}

Prophet has two trend models: a saturating growth one, which is similar to model population growths in ecosystems \cite{hutchinson1978introduction} and a piecewise model to give the flexibility of trend changes in determined learnt changepoints \cite{fb_prophet}.

The basic saturated growth model is given by

\begin{align}\label{eq:sat_growth}
		g(t) &= \frac{C}{1 + \exp\left\{ -k(t-m) \right\}}
\end{align}

\begin{equation*}
	\text{Where} \quad
	C: \text{Carrying capacity}, \quad
	k: \text{Growth rate}, \quad
	m: \text{Offset parameter}
\end{equation*}

Nonetheless, the saturated growth model in (\ref{eq:sat_growth}) does not meet all the usual requirements for some of the applications in which the saturating ceiling varies over time or in which the growth rate is not constant. Therefore, the model has been extended as follows.

Let $s_j=1,\ldots,S$ be the number of changepoints where the growth rate is allowed to change. Then, let $\delta_j$ be the change rate that happens at $s_j$, and $\bm{\delta} \in \mathbb{R}^s$ the vector defined by all $\delta_j$. This means that the growth rate at any time $t$ is the base rate $k$ plus all the adjustments up to $t$

\begin{equation}
	k + \sum_{j=1}^{t < s_j}{\delta_j}
\end{equation}

This can be rewriten as

\begin{align}
	\begin{split}
		k &+ \bm{a}(t)^T\bm{\delta} \\
		\text{Where}\quad a_j &=
		\begin{cases*}
			1\quad,\quad\text{if} \quad t > s_j \\
			0\quad,\quad\text{otherwise}
		\end{cases*}
	\end{split}
\end{align}

Also, when the growth rate is adjusted, the $m$ needs to be adjusted to ensure continuity.

\begin{equation}
	\gamma_j = \left( s_j - m - \sum_{l<j}{\gamma_l} \right) \left(1 - \frac{k+\sum_{l<j}{\delta_l}}{k+\sum_{l \leq j}{\delta_l}} \right)
\end{equation}

Then the linear trend with changepoints is defined by (\ref{eq:piece_trend}) piecewise logistic growth model by (\ref{eq:piece_satgrowth})\cite{fb_prophet}.

\begin{equation}\label{eq:piece_trend}
	g(t) = \left( k + \bm{a}(t)^T \bm{\delta} \right)t + \left( m + \bm{a}(t)^T \bm{\gamma} \right)
\end{equation}

\begin{equation}\label{eq:piece_satgrowth}
	g(t) = \frac{C(t)}
	        {1 + \exp\left\{-(k+\bm{a}(t)^T\bm{\delta})(t-(m+\bm{a}(t)^T \bm{\gamma}))\right\}
	        }
\end{equation}

\subsection{Trend forecast}

After the model has learnt from a history of $T$ time points with $S$ changepoints from which each point has a rate change $\delta_j \sim \text{Laplace}(0,\tau)$.

The trend will keep its last growth rate constant. The forecasts will be made by extrapolating the \ac{gam} and simulating samples from Laplace$(0,\lambda)$ where $\lambda$ is a variance inferred from the data from the maximum likelihood estimate of the rate scale parameter (\ref{eq:lambda_inferr}).

\begin{equation}\label{eq:lambda_inferr}
	\lambda = \frac{1}{2} \sum_{j=1}^{S}{|\delta_j|}
\end{equation}

Once $\lambda$ has been inferred the trend forecast and its uncertainty are obtained from

\begin{equation}\label{eq:trend_forecast}
\forall j > T\quad,\quad
	\begin{cases}
		\delta_j = 0 \quad \text{w.p.} \frac{T-S}{S} \\
		\delta_j \sim \text{Laplace}(0,\lambda) \quad \text{w.p.} \frac{S}{T}
	\end{cases}
\end{equation}

\pagebreak
\subsection{Seasonalities}

By using the \ac{gam} flexibility, it is possible to add different seasonality periods, 365.25 or 7 for yearly and weekly seasonalities -in days measurements-, for example. These seasonalities are modelled by the use of Fourier series\cite{harvey_fourier}.

Therefore, for every given period $P$, it can be defined that.

\begin{equation}
	s(t) = \sum_{n=1}^{N}\left( a_n \cos\left( \frac{2 \pi n t}{P}\right)  + b_n \sin\left( \frac{2 \pi n t}{P}\right) \right)
\end{equation}

Which has $2N$ params to learn:

\begin{equation*}
	\bm{\beta} = \left[ a_1, b_1, a_2, b_2, \ldots, a_N, b_N\right]^T
\end{equation*}

In order to make the structures more manageable, it is defined a matrix of seasonalities comprised of the seasonal vectors for each time step.

\begin{equation}
	\bm{X}(t) = \left[ \cos\left( \frac{2 \pi (1) t}{P}\right), \sin\left( \frac{2 \pi (1) t}{P}\right), \dots , \cos\left( \frac{2 \pi (N) t}{P}\right), \sin\left( \frac{2 \pi (N) t}{P}\right)  \right]
\end{equation}

Thus, the seasonal component is expressed then as.

\begin{align}
	s(t) &= \bm{X}(t) \bm{\beta} \\
	\text{Where} \quad \beta &\sim \mathcal{N}(0, \sigma^2)
\end{align}

\subsection{Holidays and festivities}

Prophet also allows considering non-seasonal events that can significantly affect the forecasts due to changes in peoples behaviour. Namely, Muslim Ramadan, Chinese New Year or the US's Super Bowl.

It is assumed that each holiday is independent. For each holiday $i$, $D_i$ is the set of past and future dates of it.

Same as seasonalities, a regressors matrix is generated indicating whether the time $t$ happens during holiday $i$ and a parameter $\kappa_i$ to model its influence in the forecast.

\begin{align}
	Z(t) &=  \left[   \bm{1}(t \in D_1), \ldots, \bm{1}(t \in D_L) \right] \\
	\therefore \quad h(t) &= Z(t)\bm{\kappa} \\
	\bm{\kappa} &\sim \mathcal{N}(0,\nu^2)
\end{align}

For practical cases, importing the holidays' dates can be done using the python-holidays open-source package \cite{Holidays}, for which an extract of available Swedish holidays is shown in Table \ref{table:sv_holidays}.

\begin{table}[H]
	\centering
	\begin{tabular}{|c|c|}
		\hline
		\textbf{Date} & \textbf{Holiday} \\
		\hline
		2021-01-01 &Nyårsdagen \\
		2021-01-06 &Trettondedag jul \\
		2021-04-02 &Långfredagen \\
		2021-04-04 &Påskdagen, Söndag \\
		2021-04-05 &Annandag påsk \\
		2021-05-01 &Första maj \\
		2021-05-13 &Kristi himmelsfärdsdag \\
		2021-05-23 &Pingstdagen, Söndag \\
		2021-06-06 &Sveriges nationaldag, Söndag \\
		2021-06-25 &Midsommarafton \\
		2021-06-26 &Midsommardagen \\
		2021-11-06 &Alla helgons dag \\
		2021-12-24 &Julafton \\
		2021-12-25 &Juldagen \\
		2021-12-26 &Annandag jul, Söndag \\
		2021-12-31 &Nyårsafton \\
		\hline
	\end{tabular}
	\caption{Results for Swedish holidays in 2021 from \texttt{python-holidays}}
	\label{table:sv_holidays}
\end{table}

It is possible also to include other parameters to extend the holiday to the neighbouring days. For more details, the reader can further investigate in \cite{fb_prophet}.

\subsection{Model fitting}

The seasonality and holiday components are then merged in a matrix $\bm{X}$ and the changepoints $\bm{a}(t)$, in a matrix $\bm{A}$. Then the model is expressed using Stan\cite{carpenter2017stan} as is shown in figure \ref{alg:stan}.

\begin{figure}[H]
\begin{lstlisting}
// Priors
k ~ normal(0, 5);
m ~ normal(0, 5);
epsilon ~ normal(0, 0.5);
delta ~ double_exponential(0, tau);
beta ~ normal(0, sigma);

// Logistic likelihood
y ~ normal(C ./ (1 + exp(-(k + A * delta) .* (t - (m + A * gamma)))) + X * beta, epsilon);

// Linear likelihood
y ~ normal((k + A * delta) .* t + (m + A * gamma) + X * beta, sigma);
\end{lstlisting}
\caption{Prophet Stan model.}
\label{alg:stan}
\end{figure}

Where \texttt{tau} and \texttt{sigma} are used to control the regularization levels.

Another handy feature that allows using \acp{gam} is to plot each component on its own, which allows the analyst to spot abnormalities when debugging a model. These plots will be shown and explained in the following sections.

\pagebreak
\subsection{Experiments}

The average \ac{psu} load signal shown in Figure \ref{fig:forecast_experiment_signal} has been chosen to run the following experiments. It is not a particularly easy signal since it contains some severe outliers in the first days that turned the power to almost zero, and at the end, the average power consumption seems to decrease.

It is necessary to notice that in every forecasting model, the longer the prediction horizon is, the more uncertainty and thus the lower performance. For this experiment, it has been decided to leave the last 10\% of data for testing purposes.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.6\linewidth]{forecast_experiment_signal}
	\caption{Signal used for the forecasting experiments}
	\label{fig:forecast_experiment_signal}
\end{figure}

In the following sections it is going to be shown first a pure univariate approach and later it will be added more exogenous regressors to improve the predictions. The models will be evaluated by using the following scores.

\subsubsection*{Coefficient of determination}

The coefficient of determination $R^2$ is a measure of how many variance of $\hat{y}$ is explained by the variance in $y$ in a linear regression context. It is compared against the considered \emph{worst possible linear approximation} which corresponds to the samples mean.

It is said that $R^2$ has values $[0,1]$, where $0$ corresponds to the worst linear model and 1 a perfect approximation.

Althought the current application is not a linear regression case, $R^2$ is still considered a valuable score to compare predictor performances, nonetheless, as the linearity assumption is not met, its values would reside in $(-\infty, 1]$ where 0 still being the mean value, but it is no longer considered the worst possible fit.

\begin{equation}\label{eq:rsq}
R^2 = 1 - \frac{\sum_{i}{(y_i - \hat{y}_i)^2}}{\sum_{i}{(y_i - \bar{y}_i)^2}}
\end{equation}

\subsubsection*{Mean absolute error}

It measures, in absolute terms, the deviations from the true values. \ac{mae} is preferred, given its interpretability, over \ac{rmse} \cite{MAE}.

\begin{equation}\label{eq:mae}
\text{MAE}	= \frac{1}{N} \sum_{i=1}^{N}{ \left| y_i - \hat{y}_i \right| }
\end{equation}

\subsubsection*{Mean out-of-bounds error}

As result of the fitting process, prophet's output is comprised by the mean prediction $\hat{y}_t$ and also its lower and upper boundaries for a given confidence value when constructing the predictor object which defaults at $80\%$

Let then the \ac{obe} be the out-of-the-confidence-bands error defined by (\ref{eq:obe}) where $\Delta \hat{y}_t$ is the computed confidence interval half-magnitude for the time $t$. Then the \ac{mobe} can be obtained by taking the mean of these values to obtain an overall performance score as in (\ref{eq:mobe}).

\begin{align}
\text{OBE}_i &= \bm{\min} \Big\{ \big| y_i - (\hat{y}_i - \Delta\hat{y}_i) \big| , \big	| y_i - (\hat{y}_i + \Delta\hat{y}_i) \big| \Big\} \label{eq:obe} \\
\text{MOBE} &= \frac{1}{N} \sum_{i=1}^{N}{ \text{OBE}_i }\label{eq:mobe}
\end{align}

\subsection{Univariate model implementation}

In the figure \ref{fig:prophet_uni_split} is shown the complete results of training and testing for a univariate prophet model.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.6\linewidth]{prophet_uni_split}
	\caption{Training, test and predictions from univariate Prophet}
	\label{fig:prophet_uni_split}
\end{figure}

\subsubsection*{Training fitting}

In training is obtained a very poor $R^2$ value, which from the plot in figure \ref{fig:prophet_uni_fitting} can be expected, since $\hat{y}$ looks like mostly an average of the seasonal components. Nonetheless, if it is considered that predicting an interval is acceptable, then the \ac{mobe} shows a very good result.

It can also be observed that the trend component did not overfit to the outliers.

\begin{table}[H]
	\centering
	\begin{tabular}{|c|c|}
		\hline
		\ac{mae} 	& 2.20 \\
		$R^2$ 		& 0.11 \\
		\ac{mobe} 	& 0.08 \\
		\hline
	\end{tabular}
	\caption{Univariate prophet training scores}
	\label{table:prophet_uni_train_scores}
\end{table}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.6\linewidth]{prophet_uni_fitting}
	\caption{Training fitting from univariate prophet}
	\label{fig:prophet_uni_fitting}
\end{figure}

\subsubsection*{Model learnt components}

In figure \ref{fig:prophet_uni_components} it is shown the learnt approximation of every component in the \ac{gam}. Although the trend did not overfit to the outliers, still learnt that there is a break point and changed the piecewise trend.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.7\linewidth]{prophet_uni_components}
	\caption{Learnt components from univariate prophet}
	\label{fig:prophet_uni_components}
\end{figure}

\pagebreak
\subsubsection*{Test predictions}

When it comes to the predictions performance, the $R^2$ shows even worse results. The \ac{mobe} is also high since the trend component was not able to foresee the decreasing in the middle of the test set as shown in figure \ref{fig:prophet_uni_preds}.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.6\linewidth]{prophet_uni_preds}
	\caption{Test predictions from univariate Prophet}
	\label{fig:prophet_uni_preds}
\end{figure}

\begin{table}[H]
	\centering
	\begin{tabular}{|c|c|}
		\hline
		\ac{mae} 	& 2.12 \\
		$R^2$ 		& -0.33 \\
		\ac{mobe} 	& 0.25 \\
		\hline
	\end{tabular}
	\caption{Univariate prophet prediction performance}
	\label{table:prophet_uni_test_scores}
\end{table}

\subsubsection*{Residual analysis}

The residuals in the training set still show somehow resembles the original data pattern. This could be interpreted that there is still patterns to learn from data, i.e., this model is not complete.

\begin{figure}[hptb]
	\begin{subfigure}{.47\textwidth}
		\includegraphics[width=\textwidth]{prophet_train_res_time}
		\caption{Residuals from train fit}
		\label{fig:prophet_train_res_time}
	\end{subfigure}%
	\hfill
	\begin{subfigure}{.47\textwidth}
		\includegraphics[width=\textwidth]{prophet_test_res_time}
		\caption{Residuals from test predictions}
		\label{fig:prophet_test_res_time}
	\end{subfigure}
	\caption{Univariate prophet residuals}
	\label{fig:prophet_res_time}
\end{figure}

Furthermore, if the joint distributions for $y$ and $\hat{y}$ are plotted, it can be seen how the outlier in the training set was not learnt and how the approximation becomes erratic in the test set.

\begin{figure}[hptb]
	\begin{subfigure}{.47\textwidth}
		\includegraphics[width=\textwidth]{prophet_train_res_joint}
		\caption{Train}
		\label{fig:prophet_train_res_joint}
	\end{subfigure}%
	\hfill
	\begin{subfigure}{.47\textwidth}
		\includegraphics[width=\textwidth]{prophet_test_res_joint}
		\caption{Test}
		\label{fig:prophet_test_res_joint}
	\end{subfigure}
	\caption{Univariate prophet joint distributions of $y$ and $\hat{y}$}
	\label{fig:prophet_res_joint}
\end{figure}

\subsection{Implementation using exogenous regressors}

To fully exploit the flexibility of \acp{gam}, prophet's API allows defining custom regressors, which in the current work will be called exogenous variables as a resemblance of SARIMAX models. These variables are the ones explained in chapter \ref{cha:data_analysis}.

\subsubsection*{Model learnt components}

The model components now show the additive factor of all the exogenous regressors, which seems to contain the information not learnt by the univariate training residuals.

\begin{table}[H]
	\centering
	\begin{tabular}{|c|c|}
		\hline
		\ac{mae} 	& 0.55 \\
		$R^2$ 		& 0.83 \\
		\ac{mobe} 	& 0.10 \\
		\hline
	\end{tabular}
	\caption{Multivariate prophet training scores}
	\label{table:prophet_multi_train_scores}
\end{table}

The $R^2$ score has considerably improved. Nonetheless, the other scores have become worse. The interpretation of this decline could be that the confidence band is now narrower than the univariate model. Thus, being out of the confidence band is now likely than before.

\begin{figure}[H]
	\centering
	\includegraphics[width=0.7\linewidth]{prophet_multi_components}
	\caption{Learnt components from multivariate prophet}
	\label{fig:prophet_multi_components}
\end{figure}

\subsubsection*{Test predictions}

The results of the predictions have notoriously improved in this case. The $R^2$ value is close to 0.9, which for such a complex signal shows how powerful is the model.

The \ac{mobe} has even decreased in the test set, this could be due to training outliers that still being difficult to approximate without overfitting the trend component.

\begin{table}[H]
	\centering
	\begin{tabular}{|c|c|}
		\hline
		\ac{mae} 	& 0.52 \\
		$R^2$ 		& 0.88 \\
		\ac{mobe} 	& 0.05 \\
		\hline
	\end{tabular}
	\caption{Multivariate prophet prediction performance}
	\label{table:prophet_multi_test_scores}
\end{table}

\begin{figure}[H]
	\centering
	\includegraphics[width=0.6\linewidth]{prophet_multi_preds}
	\caption{Test predictions from multivariate prophet}
	\label{fig:prophet_multi_preds}
\end{figure}

\subsubsection*{Residual analysis}

The residuals now show that the patterns have been better learnt than in the univariate case.

\begin{figure}[hptb]
	\begin{subfigure}{.47\textwidth}
		\includegraphics[width=\textwidth]{prophet_train_res_time_multi}
		\caption{Train}
		\label{fig:prophet_train_res_time_multi}
	\end{subfigure}%
	\hfill
	\begin{subfigure}{.47\textwidth}
		\includegraphics[width=\textwidth]{prophet_test_res_time_multi}
		\caption{Test}
		\label{fig:prophet_test_res_time_multi}
	\end{subfigure}
	\caption{Multivariate prophet residuals}
	\label{fig:prophet_res_time_multi}
\end{figure}

The joint distributions for $y$ and $\hat{y}$ now shows that the model is better approximating the signal.

\begin{figure}[hptb]
	\begin{subfigure}{.47\textwidth}
		\includegraphics[width=\textwidth]{prophet_train_res_joint_multi}
		\caption{Train}
		\label{fig:prophet_train_res_joint_multi}
	\end{subfigure}%
	\hfill
	\begin{subfigure}{.47\textwidth}
		\includegraphics[width=\textwidth]{prophet_test_res_joint_multi}
		\caption{Test}
		\label{fig:prophet_test_res_joint_multi}
	\end{subfigure}
	\caption{Multivariate prophet joint distributions of $y$ and $\hat{y}$}
	\label{fig:prophet_res_joint_multi}
\end{figure}

%
%\pagebreak
%\section{Boosted random forests}
%
%
%\noindent\fbox{
%	\parbox{\textwidth}
%	{
%		Draft acknowledgement:\\
%
%		The following models are still work in progress and last implementation has a bug to be fixed.
%
%		If time is not enough we may decide to drop this part for final report
%	}
%}
%
%\subsection{Model overview}
%
%\noindent\fbox{
%	\parbox{\textwidth}
%	{
%		Draft acknowledgement:\\
%		XGBoost regressor theory
%	}
%}
%
%\subsection{Regression approach}
%
%\begin{equation}
%	\label{xgb_reg}
%	y_{t+1} = \sum_{t=1}^{N}{\beta_t \cdot x_{t_i}}
%\end{equation}
%
%\begin{figure}[H]
%	\centering
%	\includegraphics[width=0.6\linewidth]{xgb_reg_overall}
%	\caption{XGBoost overall performance}
%	\label{fig:xgb_reg_overall}
%\end{figure}
%
%\subsubsection*{Training fitting}
%
%\begin{figure}[H]
%	\centering
%	\includegraphics[width=0.6\linewidth]{xgb_reg_train}
%	\caption{XGBoost train performance}
%	\label{fig:xgb_reg_train}
%\end{figure}
%
%\begin{table}[H]
%	\centering
%	\begin{tabular}{|c|c|}
%		\hline
%		\ac{mae} 	& 0.04 \\
%		$R^2$ 		& 1.00 \\
%		\hline
%	\end{tabular}
%	\caption{XGBoost regressor train scores}
%	\label{table:xgb_reg_train_scores}
%\end{table}
%
%\subsubsection*{Test predictions}
%
%\begin{figure}[H]
%	\centering
%	\includegraphics[width=0.6\linewidth]{xgb_reg_test}
%	\caption{XGBoost test performance}
%	\label{fig:xgb_reg_test}
%\end{figure}
%
%\begin{table}[H]
%	\centering
%	\begin{tabular}{|c|c|}
%		\hline
%		\ac{mae} 	& 0.44 \\
%		$R^2$ 		& 0.93 \\
%		\hline
%	\end{tabular}
%	\caption{XGBoost regressor prediction performance}
%	\label{table:xgb_reg_test_scores}
%\end{table}
%
%
%\subsubsection*{Residual analysis}
%
%\begin{figure}[hptb]
%	\begin{subfigure}{.47\textwidth}
%		\includegraphics[width=\textwidth]{xgb_reg_res_train}
%		\caption{Train}
%		\label{fig:xgb_reg_res_train}
%	\end{subfigure}%
%	\hfill
%	\begin{subfigure}{.47\textwidth}
%		\includegraphics[width=\textwidth]{xgb_reg_res_test}
%		\caption{Test}
%		\label{fig:xgb_reg_res_test}
%	\end{subfigure}
%	\caption{XGBoost regression residuals}
%	\label{fig:xgb_reg_res}
%\end{figure}
%
%\begin{figure}[hptb]
%	\begin{subfigure}{.47\textwidth}
%		\includegraphics[width=\textwidth]{xgb_reg_joint_train}
%		\caption{Train}
%		\label{fig:xgb_reg_joint_train}
%	\end{subfigure}%
%	\hfill
%	\begin{subfigure}{.47\textwidth}
%		\includegraphics[width=\textwidth]{xgb_reg_joint_test}
%		\caption{Test}
%		\label{fig:xgb_reg_joint_test}
%	\end{subfigure}
%	\caption{XGBoost regression joint distributions of $y$ and $\hat{y}$}
%	\label{fig:xgb_reg_joint}
%\end{figure}
%
%\pagebreak
%\subsection{Including autoregressive components}
%
%Nonetheless, the function \ref{xgb_reg}, despite being trained to predict the next $y$ sample, still having only a regressive nature, i.e., no autoregressive components have been used to also learn from previous outputs. Thus, the function can be extended to:
%
%\begin{equation}
%	\label{xgb_ar}
%	y_{t} = \sum_{i=1}^{N}{\beta_i \cdot x_{{t-1}_i}} + \sum_{l=1}^{L}{\phi_l \cdot y_{t-l}}
%\end{equation}
%
%Where $N$ is the amount of features and  $L$ corresponds to the amount of lags under consideration.
%
%\subsubsection*{Provisional results}
%
%\noindent\fbox{
%	\parbox{\textwidth}
%	{
%		Draft acknowledgement:\\
%		This section has a bug, which fixing will be resumed as soon as the report draft is delivered
%	}
%}
%
%\begin{figure}[H]
%	\centering
%	\includegraphics[width=0.6\linewidth]{xgb_ar_test}
%	\caption{XGBoost with autoregressive components test performance}
%	\label{fig:xgb_ar_test}
%\end{figure}
%
%\begin{figure}[hptb]
%	\begin{subfigure}{.47\textwidth}
%		\includegraphics[width=\textwidth]{xgb_ar_joint_train}
%		\caption{Train}
%		\label{fig:xgb_ar_joint_train}
%	\end{subfigure}%
%	\hfill
%	\begin{subfigure}{.47\textwidth}
%		\includegraphics[width=\textwidth]{xgb_ar_joint_test}
%		\caption{Test}
%		\label{fig:xgb_ar_joint_test}
%	\end{subfigure}
%	\caption{XGBoost with autoregressive components joint distributions of $y$ and $\hat{y}$}
%	\label{fig:xgb_ar_joint}
%\end{figure}


\documentclass[../main.tex]{subfiles}

\begin{document}
        \section{Curvature}
        We wish to construct a mathematical way to encode the ``amount" which a given curve curves, so that we may distinguish straighter curves from more tightly winding curves.

        The velocity vector \(\vec{v}\) of a vector valued function \(\vec{r}\) is tangent to the curve which \(\vec{r}\) describes, so this notion of curvature must be related to the rate of change of \(\vec{v}\), as the curvature is highest when the velocity vector changes rapidly. One issue is that parameterizations are generally not unique, but our definition of curvature should be independent of any particular parameterization, and thus an intrinsic property of the curve. To do this, we normalize \(\vec{v}\) to unit length, so that the speed with which we traverse the curve is irrelevant to the curvature.
        \begin{theorem}{}{}
                Consider a smooth curve parameterized by \(\vec{r}(t)\). Prove that \(|\vec{r}|=a\) is constant if and only if \(\vec{v}\perp\vec{r}\) for all \(t\).
                \tcblower
                \(\vec{r}\cdot\vec{r}=|\vec{r}|^2=a^2\), which is constant if and only if \(\frac{d}{dt}\vec{r}\cdot\vec{r}=0\), but
                \[
                \frac{d}{dt}\vec{r}\cdot\vec{r}=\vec{r}'\cdot\vec{r}+\vec{r}\cdot\vec{r}'=\vec{v}\cdot\vec{r}+\vec{r}\cdot\vec{v}=2\vec{v}\cdot\vec{r}
                \]
                so \(|\vec{r}(t)|=a\) is constant if and only if \(\vec{v}\cdot\vec{r}=0\), and thus \(\vec{v}\perp\vec{r}\).
        \end{theorem}
        \begin{definition}{Unit Tangent Vector}{}
                Assuming that \(\vec{r}(t)\) is smooth, so that \(\vec{r}'(t)\neq\vec{0}\), we define the \emph{unit tangent vector}
                \[
                \vec{T}(t)=\frac{\vec{r}'(t)}{|\vec{r}'(t)|}
                \]
                which is always tangent to \(\vec{r}(t)\), and has unit length \(|\vec{T}|=1\), so the unit tangent vector does not depend on parameterization, except for orientation.

                The unit tangent vector may also be defined in terms of the arc length parameterization. By noting that because the arc length parameterization has constant unit speed, we have
                \[
                \vec{T}(s)=\frac{\frac{d\vec{r}}{ds}}{\left|\frac{d\vec{r}}{ds}\right|}=\frac{d\vec{r}}{ds}
                \]
        \end{definition}
        \begin{definition}{Curvature}{}
                We define the \emph{curvature} \(\kappa\) of a curve in terms of the unit tangest vector as
                \[
                \kappa = \left| \frac{d\vec{T}}{ds}\right|
                \]
                using the arc length parameterization so that the curvature is independent of any particular parameterization.
        \end{definition}
        One difficulty with this definition is that it requires us to find the arc length parameterization to solve for the curvature, but finding the arc length parameterization of even simple curves may be difficult or impossible. Therefore, to compute \(\kappa\), we use the chain rule, noting that \(\frac{ds}{dt}=|\vec{r}'(t)|\), so we may compute the curvature of a curve using
        \[
        \kappa = \left|\frac{d\vec{T}}{dt}\frac{dt}{ds}\right|=\frac{|\vec{T}'(t)|}{|\vec{r}'(t)|}
        \]
        \begin{example}{}{}
                Consider a straight line parameterized by \(\vec{r}(t)=\vec{r}_0+\vec{v}t\). Show that this curve has \(\kappa=0\).
                \tcblower
                First, we find \(\vec{r}'(t)=\vec{v}\), so \(|\vec{r}'(t)|=|\vec{v}|\), which gives us
                \[
                \vec{T}=\frac{\vec{v}}{|\vec{v}|}
                \]
                but since \(\vec{v}\) is constant, we have
                \[
                \vec{T}'=0
                \]
                and consequently \(\kappa=0\).
        \end{example}
        \begin{example}{}{}
                Consider a circle parameterized by \(\vec{r}(t)=a\cos(t)\i+a\sin{t}\j\). Find its curvature.
                \tcblower
                First, we find
                \[
                \vec{r}'(t)=-a\sin{t}\i+a\cos{t}\j
                \]
                so
                \[
                |\vec{r}'(t)|=\sqrt{a^2\sin^2{t}+a^2\cos^2{t}}=\sqrt{a^2}=a
                \]
                which gives us
                \[
                \vec{T}(t)=\frac{\vec{r}'(t)}{a}=-\sin{t}\i+\cos{t}\j
                \]
                and
                \[
                \vec{T}'(t)=-\cos{t}\i-\sin{t}\j
                \]
                which means that \(|\vec{T}'(t)|=1\), so
                \[
                \kappa = \frac{|\vec{T}'(t)|}{|\vec{r}'(t)|} = \frac{1}{a}
                \]
        \end{example}
        \begin{example}{}{}
                Consider a helix parameterized by \(\vec{r}(t)=a\cos{t}\i+a\sin{t}\j+bt\k\). Find its curvature.
                \tcblower
                First, we find
                \[
                \vec{r}'(t)=-a\sin{t}\i+a\cos{t}\j+b\k
                \]
                so
                \[
                |\vec{r}'(t)|=\sqrt{a^2+b^2}
                \]
                which gives us
                \[
                \vec{T}(t)=\frac{1}{\sqrt{a^2+b^2}}\vec{r}'(t)
                \]
                and
                \[
                |\vec{T}'(t)|=\frac{1}{\sqrt{a^2+b^2}}\sqrt{a^2\cos^2{t}+a^2\sin^2{t}} = \frac{a}{\sqrt{a^2+b^2}}
                \]
                so the curvature is
                \[
                \kappa = \frac{\frac{a}{\sqrt{a^2+b^2}}}{\sqrt{a^2+b^2}} = \frac{a}{a^2+b^2}
                \]
                Note that in the limit as \(b\to\infty\) the curvature approaches that of a straight line, and in the limit as \(b\to 0\), the curvature approaches that of a circle.
        \end{example}{}{}
        This definition of curvature always works, but there is a formula for curvature which is often simpler to apply, namely
        \[
        \kappa = \frac{|\vec{r}'\times\vec{r}''|}{|\vec{r}'|^3}
        \]
        the proof of which is elided.
        \begin{example}{}{}
                Consider the curve parameterized by \(\vec{r}(t)=2t\i+t^2\j+\frac{t^3}{3}\j\). Find its curvature.
                \tcblower
                First, we find \(\vec{r}'(t)=2\i+2t\j+t^2\k\), so
                \[
                |\vec{r}'(t)|=\sqrt{4+4t^2+t^4}=\sqrt{(2t+t^2)^2}=2t+t^2
                \]
                We may then calculate \(\vec{r}''(t)=2\j+2t\k\), and \(\vec{r}'\times\vec{r}''=2(t^2\i-2t\j+2\k)\), which gives us
                \[
                |\vec{r}'\times\vec{r}''|=2\sqrt{t^4+4t^2+4}=2(2+t^2)
                \]
                Finally, we may calculate
                \[
                \kappa = \frac{2(2+t^2)}{(2+t^2)^3} = \frac{2}{(2+t^2)^2}
                \]
        \end{example}
\end{document}


\chapter{Basic CNN}

This model is a specialized kind of neural network for processing data that has a known, grid-like topology, or more generally, data $X^i \in \mathbb{R}^{n_1\times n_2\cdots\times n_d}$ is a order $d$ tensor. A naive way to deal with those kind of data is to reshape this tensor to be a vector( order 1 tensor), and then use the traditional deep feedforward neural network.

However, for some practical problem, this method means that the input dimension is very big that is $m = n_1\times\cdots \times n_d$. Another important reason is that this will broke the structure of this data, especially for AI task.

\section{Convolution in mathematics}
\subsection{Continuous case}
Generally speaking, convolution is a special kind of bilinear transform for multidimensional function. For example:
\begin{equation}
\ast: L^2(\mathbb{R}^n) \times L^2(\mathbb{R}^n) \to L^1(\mathbb{R}^n).
\end{equation}

So if, we fixed one function, a convolution operator is just a linear transform form one function space to another. For example, if $g \in C_0^{\infty}(\mathbb{R}^n)$, we have:
\begin{equation}
\ast g: L^2(\mathbb{R}^n) \to L^2(\mathbb{R}^n),
\end{equation}
with
\begin{equation}
(f\ast g)(x) := ((\ast g)f )(x) = \int_{\mathbb{R}^n} f(y)g(x-y) dy.
\end{equation}

\subsection{Discrete case}
When talks about discrete case, general function becomes tensor(Although, the real tensor definition is not like this):
Functions $\to$ Tensors,
\begin{align}
(f\ast g)(x) &= \int_{\bm{y}} f(x_1 - y_1,\cdots,x_d - y_d)g(y_1,\cdots,y_d) \\
\xrightarrow[Uniform~Grid]{Sampling} (F\ast G)_{i_1, \cdots, i_d} &= \sum_j F_{i_1 - j_1,\cdots,i_d - j_d}G_{j_1, \cdots, j_d}.
\end{align}

We can have the next definition for two general order $d$ tensor(comes from the $d$-dimensional function):
\begin{equation}
(T\ast M)(i_1, i_2, \cdots ,i_d) = \sum_{j_i, j_2,\cdots,j_d \in \mathbb{Z}} T(j_1, j_2,\cdots,j_d)M(i_1 - j_1, i_2 - j_2, \cdots, i_d - j_d).
\end{equation}
Almost book and notes about CNN defined discrete convolution like this, but here is a problem about the summation  index, this may only be well defined when the index for every dimension can be infinite. So in general there should be some rules for finite cases, for example:
\begin{itemize}
\item Padding $M$ with zeros
\item Repeat $M$ to make it a periodic infinite one in every dimension
\end{itemize}
Now we consider the finite case, which means $T, M \in \mathbb{R}^{n_1 \times n_2 \times \cdots n_d}$. By the definition(no matter what rules we use), $\ast M$ should be a order $2d$ tensor with some special structure because that:
\begin{equation}
\ast M \in L(\mathbb{R}^{n_1 \times n_2 \times \cdots n_d}, \mathbb{R}^{n_1 \times n_2 \times \cdots n_d}).
\end{equation}
And we call $M$ is the kernel for this convolution mapping $\ast M$.

Here a good example for the special structure for $\ast M$ is when we consider the convolution between two order $1$ tensors. We can have two vectors $\bm{x}$ and $\bm y$ then $\ast \bm y$ is exact the a special Toeplitz matrix i.e a circulant matrix.
\begin{equation}
(\ast \bm{y})_{i,j} = \bm{y}(j-i+1).
\end{equation}
We will see that, this special structure plays a very important role in deep learning, I think this special structure is a special case for general called ``Parameter Sharing".

\section{Convolution introduction in CNN (without multichannel)}
\subsection{General Convolution in CNN(with small size change)}
There is a kind of special linear map from some general finite dimension tensor space to another is called ``convolution".  Generally, we have a tensor like $X \in \mathbb{R}^{n \times m}$ and a small kernel such as $K \in \mathbb{R}^{k\times k}$, then we have:
\begin{definition}[Original Convolution with stride 1 for CNN]
\begin{equation}\label{equ:conv}
(X\ast K)_{i,j} = \sum_{s, t = 1}^k X_{i-1 + s, j-1 + t} K_{s,t}.
\end{equation}
It's easy to see that $(X\ast K) \in \mathbb{R}^{(n-k + 1) \times (m-k + 1)}$.
\end{definition}
If we use stride $p$, we have:
\begin{definition}[Original Convolution with stride $p$ for CNN]
\begin{equation}\label{equ:convstride}
(X\ast_{p} K)_{i,j} = \sum_{s, t = 1}^k X_{(i-1)p + s, (j-1)p + t} K_{s,t}.
\end{equation}
So, we have
$
(X\ast_p K) \in \mathbb{R}^{((n-k)/p + 1) \times ((m-k)/p+1)}.
$
\end{definition}
This stride properties in some application are often used as pooling(subsampling, coarsening).

Here we can define a special pooling operator as $S^p$ which likes the $C/F$ split for choosing coarse point:
\begin{equation}
S^p(X)_{i,j} = X_{(i-1)p + 1, (j-1)p + 1},
\end{equation}
then we have:
\begin{equation}\label{equ:stride}
X \ast_p K = S^p(X\ast K),
\end{equation}
with the $\ast$ and $\ast_p$ defined by \ref{equ:conv} and \ref{equ:convstride}
\begin{proof}
\begin{align}
S^p(X \ast K)_{i,j} &= (X \ast K)_{(i-1)p+1, (j-1)p + 1}  \\
&= \sum_{s,t = 1}^k X_{(i-1)p + 1 -1 +s, (j-1)p +1 -1 +t}K_{s,t}  \\
&= (X \ast_p K)_{i,j}.
\end{align}
\end{proof}

\subsection{Convolution with Padding in CNN(without changing size)}
For many case kernel size is some small odd numbers such as 1, 3, 5... So we ca have those next definition for convolution:
\begin{definition}[Convolution with Padding]

We write $K\in \mathbb R^{2k+1, 2k+1}$, we then write
\begin{equation}\label{ConvPadding}
(X \ast K)_{i,j} :=\sum_{s, t = -k}^k P^k(X)_{i + s, j + t} K_{s,t},
\end{equation}
with $\rm{P}^k$ means Padding.
\end{definition}
Here we see that, to make this definition well-defined, we have $X\ast K \in \mathbb{R}^{n-2k, m-2k}$, because if the index of $X$ start from $1$ to $n(m)$, then $X \ast K$ from $k+1$ to $n(m) - k$. To make the convolution don't change size, we can use Padding strategy like in image process.
%In fact, if the resolution is very high, which means the measure of the boundary of the image is close to 0, so in traditional image process, this strategy hasn't been studied very well. There are basic three strategy:
\begin{description}
\item[Zero Padding] The simplest and most common used strategy is zero padding with:
\begin{equation}
\rm{P}^k_0: \mathbb{R}^{n \times m} \to \mathbb{R}^{n+2k \times m+2k}
\end{equation}
with
\begin{equation}
\rm{P}_0^k(X)_{i,j} = \begin{cases}
0 &i(j)=-k+1:0 ~~\text{and}~~ i(j) = n(m)+1:n(m)+k \\
X_{i,j} &i(j) = 1:n(m)
\end{cases}
\end{equation}
\item[Reflection Padding] This strategy is defined by:
\begin{equation}
\rm{P}^k_r: \mathbb{R}^{n \times m} \to \mathbb{R}^{n+2k \times m+2k}
\end{equation}
with
\begin{equation}
\rm{P}_r^k(X)_{i,j} = \begin{cases}
X_{1-i, 1-j} &i(j)=-k+1:0 ~~\text{and}~~ i(j) = n(m)+1:n(m)+k \\
X_{i,j} &i(j) = 1:n(m) \\
0 &\text{others}
\end{cases}
\end{equation}
\item[Shift Padding] This strategy is defined by:
\begin{equation}
\rm{P}^k_s: \mathbb{R}^{n \times m} \to \mathbb{R}^{n+2k \times m+2k}
\end{equation}
with
\begin{equation}
\rm{P}_s^k(X)_{i,j} = \begin{cases}
X_{k-i, k-j} &i(j)=-k+1:0 ~~\text{and}~~ i(j) = n(m)+1:n(m)+k \\
X_{i,j} &i(j) = 1:n(m) \\
0 &\text{others}
\end{cases}
\end{equation}
\end{description}

So we can have:
\begin{equation}
\rm{dim}(X\hat{\ast}K) = \rm{dim}(\rm{P}^k(X) \ast K) = \rm{dim}(X),
\end{equation}
and the $i,j$ index is consistent for both side of the above equation where $\rm{P}^k = \rm{P}_0^k, \rm{P}_{r}^k$ or $\rm{P}_s^k$.

\begin{remark}We note all the later convolution as the \eqref{ConvPadding} without special statements and use $\ast$ stands for $\hat{\ast}$.

\end{remark}

\section{Special Structure for Convolution in CNN}
Now let us forget about the general convolution in mathematics, we go back to the method for using deep learning to deal with tensor data. Recall the general method in fully connected deep feedforward neural network, all process can be decompose into two phase: linear transform(in fact, affine map) from vector space to another vector space,  and nonlinear transform separately for every component for the output of linear transform.

So, now we consider we have a tensor like $T \in \mathbb{R}^{n_1\times n_2\cdots \times n_d}$, and we want to use some linear transform $W$ such that $W(T) \in  \mathbb{R}^{n'_1\times n'_2\cdots \times n'_d }$, this means that
\begin{equation}
W \in  L(\mathbb{R}^{n_1\times n_2\cdots \times n_d},  \mathbb{R}^{n'_1\times n'_2\cdots \times n'_{d}}) ,
\end{equation}
which means that $W$ is a order $2d$ tensor. The size for $W$ is too big!

\subsubsection{Parameter Sharing and Convolution}
Let us just use the order 1 tensor as a example to show the power for parameter sharing. For vector case, i.e $T \in \mathbb{R}^{n}$, and we want the output is in $\mathbb{R}^m$, so we have $W \in R^{m\times n}$.  So we can think about the next parameter sharing strategy:
\begin{definition}[Parameter Sharing]
Every row of $W$ is same without only some shifting operator.
\end{definition}
This strategy lead to that the convolution operator is a very special kind of  result from parameter sharing.  This means that $W$ can be defined by just a $\mathbb{R}^n$ vector.(This might be some kinds of ``low dimension" structure?) However, how to choose a suitable vector is still a big problem. By the way, for general tensor, this means that $W$ also have the same size for $T$ which may also be a little big.

\subsubsection{Sparsity for Convolution in CNN}
Thanks for the filter theory for mathematical image process, we can just choose some sparse(local supported) vector to do some meaningful operation. Such as   $(-1, 2, -1)$, means the discrete Laplacian operator, but a more strict expression should be $\bm x \in \mathbb{R}^n$, and then $W \in \mathbb{R}^{n-2 \times n}$ with
\begin{equation}
W(j,i) = w(j - i + 1)  \quad  W = \begin{pmatrix}
-1 & 2 & -1 & 0 & 0&\cdots & 0 \\
0& -1 & 2 & -1 & 0 & \cdots & 0 \\
\vdots &  & & \cdots & & & \vdots \\
0& \cdots & & & -1 & 2& -1
\end{pmatrix},
\end{equation}
and
\begin{align}
\bm x\ast w = W \bm x.
\end{align}
So we may have the next with sparse parameter strategy:
\begin{definition}[Sparsity]
Every row of $W(:,i)$ is locally supported around $i$ or $i \pm 1$.
\end{definition}

At last, we can take this kind of convolution in CNN as a results of ``parameter sharing" and ``sparsity".

%\subsection{Convolution in CNN}
%\subsubsection{Definition by ``inner product" type operator}
%For the example above, we found that we only need a vector $ k = (-1,2,-1)$ to represent this process like:
%\begin{equation}
%(W \bm x)(j) = x(j:j+2) \cdot k \quad \forall j = 1:n-2.
%\end{equation}
%So this inspires us to have the next definition, if $T \in \mathbb{R}^{n_1 \times n_2 \cdots n_d}$ and $K \in \mathbb{K}^{m_1 \times m_2\cdots m_d}$ with $ m_i \le n_i $.
%we can define
%\begin{equation}
%\hat{m}_i = \begin{cases}
%\frac{m_i - 1}{2} \quad &\text{if $m_i$ is odd} \\
%\frac{m_i}{2} \quad &\text{if $m_i$ is even}.
%\end{cases}
%\end{equation}
%So we define
%\begin{equation}
%(T \ast K)(i_1, i_2, \cdots, i_d) = T( i_1 : i_1 + (m_i-1), \cdots, i_d:i_d + (m_d -1)) \odot K \quad i_j \le n_j - m_j + 1,
%\end{equation}
%where $\odot$ means sum of all index for two tensor with the same size, i.e
%\begin{equation}
%T_1 \odot T_2 = \sum_{i_1, \cdots, i_d} T_1(i_1, \cdot, i_d) T_2(i_i,\cdots, i_d).
%\end{equation}
%This means that, for any tensor we can use another tensor with the same order and smaller size to be a kernel, and do convolution with them. So this is no longer commutative for this kind of ``convolution".

\subsection{Convolution with multichannel}
\subsubsection{Channel}
Before we start to explain every layers for general CNN model, we want to give some remarks about ``Channel".   As we know above, if we apply one certain kernel to do convolution with the original tensor, we can get some certain feature. The following are some examples of $3\times 3$ kernels.
%\begin{figure}[!htb]
	%\center{\includegraphics[width=13cm] {Kernels.png}}
	%\caption{Eight kernels makes eight channels with different type of features.}
	%\label{Kernels}
%\end{figure}
\begin{itemize}
\item A horizontal bar
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{horizontal.png}} *
\begin{pmatrix}
    0 & 0 & 0\\
    0 & 1 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{horizontal.png}}
$$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{horizontal.png}} *
\begin{pmatrix}
    1 & -1 & 0\\
    0 & 0 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{hx.png}}
$$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{horizontal.png}} *
\begin{pmatrix}
    1 & 0 & 0\\
    -1 & 0 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{hy.png}}
$$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{horizontal.png}} *
\begin{pmatrix}
    -1 & 0 & 0\\
    0 & 1 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{hxy.png}}
$$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{horizontal.png}} *
\begin{pmatrix}
    0 & -1 & 0\\
    -1 & 4 & -1\\
    0 & -1 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{hlaplace.png}}
$$

\item A vertical bar
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{vertical.png}} *
\begin{pmatrix}
    0 & 0 & 0\\
    0 & 1 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{vertical.png}}
$$

$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{vertical.png}} *
\begin{pmatrix}
    1 & -1 & 0\\
    0 & 0 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{vx.png}}
$$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{vertical.png}} *
\begin{pmatrix}
    1 & 0 & 0\\
    -1 & 0 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{vy.png}}
$$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{vertical.png}} *
\begin{pmatrix}
    -1 & 0 & 0\\
    0 & 1 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{vxy.png}}
$$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{vertical.png}} *
\begin{pmatrix}
    0 & -1 & 0\\
    -1 & 4 & -1\\
    0 & -1 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{vlaplace.png}}
$$

\item A slash bar
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{slash.png}} *
\begin{pmatrix}
    0 & 0 & 0\\
    0 & 1 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{slash.png}}
$$

$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{slash.png}} *
\begin{pmatrix}
    1 & -1 & 0\\
    0 & 0 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{sx.png}}
$$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{slash.png}} *
\begin{pmatrix}
    1 & 0 & 0\\
    -1 & 0 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{sy.png}}
$$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{slash.png}} *
\begin{pmatrix}
    -1 & 0 & 0\\
    0 & 1 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{sxy.png}}
$$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{slash.png}} *
\begin{pmatrix}
    0 & -1 & 0\\
    -1 & 4 & -1\\
    0 & -1 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{slaplace.png}}
$$

\item Identity kernel
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k0.jpg}} *
\begin{pmatrix}
    0 & 0 & 0\\
    0 & 1 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k0.jpg}}
$$
\item Kernel that taking $\frac{\partial}{\partial x}$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k0.jpg}} *
\begin{pmatrix}
    1 & -1 & 0\\
    0 & 0 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k1.jpg}}
$$
\item Kernel that taking $\frac{\partial}{\partial y}$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k0.jpg}} *
\begin{pmatrix}
    1 & 0 & 0\\
    -1 & 0 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k2.jpg}}
$$
\item Kernel that taking $\frac{\partial}{\partial x}+\frac{\partial}{\partial y}$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k0.jpg}} *
\begin{pmatrix}
    1 & 0 & 0\\
    0 & -1 & 0\\
    0 & 0 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k3.jpg}}
$$
\item Kernel that taking $-\Delta$
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k0.jpg}} *
\begin{pmatrix}
    0 & -1 & 0\\
    -1 & 4 & -1\\
    0 & -1 & 0
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k4.jpg}}
$$
\item Kernel that taking local average
$$
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k0.jpg}} *
\begin{pmatrix}
    1 & 1 & 1\\
    1 & 1 & 1\\
    1 & 1 & 1
\end{pmatrix} =
\raisebox{-.5\height}{\includegraphics[width=0.2\textwidth]{k5.jpg}}
$$
\end{itemize}

So if we want to do some AI task like classifier, we cannot only focus on just one feature, so we need many kernels to produce many features. In computer vision, they also call the several channels got from the convolution with different kernels as feature map.
Because we use more and more kernel after several layers, we need another dimension to denote those several channels, we call this dimension as ``channel dimension". We also have ``essential dimension" w.r.t ``channel dimension" which is the real tensor dimension for this date. For example, for a general colorized picture, is might be $T \in \mathbb{R}^{512\times 512 \times 3}$, here the essential dimension is the first two dimension, and the third one is the $RGB$ channel dimension to represent the colour for picture.

\subsubsection{Convolution for multichannel case}
Here in real data, we have some special data like colour image $X \in \mathbb{R}^{n \times m \times3}$. Here this image is a 2D graph, with $3$ means the $RGB$ channels. So, we cannot just think $X$ as a general $3$-order tensor.

We may use this as an example, $X \in \mathbb{R}^{n\times m \times c}$, as we mentioned before, we say that $n \times m$ is the essential dimension and $c$ is the channel dimension for $X$. If we want to do convolution for $X$, here we cannot just use $K \in \mathbb{R}^{2k+1 \times 2k+1}$ because of multichannel. A simple idea is to use also different $c$ kernels and collect them together as $K \in \mathbb{R}^{2k+1 \times 2k+1 \times c}$, and then we can do the general convolution for signal channel separately. This means that
\begin{equation}
(X \ast K)_{l} = X_{l}\ast K_{l}.
\end{equation}
But we need to recall that, even for $X \in \mathbb{R}^{n\times m \times c}$ it is in fact only stand for just one 2D image(with multi-channel). So if we talk about features in AI, $(X\ast K) \in \mathbb{R}^{n\times m \times c}$ should be reduced to the essential dimension, one simple way is just to add all the channels value together w.r.t essential dimension(this is equal to added with weights because we can reduce those DoF into the previous $c$ kernels.):
\begin{equation}
(X \hat{\ast} K) = \sum_{l=1}^{c} X_{l}\ast K_{l}.
\end{equation}
then we have
\begin{equation}
(X \hat{\ast} K) \in \mathbb{R}^{n \times m} \quad \text{with} \quad K \in \mathbb{R}^{2k+1 \times 2k+1 \times c}.
\end{equation}
So, for a given real data with $c$ channels like $X \in  \mathbb{R}^{n\times m \times c}$ if we use kernel $K \in \mathbb{R}^{2k+1 \times 2k+1 \times c}$ we can only get one signal channel output. To get multichannel form $X$, we just need to use $d$ kernels $K \in \mathbb{R}^{2k+1 \times 2k+1 \times c}$, or we say $K \in \mathbb{R}^{2k+1 \times 2k+1 \times c \times b}$, then we have:
\begin{equation}\label{6.6}
(X \tilde{\ast} K)_{g} =  X \hat{\ast} K_{g},
\end{equation}
with
\begin{equation}
(X \tilde{\ast} K) \in \mathbb{R}^{n \times m \times b} \quad \text{with} \quad K \in \mathbb{R}^{2k+1 \times 2k+1 \times c \times b}.
\end{equation}

In the next section, we set that all tensor is with
multichannel(signal channel is a special case, and we note it as $X
\in \mathbb{R}^{n \times m \times 1}$), and we just use $\ast$ no
longer $\tilde{\ast}$. All for all, the  above convolution for multi-channel tensor can be expressed as:
\begin{equation}\label{6.6}
(X \ast K)_{i,j,g} = \sum_{\bm{l}=1}^{c} \sum_{s, t = -k}^k X_{i + s, j + t,\bm{l}} K_{s,t,\bm{l},g},
\end{equation}
with
\begin{equation}
(X \ast K) \in \mathbb{R}^{n \times m \times b} \quad \text{and} \quad K \in \mathbb{R}^{2k+1 \times 2k+1 \times c \times b}.
\end{equation}

\section{Three basic layers in CNN}
A typical structure in a convolutional network consists of three layers. In the first layer, it performs several convolutions in parallel to produce a set of linear activations. In the second one, each linear activation is run through a nonlinear activation function, such as the rectified linear(ReLu) activation function. This layer is sometimes called the detector layer. In the third layer, we use a pooling function to modify the output of the layer further.
\subsubsection{Convolutional layers}
Here we set that, all the input for the convolutional layer is just one tensor, which means we need collect all channels as the channel dimension. So, for a general tensor with $\hat{c}$ channels $T \in \mathbb{R}^{n_1 \times n_2 \cdots n_d \times \hat{c}}$, we have $\tilde{c}$ convolutional kernels(filter) $K_i \in \mathbb{R}^{2k+1 \times 2k+1 \cdots 2k+1 \times \hat{c}}$ to map it to a $n_1 \times \cdots \times n_d $ tensor with $\tilde c$ channels(feature mapping).

Mathematically speaking:
\begin{equation}
\ast K_{i}: \mathbb{R}^{n_1 \times n_2 \cdots n_d \times \hat{c}} \xrightarrow{K_i} \mathbb{R}^{n_1 \times  \cdots n_d } \quad \forall i = 1:\tilde c,
\end{equation}
or like the next figure for the ``original convolution with stride 1 for CNN":
\begin{figure}[!htb]
	\center{\includegraphics[width=10cm] {Convolution_layer.png}}
	\caption{An example of 2-D convolution mapping a $3 \times 4$ tensor to a $2 \times 3$ tensor with a convolutional kernal of the size $ 2 \times 2 \times 1$}
\end{figure}

Generally speaking, $\tilde c \approx 2\hat{c}$, which means this output tensor will become longer in the channel dimension but just a little thinner in the essential dimension. We will introduce ``Pooling'' layers at last, which will keep the channel dimension but reduce the essential dimension very quickly.

\subsubsection{Nonlinear Activation layers}
This layer is very simple just like the nonlinear layers in feedforward fully connected neural network, just use some simple nonlinear $g: \mathbb{R} \to \mathbb{R}$, such as:
\begin{itemize}
\item Sigmoid or Tangent:
\begin{equation}
g = \sigma (x) = \frac{1}{1 + e^{-x}},
\end{equation}
or
\begin{equation}
g = tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}.
\end{equation}

\item Softplus:
\begin{equation}
g(x) = \zeta(x) = \log(1 + \exp{x}).
\end{equation}

\item Rectified Linear Units:
\begin{equation}
g(x) = ReLU(x) = \max\{0,x\}.
\end{equation}
ReLU function is the most commonly used one in nonlinear layers in CNN.
\end{itemize}

%Then the nonlinear layers can be expressed as:
%\begin{equation}
%f_{ReLu}: \mathbb{R}^{n_1 \times \cdots \times n_d \times \hat{c}} \xrightarrow{ReLU} \mathbb{R}^{n_1 \times  \cdots \times n_d \times \hat{c}},
%\end{equation}
%where
%\begin{equation}
%f_{ReLU}(T)(i_1,i_2,\cdots,i_d,i) = ReLU(T(i_1,i_2,\cdots,i_d,i)).
%\end{equation}

\subsubsection{Pooling layers}
For general DNN method, the convolutional layer stands for the linear transformation and then we use a nonlinear layer.  So we just need to repeat this two layers and get a DNN structure.  However, the biggest distinguish between CNN and DNN is the next layer in some degree.

Just as we mentioned before, the channel dimension will increase twice after a convolution layer but the essential dimension keeps for convolution with padding or only decrease linearly. Another big problem is that, because of the sparsity structure, it is hard to capture some "globally structure" or we see that, this model without some multi-scale structure like in image processing.
 So just as what we do in multigrid method, we need put those fine feature into some coarse level, that is exact the pooling or subsampling in image processing.

A pooling function is a little like the GMG coarse strategy.   For example, for the output of nonlinear layers with essential order $2$, for one channel $T$, we introduce the pooling method apply to every channel separately:
\begin{equation}
R(T)_{i,j} = r(T_{2i-1:2i,2j-1:2j})
\end{equation}
$r(\bm{x})$ can be:
\begin{itemize}
\item Fix position: $r(\bm{x}) = x_i$ for some fix $i$. This method can be combine into the convolution with stride as shown before.
\item Maxout: $r(\bm{x}) = \max \{x_1, \cdots, x_4\}$,
\item Average: $r(\bm{x}) = \frac{\sum_{i=1}^4 x_i}{4}$,
\item $L^2$ normal: $r(\bm{x}) = \|x\|$,
\item $\cdots$
\end{itemize}

In computer vision, they think that pooling helps to make the representation become approximately invariant to small translations of the input. Invariance means that if we translate the input by a small amount, the values of most of the pooled outputs do not change.
In each step, with the pooling function, we can transfer the $\tilde c$ channels into $\tilde c$ pooled channels (feature mapping)  separately.

\section{LeNet-5: The Pioneer of Convolutional Neural Networks}
This section is devoted to a model that is widely recognized as the first convolutional neural network: LeNet-5 \cite{Lecun1998Gradient}. In this section, we will introduce convolutional neural network via introducing LeNet-5 by explain every steps for LeNet-5.
Figure \ref{LeNet-5} shows an illustration of the architecture of LeNet-5. It consists of two pairs of Convolutional Layer and Subsampling Layer and is further connected with fully connected layer and an RBF layer for classification.

\begin{figure}[!htb]
	\center{\includegraphics[width=10cm] {LeNet-5.png}}
	\caption{The Architecture of LeNet-5}
	\label{LeNet-5}
\end{figure}
%\begin{figure}[htb]
%	\center{\includegraphics[width=10cm] {AlexNet.png}}
%	\caption{AlexNet}
%\end{figure}

\begin{itemize}
\item First of all, the input for LeNet-5 is digital picture with size $32 \times 32$ which every picture contains a number written by different writers. So, mathematically speaking, $T^j$ is a order 2 tensor with just one channel, with essential size $32 \times 32$, i.e $T^j \in \mathbb{R}^{32 \times 32 \times 1}$.  And the out put is a 10-dimensional vector $\bm y = (y_0, \cdots, y_9)$ with $y_i$ equal to the probability for the number in $T^j$ is $i$.

\item Input: $T \in  \mathbb{R}^{32 \times 32 \times 1}$

\item Input $\xrightarrow{\text{Convolution + ReLU}} C_1$: \\
This layer is not difficult to understand, this means you first use $6$ kernel $K_{1,i}, i = 1:6$ with $K_{1,i} \in  \mathbb{R}^{5 \times 5 \times 1}$, so you get $T\ast K_{1,i}$, and then you just need to use ReLU for $T \ast K_{1,i}, i=1:6$. This means:
\begin{equation}
\mathbb{R}^{32 \times 32 \times 1}  \xrightarrow{\text{Convolution ($K_{1,i}$) + ReLU}} \mathbb{R}^{28\times 28 \times 6}
\end{equation}
and
\begin{equation}
C_1 = ReLU(T\ast K_{1,i}) \quad i = 1:6.
\end{equation}

\item $C_1 \xrightarrow{\text{Subsampling(Pooling)}} S_2$: \\
This layer is general pooling layer only for essential dimension such as max pooling, average or $L^2$ norm. So we have:
\begin{equation}
\mathbb{R}^{28\times 28 \times 6} \xrightarrow{\text{Pooling}} \mathbb{R}^{14\times 14 \times 6}
\end{equation}

\item $S_2 \xrightarrow{\text{Convolution + ReLU}} C_3$:\\
This layer is a little different form the convolutional layer before because of the fact that $S_2$ is a multi-channel tensor.  So, when we do convolution for them, we need $16$ kernels as $K_{2,i} \in \mathbb{R}^{5 \times 5 \times 6}$:
\begin{equation}
\mathbb{R}^{14\times 14 \times 6}  \xrightarrow{\text{Convolution ($K_{2,i}$) + ReLU}} \mathbb{R}^{10\times 10 \times 16}
\end{equation}
and
\begin{equation}
C_3 = ReLU(S_2\ast K_{2,i}) \quad i = 1:16.
\end{equation}
There is an interesting structure for $K_{2,i}$ in the real LeNet-5. That is in the channel dimension for $K_{2,i}$ have some special "0" pattern, which can be shown as:
\begin{figure}[!htb]\label{LeNet-Channel}
	\center{\includegraphics[width=10cm] {LeNet-Channel.png}}
	\caption{The "0" Pattern for Channel Dimension of $K_{2,i}$}
\end{figure}

\item $C_3 \xrightarrow{\text{Subsampling(Pooling)}} S_4$:  \\
This layer is the same with $C_1 \xrightarrow{\text{Subsampling(Pooling)}} S_2$.

\item $S_4 \xrightarrow{\text{Subsampling(Pooling)}} C_5$:  \\
This layer use the naive idea we introduced at the beginning that we just reshape the $16$-channels $5\times 5$ tensor into a vector with dimension $400$. And then followed a fully connected feedforward neural network as a classifier.
\end{itemize}


\chapter{System Identification}

System identification can be defined as the deduction of system characteristics from measured data. \cite{NASA-RP-1138} Process of system identification includes performing identification experiment to obtain data, determining an appropriate form of the model and estimating the unknown parameters of the model with some statistically based method. \cite{SoderstromStoica1989}

\section{Linear Regression}

Least squares approximation is probably the most common method of the linear regression and it can be used to to approximate the solution of overdetermined systems.

Parameters of line given by the following equation:

\begin{align}
  y = ax + b
\end{align}

\clearpage

best fitting to the data set of $n$ points is given as follows:

\begin{align}
  a &= \frac{\sum_{i=1}^{n}{ \left( x_i - \bar{x} \right) \left( y_i - \bar{y} \right) }}{\sum_{i=1}^{n}{ \left( x_i - \bar{x} \right)^2 }} \\
  b &= \bar y - a \bar x
\end{align}

Where:

\begin{align}
  \bar{x} &= \frac{\sum_{i=1}^{n}{ x_i }}{ n } \\
  \bar{y} &= \frac{\sum_{i=1}^{n}{ y_i }}{ n }
\end{align}

\begin{figure}[h!]
  \centering
  \includegraphics[width=140mm]{eps/linear_least_squares.eps}
  \caption{Linear Least Squares}
\end{figure}