\subsection{Transport equation}
The advection-diffusion-reaction equation is the same in form for any tracer.
SELFE uses it to track salt, temperature and sediment concentration and water quality constituents.
The equation for a generic tracer $T$ is:
\beqa
  \frac{\pd T}{\pd t}+\nabla \cdot (\bs{u}T)
	&=& \frac{\pd }{\pd x} (\kappa_h \frac{\pd T}{\pd x}) + \frac{\pd }{\pd y} (\kappa_h \frac{\pd T}{\pd y})
	+\frac{\pd }{\pd z} (\kappa \frac{\pd T}{\pd z}) +\hat{Q},\label{tr1}\\
\eeqa
with vertical boundary conditions at the bed and free surface:
\beqa
  \kappa \frac{\pd T}{\pd z} &=& \hat{T}, \mbox{ at } z=\eta, \label{tr2} \\
  \kappa \frac{\pd T}{\pd z} &=& \hat{T_b}, \mbox{ at } z=-h, \label{tr3}
\eeqa
and concentration (Dirichlet, essential) boundary conditions at inflows and ocean boundaries.
where
\begin{align*}
&T(x,y,z,t)  &\text{concentration of the tracer} &\\
&\bs{u}(x,y,z,t)  &\text{3D velocity (m/s)}  &\\
&\kappa_{h} &\text{horizontal diffusivity} (m^2s^{-1}) &\\
&\kappa   &\text{vertical diffusivity}(m^2s^{-1}) &\\
&\hat{Q}  & \text{mass source}  &\\
\end{align*}

and the 3D velocity $\bs{u}$ must be provided in a mass-conserving (divergence free) form:
\beq
  \nabla \cdot (\bs{u})=0 \label{tr5}
\eeq

Horizontal mixing of constituent concentration was ignored throughout our project,
which is equivalent to $\kappa_h=0$. When shear in the main flow field is adequately
resolved, scaling arguments usually indicate that horizontal eddy diffusivity is very small
compared to the other terms and we assume that enough is introduced by the unavoidable horizontal
numerical diffusion introduced in solving the equations.


\Lecture{Jayalal Sharma}{Sept 19, 2020}{05}{Multichoosing}{Narasimha Sai Vempati}{$\alpha$}{JS}

\section{Introduction}
Consider the definition of \emph{set}. We know that it's a well defined collection of \emph{distinct} objects. From a collection of $n$ distinct symbols, the number of ways to form a \emph{set} of length $k$ is given by $\binom{n}{k}$. Now let's consider the definition of \emph{multi-set}. It's similar to that of a \emph{set}, except that it allows repetition of objects. Now it's natural ask the following question: From a collection of $n$ distinct symbols, what is the number of ways to form a \emph{multi-set} of length $k$. Multichoosing exactly answers this questions. In this lecture, we explore multichoosing in detail. We discuss several equivalent bijections to this problem and come-up with an algebraic expression for \mulnom{n}{k} (spelled out as $n$ \emph{multi-choose} $k$).

\section{Equivalent bijections} \label{sec:equi-bij}
\subsection{Non-negative solutions}\label{non-neq-sol-prob}
Formally, \mulnom{n}{k} is the number of ways of choosing $k$ objects from a set of $n$ objects where the order is not important but repetitions are allowed. For all $i=1,2,\cdots,n$, if we denote by $x_i$ the number of copies of $i^{th}$ object we choose, then we have the equation \begin{equation}\label{eqn1}
    x_1+x_2+\cdots+x_n=k
\end{equation} where each $x_i \geq 0$. Therefore, number of \emph{non-negative} integral solutions to this equation gives us the required number of ways of choosing $k$ objects from $n$ objects with given conditions. Let's look at an equivalent problem and establish a bijection between these two.

\subsection{Voting problem}\label{voting-prob} If $n$ candidates are contesting in an election and there are $k$ voters, how many ways can votes of those $k$ voters be distributed among $n$ candidates?

If we denote by $x_i$, the number of votes received by $i^{th}$ candidate and there are $k$ voters, we have $x_1+x_2+\cdots+x_n=k$ and thus, the number of ways of dividing votes among candidates is the number of non-negative solutions to the equation \ref{eqn1}. Formally, we can define a bijection $f$ from set of solutions to the equation \ref{eqn1} to set of ways of dividing the votes among $n$ candidates.
\begin{description}
\item\underline{Definition:}  $f$ takes the tuple $\vecx=(x_1,x_2,\cdots,x_n)$ and assign $x_i$ number of votes to $i^{th}$ candidate where $i=1,2,\cdots,n$.
\item\underline{Well defined:} $f$ is well defined because for every valid tuple $\vecx=(x_1,x_2,\cdots,x_n)$, we have $x_1+\cdots+x_n=k$ and thus summing over votes received by $i^{th}$ where $i=1,2,\cdots,n$ will be $k$ votes in total.
%there is a unique way of dividing the votes among candidates. In other words, for any two distinct way of dividing $k$ votes among $n$ candidates, there must exists an $i$ such that number of votes received $i^{th}$ candidate is different and thus $x_{1_i} \neq x_{2_i}$. Therefore $\vecx_1\neq\vecx_2$.
\item\underline{Injective:} $f$ is an injection because for every valid way of dividing the votes among candidates, there's a unique solution tuple in which $x_i = $ number of votes received by $i^{th}$ candidate. In other words, for any two $\vecx_1\neq\vecx_2$, there exists an $i\in[n]$ such that $x_{1_i}\neq x_{2_i}$ and $i^{th}$ candidate gets different votes. Thus $f(\vecx_1)\neq f(\vecx_2)$.
\item\underline{Surjective:} $f$ is surjective because for every way of dividing $k$ votes among $n$ candidates, there is a pre-image $\vecx=(x_1,\cdots,x_n)$ which is a valid solution to the equation \ref{eqn1} (as there are a total of $k$ voters, sum of number of votes received by each voter must sum up to $k$).
\end{description}
Thus $f$ is a bijection from the set of non-negative solutions to $x_1+\cdots+x_n=k$ to the set of ways of dividing $k$ votes among $n$ candidates.
\subsection{Non-decreasing subsequences}\label{non-dec-subseq-prob} Number of non-decreasing sequences of integers between $1$ and $n$ of length $k$. A non-decreasing sequence is of the form $\{a_1,a_2,\cdots,a_k\}$ where $1\leq a_1\leq a_2\cdots\leq a_k\leq n$. Lets define a bijection $f$ from set of non-negative integral solutions to Eqn. \ref{eqn1} to set of non-decreasing sequences between $1$ and $n$ of length $k$.
\begin{description}
\item\underline{Definition:} $f$ takes $\vecx=(x_1,\cdots,x_n)$ as input and writes the number $i$ $x_i$ times for all $i=1,2,\cdots,n$ to obtain a sequence of length $k$.
\item\underline{Well defined:} As $f$ constructs the sequence in increasing order from $1$ to $n$ by writing $i$ $x_i$ times, the resulting sequence will be non-decreasing. Therefore, $f$ is well defined.
\item\underline{Injective:} For every $\vecx_1\neq\vecx_2$, there exists an $i$ such that $x_{1_i} \neq x_{2_i}$ and thus in the resulting sequences, number $i$ is written different number of times. Therefore, $f$ is injective.
\item\underline{Surjective:} Every non-decreasing sequence of integers between $1$ and $n$ of length $k$ has a pre-image $\vecx=(x_1,\cdots,x_n)$  which is a valid solution to equation \ref{eqn1} (where $x_i$ is the number of times the number $i$ is present in the sequence and as length of sequence is $k$, all $x_i$'s where $i=1,2,\cdots,n$ sum up to $k$).
\end{description}
Thus $f$ is a bijection.
\subsection{Stars and bars problem}\label{star-bar-prob} There are $k$ stars placed horizontally. Find the number of ways to place $n-1$ bars in between those $k$ stars. Lets define a bijection $f$ from set of non-negative integral solutions to Eqn. \ref{eqn1} to set of ways of placing $n-1$ bars among $k$ stars.
\begin{description}
\item\underline{Definition:} $f$ takes $\vecx=(x_1,\cdots,x_n)$ as input and place $x_i$ number of stars between $(i-1)^{th}$ bar and $i^{th}$ bar. We leave it as an exercise to prove that $f$ is well-defined, injective and surjective.
\end{description}
\jsay{Prove that $f$ is a bijection}

\section{Algebraic expression}\label{alg-expr}
So far in Sec. \ref{sec:equi-bij}, we have established bijections between \emph{non-negatives integral} solutions of Eq. \ref{eqn1} and various other problems and argued that number of ways of solving any particular problem is equal to the number of non-negative integral solutions to Eq. \ref{eqn1}. In this section, we are interested in coming up with a concrete expression for \mulnom{n}{k} by solving it's equivalent bijection.

\paragraph{Method 1} Let's solve the \emph{stars and bars} problem defined in Sec. \ref{star-bar-prob}. Let's use the fact that any placement of $n-1$ bars among $k$ stars can be equivalently thought of as a string of length $n+k-1$ over the alphabet $\{\star,|\}$ with $k$ $\star$'s. Therefore, \begin{align*}
    \textrm{number of ways of placing } n-1 \textrm{ bars among } k \textrm{ stars } &= \textrm{number of such strings}\\
    &= \binom{n+k-1}{k}
\end{align*}

\paragraph{Method 2} Let's solve the \emph{Non-decreasing subsequences} problem defined in Sec. \ref{non-neq-sol-prob}. Let's establish a bijection $f$ from set $\beta$ of non-decreasing subsequences of integers between $1$ and $n$ of length $k$ to a set $\Gamma$ of strictly increasing subsequences of integers between $1$ and $n+k-1$ of length $k$. A strictly increasing subsequence is of the form $1\leq b_1<b_2<\cdots<b_k\leq n+k-1$
\begin{description}
\item{\underline{Definition:}} $f$ takes as input a non-decreasing subsequence $(a_1,a_2,\cdots,a_k)$ between $1$ and $n$ and for all $i=1,2,\cdots,k$ set $b_i = a_i+i-1$ and output the sequence $(b_1,b_2,\cdots,b_k)$
\item{\underline{Well defined:}} For any $(a_1,a_2,\cdots,a_k)\in\beta$, we have for all $i=1,2,\cdots,k-1$, \begin{align*}
    a_i &\leq a_{i+1}\\
    a_i+i &\leq a_{i+1}+i\\
    a_i+i-1 &< a_{i+1}+i\\
    b_i &< b_{i+1}
\end{align*}  Therefore, the subsequence $(b_1,\cdots,b_k)$ is strictly increasing subsequence and thus $f$ is well defined.
\item{\underline{Injective:}} For every non-decreasing subsequence $(a_1,\cdots,a_k)$, there's a unique strictly increasing subsequence $(b_1,\cdots,b_k)$ where for all $i=1,\cdots,k$, $b_i = a_i+i-1$. Therefore $f$ is injective.
\item{\underline{Surjective:}} For every strictly increasing subsequence $(b_1,\cdots,b_k)$, there's a pre-image $(a_1,\cdots,a_k)$ which is non-decreasing where for all $i=1,\cdots,k$, $a_i=b_i-i+1$
\end{description}

Therefore, $f$ is a bijection. The number of ways of choosing a strictly increasing subsequence $(b_1,\cdots,b_k)$ between integers $1$ and $n+k-1$ is just choosing $k$ integers from first $n+k-1$ integers and arrange them in one way(increasing order). Therefore number of ways = $\binom{n+k-1}{k}$. As $f$ is a bijection, therefore, the number of non-decreasing subsequences between $1$ and $n$ of length $k$ are $\binom{n+k-1}{k}$

\paragraph{Method 3} Let's solve the \emph{Voting} problem defined in Sec. \ref{voting-prob}. Let's ask a slightly modified question.
\begin{description}
\item \underline{Question:} How many ways to distribute $m$ votes among $n$ candidates such that each candidate gets at least one vote.
\item \underline{Answer 1:} As every candidate gets at least one vote, let's first distribute one vote each to each of the $n$ candidate and the distribute the remaining $m-n$ votes among $n$ candidates. By the bijection defined in Sec. \ref{voting-prob}, the number of ways of distributing $m-n$ votes among $n$ candidates is \mulnom{n}{m-n}
\item \underline{Answer 2:} Let's interpret votes as $\star$ s. Then the question essentially reduces to placing $n-1$ bars (since there are $n$ candidates, we divide by placing $n-1$ bars) among $m$ stars (since there are $m$ voters). $i^{th}$ candidate gets votes equal to number of stars between $(i-1)^{th}~|$ and $i^{th}~|$. However, there are two additional constraints \begin{enumerate}
    \item\label{cond1} A bar cannot be placed in the beginning or in the end (if not then either the first candidate or the last candidate gets $0$ votes)
    \item\label{cond2} We cannot place two $|$ s between same two $\star$ s (if we place $(i-1)^{th}~|$ and $i^{th}~|$ between same two $\star$ s, the $i^{th}$ candidate gets $0$ votes)
\end{enumerate}
Hence, we have to choose $n-1$ gaps among the $m-1$ gaps (because we have $m+1$ gaps and by cond. \ref{cond1} we remove two) to place $n-1~|$ s without repetitions (because repeating violates cond. \ref{cond2}). Therefore, there are $\binom{m-1}{n-1}$ ways of doing it. Thus \mulnom{n}{m-n}=$\binom{m-1}{n-1}$ and by substituting $m=n+k$, we have $$\textrm{\mulnom{n}{k}}=\binom{n+k-1}{n-1}=\binom{n+k-1}{k}$$
\end{description}
\section{Identities}
In this section, we discuss some identities on \mulnom{n}{k} and argue their proofs using the idea of either double counting or bijections.
\paragraph{Identity 1} $$\textrm{\mulnom{n}{k}}=\textrm{\mulnom{k+1}{n-1}}$$
\begin{proof}
Let's use the bijection method to prove this. Formally, lets define sets $S_1$ and $S_2$ and count their cardinalities independently and then establish a bijection from $S_1$ to $S_2$ proving that $|S_1|=|S_2|$.
\begin{description}
\item \underline{$S_1$:} Configuration of $k~\star$ s and $n-1~|$ s as described in Sec. \ref{star-bar-prob}. By the bijection defined in it, $|S_1|=$ \mulnom{n}{k}
\item \underline{$S_2$:} Configuration of $n-1~\star$ s and $k~|$ s as described in Sec. \ref{star-bar-prob}. Again, by the bijection defined in it, $|S_2|=$\mulnom{k+1}{n-1}
\item \underline{Bijection:} Let's define a bijection $f$ from $S_1$ to $S_2$. $f$ takes a configuration from $S_1$ as input and interpret $\star$ s as $|$ s and $|$ s as $\star$ s. Therefore it ends up with a configuration with $n-1~\star$ s and $k~|$ s which is a configuration is $S_2$. It's easy to observe that $f$ is a bijection.
\end{description}
As $f$ is a bijection from $S_1$ to $S_2$, we have $|S_1|=|S_2|$. This completes the proof
\end{proof}

\paragraph{Identity 2}
$$k~\textrm{\mulnom{n}{k}}=n~\textrm{\mulnom{n+1}{k-1}}$$
\begin{proof}
Let's use the method of double counting to prove this.
\begin{description}
\item \underline{Question:} In how many ways can we construct a non-decreasing sequence $1\leq a_1\leq a_2\cdots\leq a_k\leq n$ and mark one element?
\item \underline{Asnwer 1:} By the bijection established in Sec. \ref{non-dec-subseq-prob} we have \mulnom{n}{k} number of non-decreasing subsequences and for every such subsequence, we can mark any one of the $k$ elements choose. Thus the answer is $k$ \mulnom{n}{k}
\item \underline{Answer 2:} Firstly, determine the value in $[n]$ which is to be marked. Let $r$ be this value. Now, consider a non-decreasing subsequence between $1$ and $n+1$ with $k-1$ elements. Using $r$ and the non-decreasing sequence chosen, we construct a unique non-decreasing sequence between $1$ and $n$ of length $k$ with $r$ as marked in the following way:

Let $(b_1,b_2,\cdots,b_{k-1})$ with $1\leq b_1\leq b_2\leq\cdots\leq b_{k-1}\leq n+1$ be the chosen sequence,
\begin{itemize}
    \item Insert marked-$r$ in the right most position so that the resulting sequence is still sorted.
    \item As long as there's an $n+1$ in the sequence, remove it and add it as $r$ to the right of marked-$r$ in the sequence
\end{itemize}
Therefore, number of required sequences
\begin{align*}
    &= \textrm{ number of ways to choose }r \times \substack{\textrm{ number of non-decreasing sequences of length }\\ k-1 \textrm{ between } 1 \textrm{ and } n+1}\\
    &= n\times \textrm{\mulnom{n+1}{k-1}}
\end{align*}
\end{description}
This completes the proof
\end{proof}

\begin{ex}
    \item Prove the following by combinatorial arguments $$\textrm{\mulnom{n}{k}}=\sum\limits_{m=1}^{n}\textrm{\mulnom{m}{k-1}}$$ \emph{Hint: Look for bijection to number of non-decreasing subsequences}
    \item Prove the following by combinatorial arguments $$\sum\limits_{k=0}^{m}\textrm{\mulnom{n}{k}}=\textrm{\mulnom{n+1}{m}}$$ \emph{Hint: Look for bijection to Voting problem}
    \item Prove the following by combinatorial arguments $$\textrm{\mulnom{n}{k}}=\sum\limits_{m=0}^{n}\binom{n}{m}\textrm{\mulnom{m}{k-m}}$$
\end{ex}


\section{Logistic Regression}

\textbf{For the linear classification model, we assumed the class-conditional distributions to be Gaussian}
\begin{itemize}
\item We assumed $x| (t=1) \sim \mathcal{N}(\mu_+, \Sigma_+)$ and $x| (t=-1) \sim \mathcal{N}(\mu_-, \Sigma_-)$, and two class-probabilities $P(t=1)$ and $P(t=-1)$.
\item  This is called an \emph{generative model}, as we have written down a full joint model over the data.
\item  We saw that violations of the model assumption can lead to `bad' decision boundaries.
\end{itemize}

\begin{figure}
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture7/Figure44a.pdf}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture7/Figure44b.pdf}
	\end{subfigure}
	\caption{Figures taken from Bishop PRML}
\end{figure}

\textbf{For regression, we assumed Gaussian outputs, but did not need assumptions about the distribution of inputs.}
\begin{itemize}
\item For linear regression, we conditioned on $x$, and assumed a Gaussian distribution over $t$: $t|x \sim \mathcal{N}(y(x), \gamma^2)$
\item  We maximized the conditional log-likelihood $L(\omega)=\sum_n \log p(t_n|x_n, \omega)$, i.e we assumed that the $x$ were given.
\item  Therefore, this approach to linear regression works for \emph{any} distribution over $x$.
\item $x$ is typically high-dimensional, so it is difficult to make appropriate distributional assumptions for it.
\end{itemize}

\begin{figure}
	\centering
	\includegraphics[width=.4\textwidth]{./lecture7/Figure128.pdf}
	\caption{Figure taken from Bishop PRML}
\end{figure}

\textbf{We can define a discriminative model for classification by modelling the conditional class probabilities.}
\begin{itemize}
\item From the homework-exercise, we know that $P(t=1 | z(x))=\sigma(z(x))$ where $\sigma(z)=1/(1+\exp(-z))$ and $z(x)=
\omega^\top x+\omega_o$.
\item Notation is simpler if we use $0$ and $1$ as class labels, so we define $s_n=1$ as the label for the positive class, and $s_n=0$ als label for the negative class.
\item In other words, $s |x \sim \mbox{Bernoulli}(\sigma(z(x))$.
\item Also, we set $y_n=\sigma(z(x))$.
\item  The parameters of $z(x)=\omega^\top x+ \omega_o$ can be learned by maximizing the conditional log-likelihood $L(\omega)=\sum_n \log p(t_n|x_n, \omega)$ [on board]
\item  This is an \emph{discriminative} approach to classification, as we only model the labels, and not the inputs.
\item
 Decision rule and function shape of $p(t|x)$ will be the same for the generative (`Linear Discriminant Analysis') and the discriminative model, but the parameters were obtained differently.
\end{itemize}

\begin{bbbox}{Log-likelihood for logistic regression}
	Please note that we drop $\omega_0$ for simplicity, or similarly $\omega_0 = 0$.
	\begin{flalign*}
		& z(x) = \omega^{\top}x & \\
		& P\left(\lbrace t_n \rbrace, \lbrace x_n \rbrace, \omega \right) = \prod_{n=1}^N P(t_n|x_n,\omega) & \\
		& \qquad {} = \prod_{n=1}^N \left\{
				\begin{aligned}
		         	& \sigma(z(x)); & t_n = 1; & s_n = 1\\
		         	&1-\sigma(z(x)); & t_n = -1; & s_n = 0
         		\end{aligned} \right. &\\
		& \qquad {} = \prod_{n=1}^N \left\{
				\begin{aligned}
		         	& y_n; & t_n = 1; & s_n = 1\\
		         	& 1-y_n; & t_n = -1; & s_n = 0
         		\end{aligned} \right. &\\
		& \qquad {} = \prod_{n=1}^N y_n^{s_n} (1-y_n)^{1-s_n} \\
		& L_\omega = \sum_{n=1}^N \log \left( p(t_n|x_n,\omega) \right) &\\
		& \qquad {} = \sum_{n=1}^N s_n \log \left( y_n \right) + (1-s_n) \log \left( 1- y_n \right)&\\
	\end{flalign*}
\end{bbbox}

\subsection{Maximum likelihood estimation of Logistic Regression}

\textbf{Maximum likelihood estimation of Logistic Regression}

\begin{itemize}
\item This algorithm is called \emph{logistic regression}, and is a \emph{much} better algorithm than the algorithms we discussed last week.
\item Need to optimize log-likelihood numerically.
\item  People typically minimize the negative log-likelihood $\mathcal{L}$ rather than maximize the log-likelihood...
\item  To numerically minimize the negative log-likelihood, we need its gradient (and maybe its hessian) [on board]
\end{itemize}

\begin{bbbox}{Gradient and Hessian for Logistic Regression}
	Rather than maximizing the log-likelihood L people usually minimize the negative log-likelihood $\mathcal{L}$. Since there is no close form solution for finding the minimum in the case of a logistic regression we utilize iterative algorithms. Those algorithms require that we calculate the gradient of $\mathcal{L}$: \\
	\begin{flalign*}
		& \frac{\partial \mathcal{L}}{\partial \omega_i}
		    = - \sum_{n=1}^N s_n \frac{1}{y_n} y_n (1-y_n) x_n^{(i)}
		      + (1-s_n) \frac{1}{1-y_n} y_n (-1) x_n^{(i)} & \\
		& \qquad {} = - \sum_{n=1}^N x_n^{(i)} \left[ s_n - s_ny_n - y_n + s_ny_n \right] & \\
		& \qquad {} = \sum_{n=1}^N \left[ y_n -s_n \right] x_n^{(i)} & \\
		& \nabla \mathcal{L} = \sum_{n=1}^N \left[ y_n -s_n \right] x_n & \\
	\end{flalign*}
However some algorithms utilize the so called Hessian of our negative log-likelihood to reduce the number of iterations need for convergence. In word the Hessian is a square matrix of second order partial derivatives of $\mathcal{L}$. Since we have already calculated the first order derivative we can use it to find the Hessian: \\
\begin{flalign*}
	& \frac{\partial \mathcal{L}}{\partial \omega_i \partial \omega_j}
		= \frac{\partial}{\partial \omega_j} \left[ \frac{\partial \mathcal{L}}{\partial \omega_i} \right]
		= \frac{\partial \sum_{n=1}^N \left[ y_n -s_n \right] x_n^{(i)}}{\partial \omega_j}
		= \sum_{n=1}^N \left[ y_n (1- y_n) \right] x_n^{(i)}x_n^{(j)} & \\
	& \nabla \nabla \mathcal{L} = \sum_{n=1}^N \left[ y_n (1- y_n) \right] x_n x_n^{\top} & \\
	& \text{where } \left[x_n x_n^{\top} \right]_{ij} = x_n^{(i)}x_n^{(j)} &
\end{flalign*}
\end{bbbox}

\textbf{The cost-function for logistic regression is convex.}
\begin{figure}
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture7/Convex.pdf}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.45\textwidth}
		\includegraphics[width=\textwidth]{./lecture7/NotConvex.pdf}
	\end{subfigure}
\end{figure}

\begin{itemize}
\item  Fact: The negative log-likelihood is \emph{convex} -- this makes life much more easier.
\item  There are no local minima to get stuck in, and there is good optimization techniques for convex problems.
\end{itemize}

\begin{bbbox}{Notes on convex functions and the Hessian}
	A function is convex whenever it's Hessian is positive definite.\\
	A matrix is positive definite if: $v^{\top} M v > 0; \forall v: \|v\| \not= 0$ \\
	Note that having a symmetric matrix is not sufficient for a positive definite matrix. \\
	A convex funcion has only one minimum, i.e. minimizing guartuees global minimum.
\end{bbbox}

\textbf{\emph{Gradient descent} is a simple method for numerically minimizing a function.}
\begin{itemize}
\item The gradient $\nabla \mathcal{L}$ of a function points into the direction of steepest descent.
\item  Gradient descent: 'run down the gradient' $\omega_{new}=\omega_{old}-\alpha \nabla \mathcal{L}_\omega$, with learning rate $\alpha$.
\item  Slightly more sophisticated version: numerically optimize $\alpha$ for each step by doing a \emph{line search}.
\item  Convergence can be very slow if cost-function has `valleys'.
\end{itemize}

\begin{figure}
\centering
\includegraphics[width=.7\textwidth]{./lecture7/BoydGradientDescent}
\caption{Figure taken from Boyd, Convex Optimization}
\end{figure}

\begin{figure}
	\begin{subfigure}[b]{0.2\textwidth}
		\includegraphics[width=\textwidth]{./lecture7/Newton0}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.2\textwidth}
		\includegraphics[width=\textwidth]{./lecture7/Newton1}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.2\textwidth}
		\includegraphics[width=\textwidth]{./lecture7/Newton2}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.2\textwidth}
		\includegraphics[width=\textwidth]{./lecture7/Newton3}
	\end{subfigure}
	\\
	\begin{subfigure}[b]{0.2\textwidth}
		\includegraphics[width=\textwidth]{./lecture7/Newton4}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.2\textwidth}
		\includegraphics[width=\textwidth]{./lecture7/Newton5}
	\end{subfigure}
	~
	\begin{subfigure}[b]{0.2\textwidth}
		\includegraphics[width=\textwidth]{./lecture7/Newton6}
	\end{subfigure}
	\caption{\emph{Iterative Least Squares:} Approximate by parabola, minimze, iterate.}
\end{figure}

\textbf{\emph{Iterative Least Squares} is a more efficient method for minimizing the cost-function}
\begin{itemize}
\item Newton-Raphson: $\omega_{new}=\omega_{old}-\alpha (\nabla \nabla \mathcal{L})^{-1}\nabla L_\omega$
\item Pre-multiplying the gradient by the inverse-hessian speeds up convergence `along valleys' (analogy with LDA)
 \item Motivation: For quadratic functions $F(x)=a+b^\top x+ x^\top B x$, Newton-Raphson finds the minimum in one iteration.
 \item In this contex, Newton-Raphson (with $\alpha=1$) is often called \emph{iterative least squares}.
 \item Note: Newtwn's method can be bad if problem is not convex, and can be slow if it is difficult to calculate/invert the Hessian. A large number of optimization algorithms exist which do not require the (complete) Hessian (quasi Newton/BFGS, etc..).
% \item Any (reasonable) optimization algorithm requires the gradient.
\end{itemize}

\textbf{Visualizing the cost-function of logistic regression}

[on board]

\subsection{Bayesian Logistic Regression: Approximating the posterior distribution}
\textbf{Bayesian inference for this model does not have a closed form solution}
\begin{itemize}
\item Typically use Gaussian prior on $\omega$.
\item For linear regression, posterior distribution was Gaussian, with closed-form solutions for the mean and covariance.
\item For logistic regression, the posterior distribution is non-Gaussian.
\item  Popular approximation: Approximate posterior by a Gaussian
\begin{align}
p(\omega|D) \approx \mathcal{N}(\mu_{post}, \Sigma_{post})
\end{align}
\item  Different methods exist for finding `good' $\mu_{post}$ and $\Sigma_{post}$: Expectation Propagation (EP), Laplace Approximation, Variational Inference
\end{itemize}

\textbf{The Laplace-Approximation is a simple Gaussian approximation to the posterior}
\begin{itemize}
\item \emph{Laplace approximation:} $\mu_{post}=\omega_{MAP}$, $\Sigma_{post}=\left(\nabla \nabla_\omega L   \right)^{-1}$.
\item Take MAP as mean, and inverse hessian at MAP as covariance.
\item  Motivation: Curvature matching, Taylor-expansion [on board]
\item  Q: When will the Laplace approximation fail?
\end{itemize}
\begin{figure}
\includegraphics[width=.49\textwidth]{./lecture7/Figure414a.pdf}
\includegraphics[width=.49\textwidth]{./lecture7/Figure414b.pdf}
\caption{Figure taken from Bishop PRML}
\end{figure}

%\textbf{The posterior distribution can be used to calculate the predictive distribution and to optimze hyper-parameters}
%[on board]

%\section{LR's popular little brother-- support vector machines}

%\section{The exam}


\chapter{CT Systems as Linear Constant Coefficient Differential Equations}

Recall a system is a transformation of signals, turning the input signal into the output signal. While this might seem like a new concept to you, you already know something about them from your differential equations course, i.e. MATH 2214 and your circuits course.

For example, consider the following circuit:
\begin{center}
  \begin{circuitikz}[american voltages,scale=0.8, every node/.style={transform shape}]
    \draw
    (0,2.3) to[battery, l=$1\mbox{ VDC}$] (0,0)
    (3,2) node[spdt,xscale=-1,yscale=-1,anchor=in] (Sw) {}
    (0,2.3) to[short] (Sw.out 2)
    (Sw.out 1) to[short] (1.8,0)
    (0,0) to[short, -o] (8,0)
    (3,2) to[short] (4,2)
    (4,2) to[open, v=$x(t)$] (4,0)
    (4,2) to[R, l=$R$] (6,2)
    (6,2) to[short, -o] (8,2)
    (6,0) to[C, l=$C$] (6,2)
    (8,2) to[open, v=$V_C(t)$] (8,0);
  \end{circuitikz}
\end{center}
where the switch moves position at $t = 0$. The governing equation for the circuit when $t < 0$ is
\[
\frac{dV_c}{dt}(t) + \frac{1}{RC}V_c(t) = 0
\]
a \emph{homogeneous} differential equation of first-order. From a DC analysis, the initial condition on the capacitor voltage is $V_C(0^-) = 0$, so there is no current flowing prior to $t = 0$ and the solution is $V_C(t) = 0$ for $t < 0$.

After the switch is thrown, the governing equation for the circuit when $t \geq 0$ is
\[
\frac{dV_c}{dt}(t) + \frac{1}{RC}V_c(t) = \frac{1}{RC}
\]
Since the voltage across the capacitor cannot change instantaneously $V_C(0^-) = V_C(0^+) = 0$, giving the auxillary condition necessary to solve this equation, which has the form
\[
V_C(t) = A + Be^{-\frac{1}{RC}t}
\]
Using the auxillary condition we find
\[
V_C(0) = A + Be^{-\frac{1}{RC}0} = A + B = 0 \mbox{ which implies } B = -A
\]
Subsitution back into the differential equation and equating the coefficients gives $A = 1$. Thus the voltage for $t \geq 0$ is
\[
V_C(t) = 1 - e^{-\frac{1}{RC}t}
\]

Suppose we consider the voltage after the switch as the input signal $x(t)$ to the system composed of the series RC. As we have seen previously a mathematical model of the switch is the unit step $x(t) = u(t)$. Suppose we consider the capacitor voltage at the outut of the system, so that $y(t) = V_C(t)$. Then we can consider the system to be represented by the \emph{linear, constant-coefficient differential equation}
\[
\frac{dy}{dt}(t) + \frac{1}{RC}y(t) = \frac{1}{RC}x(t)
\]
where $x(t) = u(t)$ and the solution $y(t)$ is the \emph{step response}
\[
y(t) = \left(1 - e^{-\frac{1}{RC}t}\right)u(t)
\]

As we will see later this representation of systems is central to the course, so we take some time here to review the solution of such equations.

\section{Solving Linear, Constant Coefficient Differential Equations}

A linear, constant coefficient (LCC) differential equation is of the form
\[
a_0\, y + a_1\, \frac{dy}{dt} + a_2\, \frac{d^2y}{dt^2} + \cdots + a_N\, \frac{d^Ny}{dt^N}  = b_0\, x + b_1\, \frac{dx}{dt} + bb_2\, \frac{d^2x}{dt^2} + \cdots + b_M\, \frac{d^My}{dt^M}
\]
which can be written compactly as
\[
\sum\limits_{k = 0}^{N} a_k\, \frac{d^ky}{dt^k} = \sum\limits_{k = 0}^{M} b_k\, \frac{d^kx}{dt^k}
\]

It is helpful to clean up this notation using the derivative operator $D^n = \frac{d^n}{dt^n}$. For example
$D^2y = \frac{d^2y}{dt^2}$ and $D^0 y= y$. To give for form as
\[
\sum\limits_{k = 0}^{N} a_k\, D^k y = \sum\limits_{k = 0}^{M} b_k\, D^k x
\]

We can factor out the derivative operators
\[
a_0y + a_1Dy + a_2D^2y + \cdots + a_ND^Ny  = b_0\, x + b_1\, Dx + b_2\, D^2x + \cdots + b_M\, D^M x
\]
\[
\underbrace{\left(a_0 + a_1D + a_2D^2 + \cdots + a_ND^N\right)}_{\text{Polynimial in } D, Q(D)} y = \underbrace{\left(b_0 + b_1 D + b_2 D^2 + \cdots + b_M D^M\right)}_{\text{Polynimial in } D, P(D)} x
\]
to give:

\[
Q(D)y = P(D)x
\]
You learned how to solve these in differential equations (Math 2214) as
\[
y(t) = y_\text{h}(t) + y_\text{p}(t)
\]

The term $y_\text{h}(t)$ is the solution of the homogeneous equation
\[
Q(D)y = 0
\]
Given the $N-1$ auxillary conditions $y(t_0) = y_0$, $Dy(t_0) = y_1$, $D^2y(t_0) = y_2$, up to $D^{N-1}y(t_0) = y_{N-1}$.

The term $y_\text{p}(t)$ is the solution of the particular equation
\[
Q(D)y = P(D)x
\]
for a given $x(t)$.

Rather than recapitulate the solution to $y_\text{h}(t)$ and $y_\text{p}(t)$ in the general case we focus on the homogeneous solution $y_\text{h}(t)$ only. The reason is that we will use the homogeneous solution to find the impulse response below and take a different approach to solving the general case for an arbitrary input using the impulse response and convolution (next week).

To solve the homogenous system:

\textbf{Step 1:} Find the \emph{characteristic equation} by replacing the derivative operators by powers of an aribrary complex variable $s$.
\[
Q(D) = a_0 + a_1D + a_2D^2 + \cdots + a_ND^N
\]
becomes
\[
Q(s) = a_0 + a_1s + a_2s^2 + \cdots + a_Ns^N
\]
a polynomial in $s$ with $N$ roots $s_i$ for $i = 1, 2, \cdots, N$ such that
\[
(s - s_1)(s-s_2)\cdots(s-s_N) = 0
\]

\textbf{Step 2:} Select the form of the solution, a sum of terms corresponding to the roots of the characteristic equation.

\begin{itemize}
\item For a real root $s_1\in \mathbb{R}$ the term is of the form
  \[
  C_1 e^{s_1 t}.
  \]
\item For a pair of complex roots (they will always be in pairs) $s_{1,2} = a \pm jb$ the term is of the form
  \[
  C_1 e^{s_1 t} + C_2 e^{s_2 t} = e^{a t}\left(C_3\cos(bt) + C_4\sin(bt)\right) = C_5 e^{a t}\cos(bt + C_6).
  \]
\item For a repeated roots $s_1$, repeated r times, the term is of the form
  \[
  e^{s_1 t} (C_0 + C_1 t + \cdots + C_{r-1} t^{r-1}).\]
\end{itemize}

\textbf{Step 3:} Solve for the unknown constants in the solution using the auxillary conditions.

We now examine two common special cases, when $N=1$ (first-order) and when $N=2$ (second-order).

\subsection{First-Order Homogeneous LCCDE}

Consider the first order homogeneous differential equation
\[
\frac{dy}{dt}(t) + ay(t) = 0 \mbox{ for } a \in \mathbb{R}
\]
The characteristic equation is given by
\[
s + a = 0
\]
which has a single root $s_1 = -a$. The solution is of the form
\[
y(t) = Ce^{s_1 t} = Ce^{-a t}
\]
where the constant $C$ is found using the auxillary condition $y(t_0) = y_0$.

\textit{Example}: Consider the homogeneous equation
\[
\frac{dy}{dt}(t) + 3y(t) = 0 \mbox{ where } y(0) = 10
\]
The solution is
\[
y(t) = Ce^{-3 t}
\]
To find $C$ we use the auxillary condition
\[
y(0) = Ce^{-3 \cdot 0} = C = 10
\]
and the final solution is
\[
y(t) = 10e^{-3 t}
\]
\subsection{Second-Order Homogeneous LCCDE}

Consider the second-order homogeneous differential equation
\[
\frac{d^2y}{dt^2}(t) + a\frac{dy}{dt}(t) + by(t) = 0 \mbox{ for } a,b \in \mathbb{R}
\]
The characteristic equation is given by
\[
s^2 + as + b = 0
\]

Let's look at several examples to illustrate the functional forms.

Example 1:
\[
\frac{d^2y}{dt^2}(t) + 7\frac{dy}{dt}(t) + 10y(t) = 0
\]
The characteristic equation is given by
\[
s^2 + 7s + 10 = 0
\]
which has roots $s_1 = -2$ and $s_2 = -5$. Thus the form of the solution is
\[
y(t) = C_1e^{-2t} + C_2e^{-5t}
\]

Example 2:
\[
\frac{d^2y}{dt^2}(t) + 2\frac{dy}{dt}(t) + 5y(t) = 0
\]
The characteristic equation is given by
\[
s^2 + 2s + 5 = 0
\]
which has complex roots $s_1 = -1+j2$ and $s_1 = -1-j2$. Thus the form of the solution is
\[
y(t) = e^{-t}\left(C_1\cos(2t) + C_2\sin(2t)\right)
\]

Example 3:
\[
\frac{d^2y}{dt^2}(t) + 2\frac{dy}{dt}(t) + y(t) = 0
\]
The characteristic equation is given by
\[
s^2 + 2s + 1 = 0
\]
which has a root $s_1 = -1$ repeated $r=2$ times. Thus the form of the solution is
\[
y(t) = e^{-t}\left(C_1 + C_2t\right)
\]

In each of the above cases the constants, $C_1$ and $C_2$, are found using the auxillary conditions $y(t_0)$ and $y\prime(t_0)$.

\section{Finding the impulse response of a system described by a LCCDE}

As we will see next week an important response of a system is the one that corresponds to an impulse input, i.e. the \emph{impulse response} $y(t) = h(t)$ when $x(t) = \delta(t)$. Thus we focus here on a recipe for solving LCCDEs for this special case when $M \leq N$. We will skip the derivation of why this works.

Our goal is to find the solution to $Q(D)y = P(D)x$ when $x(t)=\delta(t)$.

\textbf{Step 1:} Let $y_h(t)$ be the homogeneous solution to $Q(D)y_h = 0$ for auxillary conditions
  \[
    D^{N-1}y_h(0^+) = 1 \; , \; D^{N-2}y_h(0^+) = 0 \; , \; \text{etc.} \; y_h(0^+) = 0
    \]

\textbf{Step 2:} Assume a form for $h(t)$ given by:
  \[
  h(t) = \underbrace{b_N\delta(t)}_{=0 \text{ unless } N=M} + \underbrace{\left[ P(D)y_h\right]}_{\text{apply } P(D) \text{ to } y_n(t)}u(t)
  \]

Recall from above the homogeneous solution depends on the roots of the characteristic equation $Q(D) = 0$.

\begin{itemize}
\item roots are either real, or
\item roots occur in complex conjugate pairs, or
\item repeated roots.
\end{itemize}

Example 1: Find the impulse response of the LCCDE
\[
\frac{dy}{dt}(t) + y(t) = x(t)
\]
The characteristic equation is given by
\[
s + 1 = 0
\]
which has a single root $s_1 = -1$. The solution is of the form
\[
y_h(t) = Ce^{-t}
\]
with the special auxillary condition $y(0) = 1$, so that
\[
y_h(t) = e^{-t}
\]
Since $P(D) = 1$ and $N = 1 \neq M = 0$ the impulse response is
\[
h(t) = \underbrace{b_N\delta(t)}_{=0} + \left[ \underbrace{P(D)}_{1}y_h(t)\right]u(t) = e^{-t}u(t)
\]

Example 2: Find the impulse response of the LCCDE
\[
\frac{dy}{dt}(t) + y(t) = \frac{dx}{dt}(t) + x(t)
\]
The homogeneous solution is the same as in Example 1,
\[
y_h(t) = e^{-t}
\]
however now $M = N = 1$ with $b_1 = 1$ and $P(D) = D+1$. Thus, the impulse response is
\[
h(t) = \underbrace{b_N}_{=1}\delta(t) + \left[ \underbrace{P(D)}_{D+1}y_h(t)\right]u(t) = \delta(t) + \left\{[D+1]e^{-t}\right\}u(t) = \delta(t) + [- e^{-t} + e^{-t}]u(t) = \delta(t)
\]

Example 3: Find the impulse response of the LCCDE
\[
\frac{d^2y}{dt^2}(t) + 7\frac{dy}{dt}y(t) + 10y(t) = x(t)
\]
The characteristic equation is given by
\[
s^2 + 7s + 10 = 0
\]
which has roots $s_1 = -2$ and $s_2 = -5$. Thus the form of the solution is
\[
y_h(t) = C_1e^{-2t} + C_2e^{-5t}
\]
The special auxillary conditions are $y_h(0) = 0$ and $y^\prime_h(0) = 1$. Using these conditions
\[
y_h(0) = C_1e^{-2t} + C_2e^{-5t} |_{t = 0} = C_1 + C_2 = 0
\]
\[
y^\prime_h(0) = -2C_1e^{-2t} - 5C_2e^{-5t} |_{t = 0} = -2C_1 -5C_2 = 1
\]
Solving for the constants gives $C_1 = \frac{1}{3}$ and $C_2 = -\frac{1}{3}$. Since $P(D) = 1$ and $N = 2 \neq M = 0$ the impulse response is
\[
h(t) = \underbrace{b_N\delta(t)}_{=0} + \left[ \underbrace{P(D)}_{1}y_h(t)\right]u(t) = \frac{1}{3} e^{-2t}u(t) - \frac{1}{3} e^{-5t}u(t)
\]


\section{Expressiveness}
\label{sec:expressiveness}

A major advantage of ARX cipher family is the expressiveness
\footnote{The content is inspired by a question on StackExchange:
\url{https://crypto.stackexchange.com/questions/51412/}}.
With only three primitives, ARX cipher is able to perform almost all confusion and
diffusion techniques. Theoretically, it is shown that $\textit{ARX}(n)$ primitives
can simulate all permutations over $\mathbb{Z}_{2^n}$, hence replace the S-box in
block cipher constructions. A more general conclusion states that, $\textit{ARX}(n)$
can represent all functions $\mathbb{Z}_{2^n}^m\rightarrow\mathbb{Z}_{2^n}$ for any
positive integer $m$.

\subsection{Simulating Permutations}

A \textit{permutation} of the finite set $X$ is a bijective function $\pi:\textit{X}\rightarrow X$.
Such $\pi$ is called a permutation over $X$. In other word, permutation is the
rearrangement of a finite sequence. It is trivial to see that all permutations can be
generated by solely swapping operations on any other permutation. The swapping on a
finite sequence is defined as follow:

\begin{definition}
Let sequence $X=(x_1,x_2,\cdots,x_n)$, \textit{swapping} two elements $x_i$, $x_j$ in
$X$ resulting another sequence $X'=(x'_1,x'_2,\cdots,x'_n)$ such that:
\begin{align*}
x_i &= x'_j \\
x_j &= x'_i \\
x_k &= x'_k\ \mathrm{for}\ k\notin\{i,j\}
\end{align*}
\end{definition}

Many block ciphers adopt permutations as the nonlinear components, which also called
the \textit{S-boxes} \cite{stinson2005cryptography}. ARX cipher obtains its nonlinearity
from the modular addition, which is able to form (or \textit{simulate}) more complicated
nonlinear components. A nice property of ARX family states that, it can simulate all
permutations over the message space:

\begin{theorem}
\label{thm:perm}
For all permutations $\pi$ over $\mathbb{Z}_{2^n}$, $\pi\in\textit{ARX}(n)$.
\end{theorem}

Given arbitrary permutation, say that, the permutation in natural ordering, it is able to
generate any desired permutation by a pile of swapping operations on specific pairs of
permutation elements. Then, by showing swapping can be realized by ARX operations,
Theorem \ref{thm:perm} can be proven:

\begin{proof}
We proof this by construction, that is, we will construct a function $\sigma$ such that
$\sigma\textit{(X)}$ swaps two particular elements in the permutation $X$. Note that
$\sigma$ doesn't consider the positions of the elements being swapped, however, since all
elements in a permutation are unique, this definition does not introduce any ambiguity.

The problem can therefore be simplified as, constructing an ARX function $\sigma$ such
that, for input pair $(a,b)$, there are $\sigma(a)\textit{=}b$, $\sigma(b)\textit{=}a$,
and $\sigma(x)\textit{=}x$ for $x\notin\{a,b\}$. Denote this function by $f_{a,b}(x)$.
Given multiple such functions, their composition is thus a permutation over the domain.

Without loss of generality, let $a\neq b$. For simplicity, let $x-y=x+(2^n-y)$, and
$\mathrm{r}^{-b}(x)=\mathrm{r}^{n-b}(x)$ (the right rotation). We then construct
$f_{a,b}(x)$ by the following routine:

\begin{enumerate}
\setlength\partopsep{0em}
\setlength\topsep{0em}
\setlength\itemsep{0em}
\setlength\parskip{0em}
\item let $a'=0$, $b'=b-a$, the base function $f_0(x)\textit{=}x-a$
\item find the integer $i$ such that $\mathrm{r}^{-i}(b')$ is an odd number, set
$b'=\mathrm{r}^{-i}(b')$ and function $f_1(x)\textit{=}\mathrm{r}^{-i}(f_0(x))$, denoted
by $f_1\textit{=}\mathrm{r}^{-i}(f_0)$
\item \label{enum:loopcond} loop Step \ref{enum:loop} for $b'\neq1$, otherwise, go to
Step \ref{enum:loopout}
\item \label{enum:loop} let $b'=(b'\oplus1)-1$, function $f_i\textit{=}(f_{i-1}\oplus1)-1$
\item \label{enum:loopout} let $f_k=f_{i}+2^n-2$, and the auxiliary function
$g(x)=\mathrm{r}^1(\mathrm{r}^{-1}(x+2)-1)$
\item output function $\sigma=f_k^{-1}(g(f_k))$ as $f_{a,b}$
\end{enumerate}

\noindent Here we take $f_i$ as the last function when the looping condition at Step
\ref{enum:loopcond} no longer holds. Also notice that $g(x)$ is equivalent to swapping
function $f_{2^n-2,2^n-1}(x)$. The inverse function $f_k^{-1}$ can be easily calculated
since $f_k$ is a composition of ARX primitives.

To see $\sigma$ swap $(a,b)$, calculate the following functions:

\begin{equation*}
\setlength{\abovedisplayshortskip}{0em}
\setlength{\abovedisplayskip}{0em}
\begin{split}
f_0(a) &= 0\\
f_1(a) &= 0\\
f_i(a) &= 0\\
f_k(a) &= 2^n-2\\
g(f_k(a)) &= 2^n-1\\
\end{split}
\quad\quad
\begin{split}
f_0(b) &= b-a\\
f_1(b) &= (b-a)/2^i\\
f_i(b) &= 1\\
f_k(b) &= 2^n-1\\
g(f_k(b)) &= 2^n-2\\
\end{split}
\end{equation*}

\noindent Notice that $f_k(b)=2^n-1$ indicates the inverse function $f_k^{-1}(2^n-1)=b$,
hence $\sigma(a)=f_k^{-1}(g(f_k(a)))=b$. Similarly, there are $\sigma(b)=a$.

For $x\notin\{a,b\}$, there are $f_k(x)\notin\{2^n-2,2^n-1\}$ and thus $g(x)=x$. It
follows that $\sigma(x)=f_k^{-1}(g(f_k(x)))=f_k^{-1}(f_k(x))=x$ hence complete the proof.
\end{proof}

A Python3 implementation of the above construction is presented in Appendix
\ref{ssec:permarx}, it shows that this construction is feasible at software level.

\subsection{Completeness of \textbf{\textit{ARX}(n)}}

A more general statement regarding the expressiveness of ARX ciphers is given as:

\begin{theorem}
\label{thm:func}
For all function $f:\mathbb{Z}_{2^n}\rightarrow\mathbb{Z}_{2^n}$, $f\in\textit{ARX}(n)$.
\end{theorem}

Formally, it is said that ARX operations are functionally complete in the set of
functions over $\mathbb{Z}_{2^n}$ \cite{khovratovich2010rotational}. Khovratovich and
Nikoli{\'c} give a proof by construction for this theorem\footnote{the proof here does
not strictly follow the origin, as it has several mistakes and is in different notation}:

\begin{proof}
Given a sequence $X=x_1x_2\cdots x_n$, the task is to compute all functions $F(X)$ via
ARX primitives.

Firstly, let $$s_i(X)=\mathrm{r}^1\Bigg(\sum^{2^{n-1}}\mathrm{r}^i(X)\Bigg)$$ which
moves the $i$-th bit of X to the rightmost while keeps all other bits 0. To see this,
note that the innermost $\mathrm{r}^i(X)=x_{i+1}...x_nx_1...x_i$, adding itself by
$2^{n-1}$ times is equivalent to applying left-shifting by $n-1$ bits, that results
in $x_i$ at the leftmost while all other bits are $0$s. In other word, $s_i(X)=00...0x_i$.

Then define function $$M_k(X,Y)=\mathrm{r}^{-1}\Bigg(\sum^2\mathrm{r}^{-1}(s_k(X)+s_k(Y))\Bigg)$$
for two sequences $X$ and $Y$. $M_k$ compute the product of two bits $x_k$ and $y_k$,
\ie, $M_k(X,Y)=x_ky_k$. This is equivalent to the bitwise-and of $x_k$ and $y_k$.
The innermost sum $s_k(X)+s_k(Y)=00...0(x_ky_k)(x_k\oplus y_k)$. By rotating it by 1
bit and multiplying by 2, the product $x_ky_k$ is left at the $n\textrm{-}1$-th bit.

Next, compute the equivalence trial $$j_C(X)=\begin{cases}
1\textrm{ , if }X=C=c_1c_2...c_n\\
0\textrm{ , otherwise}\\
\end{cases}$$ the function can be realized by function $M_k$ and constant 1, that is,
$j_C(X)=\Big[\bigoplus_{i=1}^nM_i(X,C)\Big]\oplus1$.

Finally, let function $J_{C_1,C_2}(X)=C_2\cdot j_{C_1}(X)$, that evaluates to $C_2$ if
$X=C_1$ and to 0 for other values. The output function is $f=J_{\textit{X}, F(\textit{X})}$
\end{proof}

Actually, Khovratovich and Nikoli{\'c} even proof that all functions can be realized by
only $AR(n)$. This is proofed by the fact that xor can be realized with $AR$ operations.
The reader may refer \cite{khovratovich2010rotational} for the complete proof of this
statement.


\section{Integrals of VVFs}
VVFs can also be integrated. The integration operation also distributes inside the vector, for both definite and indefinite integrals.
\begin{equation*}
	\int{\vec{r}(t)\mathrm{d}t} = \left<\int{x(t)\mathrm{d}t},\int{y(t)\mathrm{d}t},\int{z(t)\mathrm{d}t}\right>
\end{equation*}
For indefinite integrals, the result will have a vector of constants will be added.


\chapter{Encryption}

\epigraph{Any one who considers arithmetical methods of producing random digits is, of course, in a state of sin.}{\textit{John von Neumann}}

When information is being transmitted, it is usually possible for the message to be read by people other than the intended recipient. The idea of encryption is to change the message in such a way that only a particular recipient will be able to recover the original message. This is achieved by altering a message using some kind of per-arranged secret information that has only been shared with the recipient you want to be able to read the message.\\

The original message is called the plaintext message. The encrypted message is called the ciphertext. And the secret information is called the key.\\

\section{One-Time Pad}

The first such algorithm we will look at is what is called a one-time pad. A one-time pad is a unique reference material that contains purely random values, which acts as the secret key. The first issue is how to get truly random anything, but assuming you have it you then make an exact copy of the one-time pad and share it with the person you wish to be able to send a message (and noone else). It is important that noone else be able to see this pad (the key), because anyone who has it will be able to read the message. So it must be shared in a known secure manner, such as in person. It might look something like the following.\\

\begin{center}
	\begin{tabular}{c | c | c | c }
		 \texttt{zyhdjkxs} & \texttt{noujnbpx} & \texttt{erdtgnhb} & \texttt{txqursil}\\
		 \texttt{armlijnm} & \texttt{wkrcsjus} & \texttt{xcmyrzfj} & \texttt{iygoosom}\\
		 \texttt{woqzafpi} & \texttt{neqxbysh} & \texttt{afpvoxww} & \texttt{kvepxkwu}\\
		 \texttt{boxllkib} & \texttt{tiohhxsg} & \texttt{zbylmrgp} & \texttt{ipuilvvb}\\
		 \texttt{xedpjdqx} & \texttt{yiyrrndi} & \texttt{bbqptpmy} & \texttt{oxstqgrn}\\
	\end{tabular}
\end{center}

At some later time you can encrypt a message using the values on the one-time pad. Basically, each letter is assigned a number value. One random numeric value is added to one plaintext numeric value, and taking the modulus of the maximum number of possibilities. The result of this operation is the ciphertext.\\

\begin{center}

	\begin{tabular}{c | c }

		a = 0 & n = 13 \\
		b = 1 & o = 14 \\
		c = 2 & p = 15 \\
		d = 3 & q = 16 \\
		e = 4 & r = 17 \\
		f = 5 & s = 18 \\
		g = 6 & t = 19 \\
		h = 7 & u = 20 \\
		i = 8 & v = 21 \\
		j = 9 & w = 22 \\
		k = 10 & x = 23 \\
		l = 11 & y = 24 \\
		m = 12 & z = 25 \\
	\end{tabular}
\end{center}

Suppose I want to send the message "Meet me at noon", this is the plaintext and has 12 letters (ignoring the spaces). I have already shared the above one-time pad with the person that I want to read the message. Since the message has 12 letters, I need to use 12 random values from the one-time.\\


\section{Partial Derivatives}
\noindent
The single-variable calculus idea of tangent lines doesn't work for higher dimensional surfaces because we can draw many different lines that are tangent to the surface, depending on which plane we use to slice the surface. That is, from which direction we approach the surface.
\input{./differentialMultivariableCalculus/partialDerivativesXYZ}
\input{./differentialMultivariableCalculus/tangentPlanes}
\input{./differentialMultivariableCalculus/linearApproximations}


\section{More about categories}
%moved defn of Fun(C,D) to prev. section
%moved split epi/mono into single definition
%the example on splitting changed to an exercise, since it was leaving stuff to the reader anyway
Let $\mathrm{Vect}_{\mathbf{C}}$ be the category of $\mathbf{C}$-vector spaces with $\mathbf{C}$-linear transformations. Given such a vector space $V$, taking the dual gives another vector space $V^\ast=\Hom(V,\mathbf{C})$, and a linear transformation $f:V\to W$ dualizes to $f^\ast:W^\ast\to V^\ast$. This process resembles a functor $\mathrm{Vect}_{\mathbf{C}} \to \mathrm{Vect}_{\mathbf{C}}$, except that it reverses the direction of morphisms.
\begin{definition}
Let $\cc$ be a category. Define its \emph{opposite category} $\cc^{op}$ to have the same objects as $\cc$ but with morphisms reversed, so that for all $X,Y\in\mathrm{ob}(\cc)$, we have $\cc^{op}(X,Y) = \cc(Y,X)$. Composition in $\cc^{op}$ is the same as in $\cc$.

If $\cd$ is another category, then a \emph{contravariant functor} $\cc\to \cd$ is an ordinary (or \emph{covariant}) functor $\cc^{op}\to\cd$.
\end{definition}
Let $\cc$ be a category, and let $Y\in\mathrm{ob}(\cc)$. Consider the functor $\cc^{op}\to\set$ that takes $X\in\mathrm{ob}(\cc)$ to the set $\cc(X,Y)$, and takes a map $f\colon X\to W$ to the map $-\circ f\colon \cc(W,Y)\to \cc(X,Y)$ which is precomposition by $f$. This functor, denoted $\cc(-,Y)$, is called the functor \emph{represented by $Y$}. Similarly, there is a functor $\cc(-,Y)$ which is called the functor \emph{corepresented by $Y$}. Note that $\cc(-,Y)$ is contravariant and $\cc(Y,-)$ is covariant.

Recall that $\Deltab$ has objects $[0],[1],[2],\cdots$ and there is a functor $\Deltab\to\mathbf{Top}$ that sends $[n]\mapsto\Delta^n$ (see Exercise \ref{exer:simplicialidentities}). Let $X$ be a space, and consider the functor $\mathbf{Top}^{op}\to\set$ represented by $X$. Composing these functors gives a functor $\Delta^{op}\to\set$ which sends $[n]\mapsto\mathbf{Top}(\Delta^n,X)=:\Sin_n(X)$. This is precisely the singular simplicial set of $X$.
\begin{prop}
Simplicial sets are precisely functors $\Deltab^{op}\to\set$, i.e., $s\set=\mathbf{Fun}(\Deltab^{op},\set)$. More generally, simplicial objects in a category $\cc$ are functors $\Deltab^{op}\to\cc$, i.e., $s\cc=\mathbf{Fun}(\Deltab^{op},\cc)$.
\end{prop}
If $\Deltab_{inj}$ is the subcategory of $\Deltab$ whose morphisms are only the injective maps, then $ss\cc:=\mathbf{Fun}(\Deltab^{op}_{inj},\cc)$ is the category of semi-simplicial objects in $\cc$ (which differ from simplicial objects in that they only have face maps).
\begin{definition}
Let $X,Y\in \cc$. We say that a morphism $f\colon X\rightarrow Y$ is a \emph{split epimorphism} if there exists $g:Y\to X$ (often called a section or a splitting) such that $Y\xrightarrow{g}X\xrightarrow{f}Y$ is the identity $1_Y$.

We say that a morphism $g\colon Y\rightarrow X$ is a \emph{split monomorphism} if there exists $f\colon X\to Y$ such that $Y\xrightarrow{g}X\xrightarrow{f}Y$ is the identity $1_Y$.
\end{definition}
\begin{example}
Let $\cc=\set$. If $f:X\to Y$ is a split epimorphism with $f\circ g = 1_Y$, then for every $y\in Y$, we have $f(g(y))=y$ and thus $f$ is surjective. Is every surjective map a split epimorphism? Constructing a splitting $g$ amounts to picking $g(y)\in f^{-1}(y)$ for all $y\in Y$, so this reduces to the axiom of choice.

Now assume that $g: X\to Y$ is a split monomorphism. Suppose that $y,y^\prime\in Y$ are such that $g(y)=g(y^\prime)$. Applying $f$ gives $y=y^\prime$, and thus split monomorphisms are injective. Conversely, if $Y$ is nonempty, then every injection $g:Y\to X$ is a split monomorphism.
\end{example}
\begin{example}
A morphism is an isomorphism if and only if it is both a split epi and a split mono.
\end{example}
\begin{lemma}
If $f:X\to Y$ is a split epi (resp. split mono) in $\cc$, and $F:\cc\to \cd$, then $F(f)$ is a split epi (resp. split mono) in $\cd$.
\end{lemma}
\begin{proof}
If $g$ splits $f$ in $\cc$, then $F(g)$ splits $F(f)$ in $\cd$.
\end{proof}
\begin{exercise}\label{ex:split-epi-mono-isomorphism}
Suppose $A,B\in \cc=\mathbf{Ab}$ and $f:A\to B$ is a split epi, so that $fg=1$ for some $g\colon B\rightarrow A$. Let $i\colon \ker f\to A$ be the inclusion, and consider the sum $\ker f\oplus B\xrightarrow{[i,g]} A$. Show that $[i,g]$ is an isomorphism.

If $g:B\to A$ is a split mono, there exists $f:A\to B$ so that $fg=1$. Let $p\colon A\rightarrow \mathrm{coker}(g)$ be the quotient map, and consider the map $A\xrightarrow{\begin{pmatrix}
p \\ f
\end{pmatrix}}\mathrm{coker}(g)\oplus B$. Show that this is an isomorphism.
\end{exercise}
We have to get into some topology, since it's on our agenda. In the category $\mathbf{Top}$, the one-point space $\ast$ is terminal, meaning that for any space $X$ there is a unique map $X\to\ast$. This induces a map
\[H_n(X)\to H_n(\ast)=\begin{cases}\mathbf{Z} & n=0\\
0 & \text{else}\end{cases}\]
which is called the \emph{augmentation map}.

Take a 0-cycle $\sum a_ix_i$ where the $x_i$ are points in $X$. Consider its homology class $\left[\sum a_ix_i\right]\in H_0(X)$. Under the induced map above, this is sent to $\left[\sum a_i\ast\right]=\left(\sum a_i\right)\left[\ast\right]$.
\begin{definition}
A \emph{pointed space} is a pair $(X,\ast)$, with $\ast\in X$ called the \emph{basepoint}.
\end{definition}
Let $\ast\to X\to\ast$ be the inclusion of the basepoint followed by the unique map to $\ast$. In homology we get a map $\mathbf{Z}\xrightarrow{\eta} H_\ast(X)\xrightarrow{\epsilon}\mathbf{Z}$, where the composition is the identity. The map $\epsilon$ is the augmentation map described above, so we see that it is a split epimorphism. This means that, by Exercise \ref{ex:split-epi-mono-isomorphism}, $ H_\ast(X)\cong \mathbf{Z}\oplus\mathrm{coker}\eta \cong\mathbf{Z}\oplus\ker\epsilon$. The \emph{reduced homology} of $(X,\ast)$ is $ H_\ast(X,\ast)=\mathrm{coker}\eta$. It's isomorphic to $ H_\ast(X)$ in dimensions greater than $0$, but differs by a factor of $\mathbf{Z}$ in dimension $0$.


\section{Inner Product Space}

\subsection{Inner Product and Norm}

\begin{definition}
	An \cindex{inner product} on $V$ is a function $V \rightarrow V \rightarrow F$ ($F$ is either $C$ or $R$) that $\forall x,y,z \in V$ and $\forall c \in F$ that:
	\begin{enumerate}
		\item $\innerproduct{x+z}{y} = \innerproduct{x}{y} + \innerproduct{z}{y}$ \label{firstproductdefinition}
		\item $\innerproduct{cx}{y} = c \innerproduct{x}{y}$ \label{secondproductdefinition}
		\item $\overline{\innerproduct{x}{y}} = \innerproduct{y}{x}$
		\item $\innerproduct{x}{x} > 0$ if $x \neq 0$
	\end{enumerate}
	Item (\ref{firstproductdefinition}) and (\ref{secondproductdefinition}) means the inner product is \emph{linear in first component}.
	Please be noted that the result of inner product could be a complex value, but the result of $\innerproduct{x}{x}$ is a non-negative real number.
	\qed
\end{definition}

\begin{theorem}
	properties of inner product:
	\begin{enumerate}
		\item $\innerproduct{x}{y+z} = \innerproduct{x}{y} + \innerproduct{x}{z}$ \label{firstproductproperty}
		\item $\innerproduct{x}{cy} = \overline{c} \innerproduct{x}{y} $ \label{secondproductproperty}
		\item $\innerproduct{x}{x} = 0$ if and only if $x = 0$.
		\item If $\innerproduct{x}{y} = \innerproduct{x}{z}$ for all $x \in V$, then $y=z$.
	\end{enumerate}
	Item (\ref{firstproductproperty}) and (\ref{secondproductproperty}) means the inner product is \cindex{conjugate linear} in second component.
\end{theorem}

\begin{definition}
	the \cindex{standard inner product} on $F^n$ for $x=\rowvector{a_1,a_2,\dots,a_n}$ and $y=\rowvector{b_1,b_2,\dots,b_n}$ is:
	\begin{equation}
		\innerproduct{x}{y} = \sum_{i=1}^n a_i \overline{b_i}
	\end{equation}
	when $F=R$, it is usually called \cindex{dot product} and denoted as $x \cdot y$.
\end{definition}

\begin{definition}
	For $A \in M_{m \times n}(F)$, the \cindex{conjugate transpose} or \cindex{adjoint} of $A$ is $A^* \in M_{n \times m}(F)$ that $(A^*)_{ij} = \overline{A_{ji}}$. If $A$ is complex, $A^* = \overline{A^\top}$ . If $A$ is real, $A^*$ is $A^\top$.
\end{definition}

\begin{definition}[Forbenius Inner Product]
    Let $V=M_{n \times n} (F)$, the \cindex{Forbenius Inner Product} is defined as:
    \begin{equation}
        \innerproduct{A}{B} = \trace{B^* A}
    \end{equation}
\end{definition}

\begin{theorem}
    For square matrix $A_{n \times n}$, we have
    \begin{equation}
        \innerproduct{A}{A} = \sum_{i=1}^n \sum_{j=1}^n \absolutevalue{A_{ij}}^2 \geq 0
    \end{equation}
\end{theorem}

\begin{definition}\label{hinnerproductspace}
	The continuous complex-valued function on interval $[0, 2\pi]$ is a inner product space $H$:
	\begin{equation}
		\innerproduct{f}{g} = \frac{1}{2\pi} \int_{0}^{2\pi} f(t) \overline{g(t)} dt
	\end{equation}
\end{definition}

\begin{definition}
	the \cindex{norm} or \cindex{length} of $x$ is:
	\begin{equation}
	    \norm{x} = \sqrt{\innerproduct{x}{x}}
	\end{equation}
\end{definition}

\begin{theorem}
	the property of norm:
	\begin{itemize}
		\item $\norm{cx} = \absolutevalue{c} \cdot \norm{x}$
		\item $\norm{x} = 0 \iff x = 0$
		\item \cindex{Cauchy-Schwarz Inequality} $\absolutevalue{\innerproduct{x}{y}} \leq \norm{x} \cdot \norm{y}$
		\item \cindex{Triangle Inequality} $\norm{x+y} \leq \norm{x} + \norm{y}$
	\end{itemize}
\end{theorem}

\begin{theorem}\label{zerotforalltx}
    If $\forall x \in C$,$\innerproduct{T(x)}{x} = 0$. Then $T = 0$.\footnote{For it to work in all $V$, $T$ needs to be self-adjoint. See \thmref{zerotforalltxforselfadjoint} on page \pageref{zerotforalltxforselfadjoint}.}
\end{theorem}
\begin{proof}
    \begin{equation*}
        \begin{aligned}
            \innerproduct{T(x+y)}{x+y} &= \innerproduct{T(x)}{y} + \innerproduct{T(y)}{x}  &= 0 \\
            \innerproduct{T(x+iy)}{x+iy} &= \innerproduct{T(x)}{y} -\innerproduct{T(y)}{x} &= 0
        \end{aligned}
    \end{equation*}
    So $\forall y \in V$, $T(x) = 0$. So $\forall x \in V$,$T(x) = 0$ and $T = 0$.
\end{proof}

\begin{theorem}
    \begin{equation}
        \norm{u+v}^2 + \norm{u-v}^2 = 2 \left(\norm{u}^2 + \norm{v}^2 \right)
    \end{equation}
\end{theorem}

% orthogonal

\subsection{Orthogonal and Gram-Schmidt Process}

\begin{definition}
	$x$ and $y$ are \cindex{orthogonal} if $\innerproduct{x}{y} = 0$. A subset $S$ of $V$ is orthogonal if any two vectors in $S$ are orthogonal. A subset $S$ of $V$ is \cindex{orthonormal} if $S$ is orthogonal and consists entirely of unit vectors.
\end{definition}

\begin{definition}
    \begin{equation}
        \innerproduct{x}{y} = \norm{x} \cdot \norm{y} \text{cos}(\theta)
    \end{equation}
\end{definition}

\begin{definition}
	A vector is \cindex{unit vector} if $\norm{x} = 1$. A \cindex{normalizing} to non-zero $x$ is $\dfrac{1}{\norm{x}} x$.
\end{definition}

\begin{theorem}
    Let $f_n (t) = e^{i nt}$ where $0 \leq t \leq 2 \pi$. All $f_i$ are orthogonal.
\end{theorem}
\begin{proof}
    \begin{equation}
        \begin{aligned}
            \innerproduct{f_m}{f_n} &= \frac{1}{2 \pi} \int_0^{2 \pi} e^{imt} \overline{e^{int}} \dif{t} \\
            &= \frac{1}{2\pi} \int_0^{2\pi} e^{i (m-n) t} \dif{t} \\
            &= \eval{\frac{1}{2\pi (m-n)} e^{i(m-n)t}}_0^{2\pi} \\
            &= 0
        \end{aligned}
    \end{equation}
\end{proof}

\begin{theorem}[\cindex{Pythagorean Theorem}]
    Suppose $u$ and $v$ are orthogonal in $V$, then
    \begin{equation}
        \norm{u + v}^2 = \norm{u}^2 + \norm{v}^2
    \end{equation}
\end{theorem}

\begin{theorem}
    For a finite dimensional subspace $U$ of $V$, we have
    \begin{equation}
        V = U \oplus U^\bot
    \end{equation}
\end{theorem}

\begin{definition}
	A \cindex{orthonormal basis} for $V$ is an ordered basis that is orthonormal.
\end{definition}

\begin{theorem}
	Let $S=\set{ v_1, v_2, \dots, v_k }$ be an orthogonal subset of $V$ consisting of non-zero vectors. If $y \in \vectorspan{S}$, then
	\begin{equation}
		y = \sum_{i=1}^k \frac{\innerproduct{y}{v_i}}{\norm{v_i}^2} v_i
	\end{equation}
	Define the projection of vector $a$ onto vector $u$ as $\projection{a}{u} = \dfrac{\innerproduct{a}{u}}{\norm{u}^2}$. So
	\begin{equation}
	    y = \sum_{i=1}^k \left(\projection{y}{v_i}\right) v_i
	\end{equation}
	If $S$ is orthonormal, then
	\begin{equation}
		y = \sum_{i=1}^k \innerproduct{y}{v_i} v_i
	\end{equation}
\end{theorem}
\begin{proof}
	let $\displaystyle y = \sum_{i=1}^k a_i v_i$. we have
	\begin{equation*}
		\innerproduct{y}{v_i} = \innerproduct{\sum_{i=1}^k a_i v_i}{v_j} = \sum_{i=1}^k a_i \innerproduct{v_i}{v_j} = a_j \norm{v_j}^2
	\end{equation*}
	So $a_j = \dfrac{\innerproduct{y}{v_j}}{\norm{v_j}^2}$.

\end{proof}

\begin{theorem}
	An orthogonal subset of $V$ is linearly independent.
\end{theorem}

\begin{definition}[\cindex{Gram-Schmidt process}]
	Let $S=\set{w_1, w_2, \dots, w_n }$ be linearly independent subset of $V$. Define $S^\prime=\{v_1,v_2,\dots,v_n  \}$, where $v_1=w_1$ and
	\begin{equation}
		v_k = w_k - \sum_{j=1}^{k-1} \frac{\innerproduct{w_k}{v_j}}{\norm{v_j}^2} v_j
	\end{equation}
	then $S^\prime$ is an orthogonal set of non-zero vectors that $\vectorspan{S^\prime} = \vectorspan{S}$. The process is that for the $k$-th basis $w_k$, first project it on top of the $k-1$ orthogonal vectors $\displaystyle \sum_{j=1}^{k-1} \dfrac{\innerproduct{w_k}{v_j}}{\norm{v_j}^2} v_j$, and calculate the reciprocal vector $\displaystyle w_k - \sum_{j=1}^{k-1} \dfrac{\innerproduct{w_k}{v_j}}{\norm{v_j}^2} v_j$.
	\qed
\end{definition}

\begin{theorem}[\cindex{QR Decomposition}]
    Let $A_{m \times n} = \rowvector{a_1, a_2, \dots, a_n}$ with $\rank{A} = n$, so $\set{a_i}$ is linearly independent. Use Gram-Schmidt process to form $n$ orthonomal basis:
    \begin{equation*}
        \begin{aligned}
            u_1 &= a_1 & \text{ , } e_1 = \frac{u_1}{\norm{u_1}} \\
            u_2 &= a_2 - \projection{a_2}{u_1} & \text{ , } e_2 = \frac{u_2}{\norm{u_2}} \\
            \dots \\
            u_n &= a_n - \sum_{j=1}^{n-1} \projection{a_n}{u_j} & \text{ , } e_n = \frac{u_n}{\norm{u_n}}
        \end{aligned}
    \end{equation*}
    Then $\forall k$, $\displaystyle a_k = \sum_{j=1}^k \innerproduct{a_k}{e_k} e_k$. So
    \begin{equation}
        A = QR = [e_1, e_2, \dots, e_n] \times \begin{bmatrix}
            \innerproduct{a_1}{e_1} & \innerproduct{a_2}{e_1} & \innerproduct{a_3}{e_1} & \cdots & \innerproduct{a_n}{e_1}\\
            0 & \innerproduct{a_2}{e_2} & \innerproduct{a_3}{e_2} & \cdots & \innerproduct{a_n}{e_2}\\
            0 & 0 & \innerproduct{a_3}{e_3} & \cdots & \innerproduct{a_n}{e_3} \\
            \vdots & \vdots & \vdots & \ddots & \vdots \\
            0 & 0 & 0 & 0 & \innerproduct{a_n}{e_n}
        \end{bmatrix}
    \end{equation}

    The $Q$ is an orthonormal matrix. $R$ could be calculated by:
    \begin{equation}
        R = Q^\top Q R = Q^\top A
    \end{equation}
\end{theorem}

\begin{theorem}\label{vectorinorthonormalbasis}
	If $V$ has an orthonormal basis $\beta=\set{v_1,v_2,\dots,v_n}$, then $\forall x\in V$,
	\begin{equation}
		x = \sum_{i=1}^n \innerproduct{x}{v_j} v_i
	\end{equation}
\end{theorem}

\begin{definition}
    Let $\beta$ be an orthonormal subset (not basis) of $V$. For $x \in V$, the \cindex{Fourier coefficients} of $x$ relative to $\beta$ are $\innerproduct{x}{y_i}$ for all $y_i \in \beta$.
\end{definition}

\begin{theorem}\label{matrixelementasinnerproductresult}
	Let $V$ with an orthonormal basis $\beta=\set{v_1,v_2,\dots,v_n}$. $T$ is a linear operator on $V$ and let $A= \coordinate{T}_\beta$. then $A_{ij}=\innerproduct{T(v_j)}{v_i}$.
\end{theorem}
\begin{proof}
	From \thmref{vectorinorthonormalbasis} we have
	\begin{equation*}
		T(v_j) = \sum_{i=1}^n \innerproduct{T(v_j)}{v_i} v_i
	\end{equation*}
\end{proof}

\begin{definition}
    Let $S$ be nonempty subset of $V$. The \cindex{orthogonal complement} of $S$ is $S^\bot$ that $\forall x \in S, \forall y \in S^\bot, \innerproduct{x}{y} = 0$.
\end{definition}

\begin{theorem}\label{orthogonalprojection}
    Let $W$ be a subspace of $V$. For $y \in V$, there is \emph{unique} $u \in W$ and $z \in W^\bot$ that $y = u + z$. $u$ is the \cindex{orthogonal projection} of $y$ on $W$. If $\set{v_1, v_2, \dots, v_k}$ is an orthonormal basis of $W$, then
    \begin{equation}
        \begin{aligned}
            u &= \sum_{i=1}^k \innerproduct{y}{v_i} v_i \\
            z &= y -   \sum_{i=1}^k \innerproduct{y}{v_i} v_i
        \end{aligned}
    \end{equation}
\end{theorem}

\begin{theorem}
    For $S=\set{v_1, v_2, \dots, v_k}$ be an orthogonal subset of $V$. For $\forall y \in V$, the orthogonal projection of $y$ on $S$ is $\displaystyle u = \sum_{i=1}^k \dfrac{\innerproduct{y}{v_i}}{\norm{v_i}^2} v_i$. If $S$ are orthonormal, $\displaystyle u = \sum_{i=1}^k \innerproduct{y}{v_i} v_i$. If $y$ is in span of $S$, then $y = u$.
\end{theorem}

\begin{theorem}
    Let $y$,$u$,$z$ as defined in \thmref{orthogonalprojection}. $u$ is the closest vector in $W$ to $y$ that is $\forall x \in W \left(\norm{y-x} \geq \norm{y - u} \right)$.
\end{theorem}
\begin{proof}
    \begin{equation*}
        \norm{y - x}^2 = \norm{u + z - x}^2 = \norm{(u - x) + z}^2 = \norm{u - x}^2 + \norm{z}^2 \geq \norm{z}^2 = \norm{y - u}^2
    \end{equation*}
\end{proof}

% Adjoint of Linear Operator
\subsection{Adjoint of Linear Operator}

\begin{theorem}[\cindex{Riesz Representation Theorem}]\label{uniquelinearoperatortof}
    Let $g: V \rightarrow F$ be a linear transformation. Then there exist a unique $y \in V$ that $\forall x \in V$, $g(x) = \innerproduct{x}{y}$. The $y$ is
    \begin{equation}
        y = \sum_{i=1}^n \overline{g(v_i)} v_i
    \end{equation}

    So every vector in the dual space\footnote{Defined in \thmref{dualspacedefinition} on page \pageref{dualspacedefinition}.} can be represented by an inner product.
\end{theorem}
\begin{proof}
    Define $h(x) = \innerproduct{x}{y}$ with $y$ defined above. So
    \begin{equation*}
        h(v_j) = \innerproduct{v_j}{y} = \innerproduct{v_j}{\sum_{i=1}^n \overline{g(v_i)} v_i} = \sum_{i=1}^n \innerproduct{v_j}{\overline{g(v_i)} v_i} = \sum_{i=1}^n g(v_i) \innerproduct{v_j}{ v_i} = g(v_j)
    \end{equation*}
\end{proof}

\begin{theorem}
    Let $T$ be a linear operator on $V$. Then there existing a unique linear operator $T^* : V \rightarrow V$ that $\innerproduct{T(x)}{y}=\innerproduct{x}{T^*(y)}$ for all $x,y \in V$. $T^*$ is called the \cindex{adjoint} of $T$.
\end{theorem}
\begin{proof}
    For each $y$, $\innerproduct{T(x)}{y}$ is a linear operator from $V$ to $F$, so by \thmref{uniquelinearoperatortof}, $\exists y'$ that $\innerproduct{T(x)}{y} = \innerproduct{x}{y'}$. Define $T^*$ as $T^*(y) = y'$.
\end{proof}

\begin{theorem}
    \begin{equation}
        \begin{aligned}
            \innerproduct{T(x)}{y} &= \innerproduct{x}{T^*(y)} \\
            \innerproduct{x}{T(y)} &= \innerproduct{T^*(x)}{y}
        \end{aligned}
    \end{equation}
    So $^*$ is added to $T$ when change the location of $T$.
\end{theorem}
\begin{proof}
    \begin{equation*}
        \innerproduct{x}{T(y)} = \overline{\innerproduct{T(y)}{x}} = \overline{\innerproduct{y}{T^*(x)}} = \innerproduct{T^*(x)}{y}
    \end{equation*}
\end{proof}

\begin{theorem}
    Let $\beta$ be a orthonormal basis for $V$. If $T$ is a linear operation on $V$ then
    \begin{equation}
        [T^*]_\beta = \left([T]_\beta \right)^*
    \end{equation}
    Let $A$ be an $n \times n$ matrix. Then
    \begin{equation}
        L_{A^*} = \left(L_A \right)^*
    \end{equation}
\end{theorem}
\begin{proof}
    Let $A=[T]_\beta$, $B=[T^*]_\beta$, and $\beta=\{v_1, v_2, \dots, v_n \}$. Then according to \thmref{matrixelementasinnerproductresult}:
    \begin{equation*}
        B_{ij} = \innerproduct{T^*(v_j)}{v_i} = \overline{\innerproduct{v_i}{T^*(v_j)}} = \overline{\innerproduct{T(v_i)}{v_j}} = \overline{A_{ji}} = (A^*)_{ij}
    \end{equation*}
\end{proof}

\begin{theorem}
    Let $T$ and $U$ be linear operator on $V$, then
    \begin{enumerate}
        \item $(aT+bU)^* = \overline{a} T^* + \overline{b} U^*$
        \item $(UT)^* = T^* U^*$
        \item $T^{**} = T$
    \end{enumerate}
\end{theorem}

\begin{definition}
    Let $T : V \rightarrow W$ be a linear transformation where $V$ and $W$ are finite dimensional inner product space with inner product $\innerproduct{\cdot{}}{\cdot{}}_V$ and $\innerproduct{\cdot{}}{\cdot{}}_W$. A function $T^* : W \rightarrow V$ is called \cindex{adjoint} of $T$ if $\innerproduct{T(x)}{y}_W = \innerproduct{x}{T^*(y)}_V$.
\end{definition}

\begin{theorem}
    Let $T^*$ be an adjoint of $T: V \rightarrow W$. If $\beta$ and $\gamma$ are orthonormal basis for $V$ and $W$, then
    \begin{equation}
        [T^*]_\beta^\alpha = \left([T]_\beta^\alpha\right)^*
    \end{equation}
\end{theorem}

\begin{theorem}
    Let $T^*$ be an adjoint of $T: V \rightarrow W$, we have:
    \begin{equation}
        \innerproduct{T^*(x)}{y}_V = \innerproduct{x}{T(y)}_W
    \end{equation}
\end{theorem}

\begin{theorem}\label{nullandreciprocaladjoint}
    If $V$ is finite dimentional, let $T$ be a linear operator on $V$, then
    \begin{equation*}
        \begin{aligned}
            \rangespace{T^*}^\bot &= \nullspace{T}\\
            \rangespace{T^*} &= \nullspace{T}^\bot \\
            \rangespace{T}^\bot &= \nullspace{T^*}\\
            \rangespace{T} &= \nullspace{T^*}^\bot
        \end{aligned}
    \end{equation*}
    So $\rangespace{T^*} \bot \nullspace{T}$.
\end{theorem}
\begin{proof}
    If $m \in R(T^*)^\bot$, $\forall x \in V$, $0 = \innerproduct{m}{T^*x} = \innerproduct{T(m)}{x}$, so $m \in N(T)$.
\end{proof}

% Example in statistics
\subsection{Examples in Statistics}\label{consistentandinconsistentequation}

The following two examples show that for linear equation $Ax - y = 0$,
\begin{enumerate}
    \item if it is consistent, that is there is solution, we want to find the solution with minimal norm.
    \item If it is inconsistent, that is no solution, we want a result that has the least norm.
\end{enumerate}

The same topic is discussed in pseudo inverse.

% Least Square Approximation
\subsubsection{Least Square Approximation}

\begin{definition}
    The \cindex{Least Square Approximation} is a problem that for $A = \begin{bmatrix}
        t_1 & 1 \\
        t_2 & 1 \\
        \vdots & \vdots \\
        t_m & 1
    \end{bmatrix}$,  $y = \begin{bmatrix}
        y_1\\
        y_2\\
        \vdots\\
        y_m
    \end{bmatrix}$, find $x_0 = \begin{bmatrix}
        c \\
        d
    \end{bmatrix}$ that minimize $\norm{Ax-y}$.
\end{definition}

\begin{definition}
    For $x,y \in F^n$, define $\innerproduct{x}{y}_n = y^* \times x$.
\end{definition}

\begin{theorem}
    Let $A \in M_{m \times n} (F)$, $x \in F^n$, $y\in F^m$, then
    \begin{equation}
        \innerproduct{Ax}{y}_m = \innerproduct{x}{A^* y}_n
    \end{equation}
\end{theorem}
\begin{proof}
    $\innerproduct{Ax}{y}_m = y^* \times (Ax) = (y^* \times A) x = (A^* y)^* x = \innerproduct{x}{A^* y}_n$
\end{proof}

\begin{theorem}
    Let $A \in M_{m\times n} (F)$. Then\footnote{See \thmref{rankofadjoint} for another proof.}
    \begin{equation}
        \rank{A^*A} = \rank{A}
    \end{equation}
    So if $\rank{A} = n$, $A^*A$ is invertible.
\end{theorem}
\begin{proof}
    For equation $A^*Ax = 0$ and $Ax = 0$. $Ax=0$ implies that $A^*Ax =0$. Then assume $A^*Ax = 0$, then
    \begin{equation*}
        0 = \innerproduct{0}{x}_n = \innerproduct{A^*Ax}{x}_n = \innerproduct{Ax}{A^{**}x}_m = \innerproduct{Ax}{Ax}_m
    \end{equation*}
\end{proof}

\begin{theorem}
    Let $A \in M_{m\times n} (F)$, $y \in F^m$. Then there exists $x_0 \in F^n$ that $(A^*A) x_0 = A^* y$ and $\forall x \in F^n$, $ \norm{Ax_0 - y} \leq \norm{Ax-y}$. If $\rank{A} = n$, then $x_0 = (A^*A)^{-1} A^* y$.
\end{theorem}
\begin{proof}
    Define $W=\rangespace{L_A}$. There exists a $x_0$ that is closest to $y$ that $Ax_0 - y \in W^\bot$, so $\innerproduct{Ax}{Ax_0 - y}_m = 0$. So $\innerproduct{x}{A^*(Ax_0 - y)}_n = 0$, so $A^*(Ax_0 - y) = 0$ and $(A^*A) x_0 = A^* y$.
\end{proof}

% Minimal Solution to Linear Equations
\subsubsection{Minimal Solution to Linear Equations}

\begin{definition}
    A solution $s$ is \cindex{minimal solution} of $Ax=b$ if $\norm{s} \leq \norm{u}$ for any solution $u$.
\end{definition}

\begin{theorem}
    Let $A \in M_{m\times n} (F)$, $y \in F^m$. Suppose $Ax=y$ is consistent. Then there exists unique minimal solution $s \in R(L_{A^*})$ of $Ax=y$. And $s$ is the only solution in $R(L_{A^*})$. If $u$ is a solution to $(AA^*) u = y$, then $s = A^* u$.
\end{theorem}
\begin{proof}
    By \thmref{nullandreciprocaladjoint} define $W = R(L_{A^*})$ and $W^\bot = N(L_A)$. $\forall x$ that $Ax = y$, we have $s \in W$ and $t \in W^\bot$ that $x=s+t$. So $y = Ax = A(s + t) = As + At = As$. So $s$ is a solution to $Ax=y$. From \thmref{equationfromoneandnullspace}, all solution to $Ax=y$ has the form $x' = s + t'$ where $t' \in W^\bot$. And $\norm{x'}^2 = \norm{s + t'}^2 = \norm{s}^2 + \norm{t'}^2 \geq \norm{s}^2$.
\end{proof}


\documentclass{article}
\title{ALO for Elastic Net with Intercept through Generalized LASSO}
\author{Yuze Zhou}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\begin{document}
\section{Dual Formulation of Elastic Net with Intercept}
\paragraph{}First, to write the optimization problem of elastic net in a matrix form, and denote $D =[0, I]$, the optimization problem becomes:
\begin{center}
$\hat{\beta} = \arg \min\limits_{\beta} \frac{1}{2}||y-X\beta||_{2}^{2} + \lambda_{1}||D\beta||_{1}+\lambda_{2}||D\beta||_{2}^{2}$
\end{center}
\paragraph{}The augmented Lagrangian for the problem is:
\begin{center}
$L = \frac{1}{2}||y-z||_{2}^{2} + \lambda_{1}||\omega||_{1}+\lambda_{2}||\omega||_{2}^{2} + u^{\tau}(z-X\beta) + v^{\tau}(\omega-D\beta)$
\end{center}
\paragraph{}By taking the derivatives with respect to $z$ and $\beta$, we could obtain:
\begin{center}
$0 = \frac{\partial L}{\partial z} = z-y+u$\\
$0 = \frac{\partial L}{\partial \beta} = -X^{\tau}u-D^{\tau}v$
\end{center}
\paragraph{}Since the first column of $X$ is filled with ones and the first column of $D$ is filled with zeros, the first row of $-X^{\tau}u-D^{\tau}v = 0$ gives $\textbf{1}^{\tau} u = 0$ and due to $D = [0,I]$, the rest rows give that $-X_{j}^{\tau}u = v_{j}$. Moreover, since the rest dimensions of $\omega$ is penalized element-wisely in the augmented Lagrangian, we can minimize over $\omega$ by minimizing over each $\omega_{i}, \quad i \geq 2$, that is, we have to minimize $\lambda_{1}|\omega_{i}| + \lambda_{2}\omega_{i}^{2} - u^{\tau}X_{i}\omega_{i}$ for each dimension of $\omega$, where $X_{i}$ denotes the $i$th column of $X$, therefore:
\begin{center}
$\min\limits_{\omega_{i}} \lambda_{1}|\omega_{i}| + \lambda_{2}\omega_{i}^{2} - u^{\tau}X_{i}\omega_{i} $\\
$ $\\
$= \left\{
\begin{aligned}
0 \quad if \quad |u^{\tau}X_{i}| \leq \lambda_{1}\\
-\frac{(\lambda_{1}-|u^{\tau}X_{i}|)^{2}}{4\lambda_{2}} \quad if \quad |u^{\tau}X_{i}| > \lambda_{1}\\
\end{aligned}
\right.
$
\end{center}
\paragraph{}By taking all the above back to the Lagrangian, we obtain the dual problem as:
\begin{center}
$d^{*} = \min\limits_{u} \frac{1}{2}||y-u||_{2}^{2} + \sum\limits_{j: |X_{j}^{\tau}u| > \lambda_{1}}\frac{(\lambda_{1}-|u^{\tau}X_{i}|)^{2}}{4\lambda_{2}}$\\
$subject \quad to \quad \textbf{1}^{\tau}u = 0$
\end{center}
\paragraph{}First denote $f(u) = \sum\limits_{j: |X_{j}^{\tau}u| > \lambda_{1}}\frac{(\lambda_{1}-|u^{\tau}X_{i}|)^{2}}{4\lambda_{2}}$, clearly $f(u)$ is of quadratic form, $f(u) = \frac{1}{2}u^{\tau}Au+a^{\tau}u+b$, where $b$ is a constant and does not matter in the optimization of the dual problem, $A$ and $a$ are:
\begin{center}
$A = \frac{1}{2\lambda_{2}}X_{E}X_{E}^{\tau}$\\
$E := \{i: |X_{i}^{\tau}u| > \lambda \}$
\end{center}
\begin{center}
$a = \frac{\lambda_{1}}{2\lambda_{2}}(\sum\limits_{i:X_{i}^{\tau}<-\lambda_{1}}X_{i}-\sum\limits_{i:X_{i}^{\tau}>\lambda_{1}}X_{i})$
\end{center}
\paragraph{}The dual problem could also be written in a proximal form:
\begin{center}
$\hat{u} = \textbf{prox}_{\tilde{f}}(y)$\\
$\tilde{f} = \textbf{I}(\textbf{1}^{\tau}u=0)f(u) + \textbf{I}(\textbf{1}^{\tau}u \neq 0)\infty$
\end{center}
\paragraph{}After transforming $f(u)$ into a quadratic form, we could write the Lagrangian for the dual problem back again:
\begin{center}
$L = \frac{1}{2}||y-u||_{2}^{2} + \frac{1}{2}u^{\tau}Au+a^{\tau}u+b + \lambda\textbf{1}^{\tau}u$
\end{center}
\paragraph{}By taking the derivative with respect to $u$, we could obtain:
\begin{center}
$\frac{\partial L}{ \partial u} = u - y +Au+a+\lambda\textbf{1} = 0$
\end{center}
\paragraph{}By shifting the terms, $u$ could be written as a formula of $y$ and $\lambda$: $u = (I+A)^{-1}(y-a-\lambda\textbf{1})$, by taking the derivative with respect to $y$ at both sides, we could obtain the Jacobian matrix $J$ of the proximal operator $\textbf{prox}(\tilde{f})$ at $y$ as:
\begin{center}
$J = (I+A)^{-1} -(I+A)^{-1}\textbf{1}\nabla(\hat{\lambda})^{\tau}$
\end{center}
where $\nabla(\hat{\lambda})^{\tau}$ denotes the gradient of $\lambda$ as a function of $y$.
\paragraph{}By taking $u = (I+A)^{-1}(y-a-\lambda\textbf{1})$ back to the Lagrangian, the dual problem will become a second-order equation of $\lambda$:
\begin{center}
$d^{*} = \max\limits_{\lambda} \frac{1}{2}||y-(I+A)^{-1}(y-a-\lambda\textbf{1})||_{2}^{2}+\frac{1}{2}(y-a-\lambda\textbf{1})^{\tau}(I+A)^{-1}A(I+A)^{-1}(y-a-\lambda\textbf{1})+a^{\tau}(I+A)^{-1}(y-a-\lambda\textbf{1})+\lambda\textbf{1}^{\tau}(I+A)^{-1}(y-a-\lambda\textbf{1})$
\end{center}
\paragraph{}More specifically, the second-order term is:
\begin{center}
$\frac{1}{2}\textbf{1}^{\tau}(I+A)^{-2}\textbf{1}+\frac{1}{2}\textbf{1}^{\tau}(I+A)^{-1}A(I+A)^{-1}\textbf{1}-\textbf{1}^{\tau}(I+A)^{-1}\textbf{1}$
\end{center}
and the first-order term is:
\begin{center}
$2\textbf{1}^{\tau}(I+A)^{-1}(y-a)-\textbf{1}^{\tau}(I+A)^{-2}(y-a)-\textbf{1}^{\tau}(I+A)^{-1}A(I+A)^{-1}(y-a)$
\end{center}
\paragraph{}Thus by solving the second-order equation, we could obtain
\begin{center}
$\hat{\lambda} = \frac{2\textbf{1}^{\tau}(I+A)^{-1}(y-a)-\textbf{1}^{\tau}(I+A)^{-2}(y-a)-\textbf{1}^{\tau}(I+A)^{-1}A(I+A)^{-1}(y-a)}{\textbf{1}^{\tau}(I+A)^{-2}\textbf{1}+\textbf{1}^{\tau}(I+A)^{-1}A(I+A)^{-1}\textbf{1}-2\textbf{1}^{\tau}(I+A)^{-1}\textbf{1}}$
\end{center}
and the gradient
\begin{center}
$\nabla(\hat{\lambda}) = \frac{2(I+A)^{-1}\textbf{1}-(I+A)^{-2}\textbf{1}-(I+A)^{-1}A(I+A)^{-1}\textbf{1}}{\textbf{1}^{\tau}(I+A)^{-2}\textbf{1}+\textbf{1}^{\tau}(I+A)^{-1}A(I+A)^{-1}\textbf{1}-2\textbf{1}^{\tau}(I+A)^{-1}\textbf{1}}$\\
\end{center}
\paragraph{}By taking the gradient back to $J = (I+A)^{-1} -(I+A)^{-1}\textbf{1}\nabla(\hat{\lambda})^{\tau} = (I+A)^{-1} -\frac{(I+A)^{-1}\textbf{1}\textbf{1}^{\tau}(I+A)^{-1}}{\textbf{1}^{\tau}(I+A)^{-1}\textbf{1}}$, we could obtain the Jacobian.
\section{Proof of the Equivalence between the primal and the dual solutions}
\paragraph{}First recall the alo formula from the dual approach $y^{/i} = y_{i}-\frac{u_{i}}{J_{ii}} = \frac{J_{ii}-1}{J_{ii}}y_{i}+\frac{1}{J_{ii}}x_{i}\hat{\beta}$ and the primal formula from the primal approach $y^{/i} = x_{i}\hat{\beta} + \frac{H_{ii}}{1-H_{ii}}(x_{i}\hat{\beta}-y_{i}) = -\frac{H_{ii}}{1-H_{ii}}y_{i}+\frac{1}{1-H_{ii}}x_{i}\hat{\beta}$. In the following section, we're going to show that $H+J =I$, thus giving $H_{ii}+J_{ii} = 1$, and that the solutions given by both the primal and the dual approach are equivalent.
\paragraph{}First, using matrix inverse lemma, we could calculate the inverse of $(I+A) = (I+\frac{1}{2\lambda_{2}}X_{E}X_{E}^{\tau})$ as:
\begin{center}
$\begin{aligned}
(I+A)^{-1} &= (I+\frac{1}{2\lambda_{2}}X_{E}X_{E}^{\tau})^{-1}\\
&= I - X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau}
\end{aligned}
$
\end{center}
therefore the matrix $J$ is:
\begin{center}
$
\begin{aligned}
J &= (I+A)^{-1} -\frac{(I+A)^{-1}\textbf{1}\textbf{1}^{\tau}(I+A)^{-1}}{\textbf{1}^{\tau}(I+A)^{-1}\textbf{1}}\\
&= I - X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau} - \frac{(\textbf{1}-X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau}\textbf{1})(\textbf{1}^{\tau}-\textbf{1}^{\tau}X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau})}{\textbf{1}^{\tau}(I-X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau})\textbf{1}}
\end{aligned}
$
\end{center}
\paragraph{}Now recall that $H = [1,X_{E}]([1,X_{E}]^{\tau}[1,X_{E}]+diag(0,2\lambda_{2},\cdots, 2\lambda_{2}))[1,X_{E}]^{\tau}$, by adopting block inverse, we could derive $H$ as:
\begin{center}
$
\begin{aligned}
H &= [1,X_{E}]
\left(
\begin{array}{cc}
n & \textbf{1}^{\tau}X_{E}\\
X_{E}^{\tau}\textbf{1} & X_{E}^{\tau}X_{E} + 2\lambda_{2}I\\
\end{array}
\right)
[1,X_{E}]^{\tau}\\
&= [1,X_{e}]\\
&\left(
\begin{array}{cc}
\frac{1}{\textbf{1}^{\tau}(I-X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau})\textbf{1}} & \frac{-\textbf{1}^{\tau}X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}}{\textbf{1}^{\tau}(I-X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau})\textbf{1}}\\
\frac{-(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau}\textbf{1}}{\textbf{1}^{\tau}(I-X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau})\textbf{1}}&(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}+\frac{(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau}\textbf{1}\textbf{1}^{\tau}X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}}{\textbf{1}^{\tau}(I-X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau})\textbf{1}}\\
\end{array}
\right)\\
&[1,X_{E}]^{\tau}\\
& = X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau} + \frac{(\textbf{1}-X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau}\textbf{1})(\textbf{1}^{\tau}-\textbf{1}^{\tau}X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau})}{\textbf{1}^{\tau}(I-X_{E}(2\lambda_{2}I+X_{E}^{\tau}X_{E})^{-1}X_{E}^{\tau})\textbf{1}}\\
& = I-J
\end{aligned}
$
\end{center}
\paragraph{}Now that we have showed that $H+J=I$, we could also conclude that $J_{ii}+H_{ii}=1$ and that the alo solutions given by both the primal and dual approaches are equivalent.
\end{document}


\subsection{Divergence}
\noindent
In 2D we define the divergence of a vector field $\vec{F}(x,y) = \langle P(x,y), Q(x,y) \rangle$ as $\text{div}(\vec{F}) = P_x+Q_y$. This tells how the separation between particles in the vector field change over time. Positive divergence at some point means that particles tend to move away from each other, and that point is acting like a “source.” Negative divergence at some point means that particles tend to move towards each other, and that point is acting like a “sink.”

[INSERT IMAGE]

\noindent
This operation extends into higher dimensions. We define the ``del operator'' as
\begin{equation*}
	\nabla = \left< \frac{\partial}{\partial x}, \frac{\partial}{\partial y},\ldots\right> \text{ so that } \text{div}(\vec{F}) = \nabla \cdot \vec{F}.
\end{equation*}
The del operator is not coordinate system independent. The above version only works for Cartesian coordinates. For spherical coordinates,
\begin{equation*}
	\nabla \cdot \vec{F} = \frac{1}{\rho^2}\frac{\partial(\rho^2 F_\rho)}{\partial\rho} + \frac{1}{\rho\sin{\theta}}\frac{\partial}{\partial\theta}(F_\theta \sin{\theta}) + \frac{1}{\rho\sin{\theta}}\frac{\partial}{\partial\phi}F_\phi  \text{ where } \vec{F} = \langle F_\rho, F_\theta, F_\phi \rangle.
\end{equation*}
Thankfully, vector fields in spherical coordinates are rare, and it's usually easier to convert to Cartesian coordinates before doing any calculations.\\

\begin{definition}
	If $\nabla \cdot \vec{F} = 0$, then $\vec{F}$ is incompressible.
\end{definition}
\noindent
This aligns with the idea of incompressible fluids in physics and can simplify or remove the need for some calculations.

\input{./vectorAnalysis/laplacian}


\section{Differentiation}

Differentiation is a really important concept of calculus.
With this you can calculate the slope of a function in a specific point.

\subsection{The product rule}
The product rule is a simple rule to differentiate two functions which are in a product.
The rule is as following:

\[(f * g)'(x)=f'(x) * g(x) + f(x) * g'(x)\]

\subsection{The quotient rule}
The quotient rule is a rule to differentiate a function which contains the quotient of two differentiable functions.
It works as following:

\[
  (\frac{f}{g})'(x) =
  \frac{g(x) * f'(x) - g'(x) * f(x)}{{g'(x)}^2}
\]

\subsection{The chain rule}
Some functions cannot be differentiated that easily.
You will need the chain rule for that.
Take the next function:

\[f(x) = \sqrt{x^2+1}\]
To apply the chain rule you need to substitute a part of the function with \(u(x)\).
You should choose a part which, when removed, makes the function way easier.
The previous function would look like this:

\[f(x) = \sqrt{u(x)}\]
\[u(x) = x^2+1\]
Now you can differentiate the functions separately.
This would look as following:

\[
  f'(x) =
  \frac{1}{2\sqrt{u(x)}} * u'(x) =
  \frac{1}{2\sqrt{x^2+1}} * 2x
\]
You can apply this to more than the square root, like power functions.
A more general rule for the chain rule is as following:

\[(f \circ g)'(x)=f'(g(x)) * g'(x)\]

\subsection{Implicit differentiation}
Till now we have assumed that we differentiate over a {\bf explicit function}.
This means it has the form of \(f(x) = y\).
But this is not always the case.
Take the following example:

\[x^3 + y^3 = 6xy\]
This is {\it not\/} an easy function to differentiate.
With implicit differentiation you differentiate both sides with respect to \(x\).
We can differentiate \(y^3\) with the chain rule.
So \(y^3\) will become \(3y^2 * \frac{dy}{dx}\).
Trying to do that with the example function gives you:

\[(x^3 + y^3)' = (6xy)'\]
\[3x^2 + 3y^2 \frac{dy}{dx} = 6y + 6x \frac{dy}{dx}\]
\[y^2 \frac{dy}{dx} - 2x \frac{dy}{dx} = 2y - x^2\]
\[\frac{dy}{dx}(y^2 - 2x) = 2y - x^2\]
\[\frac{dy}{dx} = \frac{2y - x^2}{y^2 - 2x}\]

\subsection{Linear approximation}
If you zoom into a function with the respective tangent line it\'ll start to look like that function.
With this you can approximate the value of a function with a tangent line close to that value.
The following is called the {\bf linearization} of \(f\) in \(a\):

\[l(x) = f(a) + f'(a)(x-a)\]
For example, if we want to calculate \(f(x) = \sqrt{x}\) with \(x = 4.36\).
We take an \(a\) we know close to that point, \(4\) would make sense.
This gives the following function:

\[l(x) = 2 + \frac{1}{4}(x - 4) = 1 + \frac{x}{4}\]
Now we can fill in our \(x\) into \(l(x)\).

\[l(4.36) = 1 + \frac{4.36}{4} = 1 + \frac{109}{100} = 2.09 \approx \sqrt{4.36}\]


%---------------------------Distortion-----------------------------
\section{Distortion}

Let $A$ be the area as defined in \S\ref{s:quad-area}
and $A_m = 4$ be the area of a ``master'' quadrilateral with vertices
\[
\begin{array}{lcrcrcrl}
  \vec P_0 &= (&-1&,&-1&,& 0&)\\
  \vec P_1 &= (& 1&,&-1&,& 0&)\\
  \vec P_2 &= (& 1&,& 1&,& 0&)\\
  \vec P_3 &= (&-1&,& 1&,& 0&).
\end{array}
\]
Now define $|J|$ as the minimum value of the
determinant of the Jacobian evaluated at all Gauss points of the element.
The distortion is then
\[
q = \frac{|J| A_m}{A} = \frac{4|J|}{A}.
\]
Distortion is a measure of how well-behaved the mapping from
parameter space to world coordinates is.

\quadmetrictable{distortion}%
{$1$}%                                      Dimension
{$[0.5,1]$}%                                Acceptable range
{$[0,1]$}%                                  Normal range
{$[-DBL\_MAX,DBL\_MAX]$}%                   Full range
{$1$}%                                      Unit square
{\cite{ideas:xx}}%                          Citation
{v\_quad\_distortion}%                      Verdict function name