\section{Optimization Functions}

The functions in this section are models with analytic optimal points.
\setcounter{tocdepth}{4}
\subsection{Continuous}
\subsubsection{Unconstrained}
\paragraph{Beale's Function}
Beale's function has a two-dimensional input space and a single global/local minimum.
See \url{https://en.wikipedia.org/wiki/Test_functions_for_optimization}.

\begin{itemize}
  \item Objective Function: $f(x,y) = (1.5-x+xy)^2+(2.25-x+xy^2)^2+(2.625-x+xy^3)^2$
  \item Domain: $-4.5 \leq x,y \leq 4.5$
  \item Global Minimum: $f(3,0.5)=0$
\end{itemize}

\paragraph{Rosenbrock Function}
The Rosenbrock function can take a varying number of inputs.  For up to three inputs, a single global minimum
exists.  For four to seven inputs, there is one local minimum and one global maximum.
See \url{https://en.wikipedia.org/wiki/Rosenbrock_function}.

\begin{itemize}
  \item Objective Function: $f(\vec x) = \sum_{i=1}^{n-1}\left[100\left(x_{i+1}-x_i^2\right)^2+\left(x_i-1)^2\right) \right]$
  \item Domain: $ -\infty \leq x_i \leq \infty \hspace{10pt} \forall \hspace{10pt} 1\leq i \leq n$
  \item Global Minimum: $f(1,1,\cdots,1,1)=0$
  \item Local minimum ($n\geq4$): near $f(-1,1,\cdots,1)$
\end{itemize}

\paragraph{Goldstein-Price Function}
The Goldstein-Price function is a two-dimensional input function with a single global minimum.
See \url{https://en.wikipedia.org/wiki/Test_functions_for_optimization}.

\begin{itemize}
  \item Objective Function:
    \begin{align}
      f(x,y) =& \left[1+(x+y+1)^2\left(19-14x+3x^2-14y+6xy+3y^2\right)\right] \\ \nonumber
        & \cdot\left[30+(2x-3y)^2(18-32x+12x^2+48y-36xy+27y^2)\right]
    \end{align}
  \item Domain: $-2 \leq x,y \leq 2$
  \item Global Minimum: $f(0,-1)=3$
\end{itemize}

\paragraph{McCormick Function}
The McCormick function is a two-dimensional input function with a single global minimum.
See \url{https://en.wikipedia.org/wiki/Test_functions_for_optimization}.

\begin{itemize}
  \item Objective Function: $f(x,y) = \sin(x+y) + (x-y)^2 - 1.5x + 2.5y + 1$
  \item Domain: $-1.5 \leq x \leq 4$, $-3 \leq y \leq 4$
  \item Global Minimum: $f(-0.54719,-1.54719) = -1.9133$
\end{itemize}

\paragraph{2D Canyon}
The two-dimensional canyon offers a low region surrounded by higher walls.

\begin{itemize}
  \item Objective Function: $f(x,y) = xy\cos(x+y)$
  \item Domain: $0 \leq x,y \leq \pi$
  \item Global Minimum: $f(1.8218,1.8218)=-2.90946$
\end{itemize}

\paragraph{Stochastic Diagonal Valley}
The stochastic three-dimensional diagonal valley offers a low region surrounded by higher walls.

\begin{itemize}
  \item Objective Function: $f(x,y,stoch) = stoch * R + (x+0.5)^2 + (y-0.5)^2 + \sqrt{(x - \cfrac{x}{x-y})^2 + (y - \cfrac{y}{y-x})*^2} * 10, while R\sim ([-0.1,0.1])$
  \item Domain: $0 \leq x,y \leq 1$
  \item Global Minimum if $stoch =0$: $f(-0.5,0.5)=0$
\end{itemize}

\paragraph{Parabolic Valley}
The parabolic valley is dominated by distance from $x=y$ but secondarily motivated by $x+y$.

\begin{itemize}
  \item Objective Function: $f(x,y) = 100 (x - y)^2 + (x + y)^2$
  \item Domain: $-1 \leq x,y \leq 1$
  \item Global Minimum: $f(0,0)=0$
\end{itemize}

\paragraph{Paraboloid}
The elliptic paraboloid is a two-dimensional function with a single global minimum.

\begin{itemize}
  \item Objective Function: $f(x,y) = 10(x+0.5)^2 + 10(y-0.5)^2 $
  \item Domain: $-2 \leq x \leq 2$, $-2 \leq y \leq 2$
  \item Global Minimum: $f(-0.5,0.5) = 0$
\end{itemize}

\paragraph{Time-Parabola}
This model features the sum of a parabola in both $x$ and $y$; however, the parabola in $y$ moves in time and
has a reduced magnitude as time increases. As such, the minimum is always found at $x=0$ and at $(t-y)=0$, with
$y$ values at low $t$ being more impactful than at later $t$, and $x$ as impactful as $y(t=0)$.

\begin{itemize}
  \item Objective Function: $f(x,y,t) = x^2 + \sum_{t} (t-y)^2 \exp{-t}$
  \item Domain: $-10 \leq x,y,t \leq 10$
  \item Global Minimum: $f(0,y=t,t) = 0$
\end{itemize}

\paragraph{Eggholder}
This is a 2D function with a plethora of local minima and maxima (i.e., non-convex). It is challenging for most optimizers to find the global minimum for such a function. Metaheuristic methods are prefered for their ability to avoid getting stuck in local optima.

\begin{itemize}
	\item Objective Function: $f(x,y,) =-(y+47) \sin(\sqrt{|\frac{x}{2}+(y+47)|}) - x\sin(\sqrt{|x - (y+47)|})$
	\item Domain: $-512 \leq x,y \leq 512$
	\item Global Minimum: $f(x=512, y=404.2319) = -959.6407$
\end{itemize}

\subsubsection{Constrained}
\paragraph{Mishra's Bird Function}
The Mishra bird function offers a constrained problem with multiple peaks, local minima, and one steep global
minimum.
See \url{https://en.wikipedia.org/wiki/Test_functions_for_optimization}.

\begin{itemize}
  \item Objective Function: $f(x,y) = \sin(y)\exp[1-\cos(x)]^2 + \cos(x)\exp[1-\sin(y)]^2 + (x-y)^2$
  \item Constraint: $(x+5)^2 + (y+5)^2 < 25$
  \item Domain: $-10 \leq x \leq 0$, $-6.5 \leq y \leq 0$
  \item Global Minimum: $f(-3.1302468, -1.5821422) = -106.7645367$
\end{itemize}

\paragraph{ND Slant}
This trivial linear problem extends to arbitrary dimension and has optimal points at infinities,
which is usually outside the domain of exploration.
\begin{itemize}
  \item Objective Function: $f(\vec x) = 1 - \frac{1}{N}\sum_{x_i\in \vec x} x_i$
  \item Domain: $0 \leq x_i \leq 1 \forall x_i \in \vec x$
  \item Global Minimum: $f(\{1\}_N) = 0$
\end{itemize}

% \paragraph{Rosenbrock with cubic and linear constraints}
% Here the Rosenbrock is another two dimensional highly nonlinear function. The global maximum is on the intersection of the two constraints which renders this problem challenging for optimizers that uses random decisions for next points such as simulated annealing and genetic algorithms.
% \begin{itemize}
% 	\item Objective Function: $f(x,y) = (1-x^2) + 100(y-x^2)^2$
% 	\item Domain: $-1.5 \leq x \leq 1.5$ and $ -0.5 \leq y \leq 2.5$
% 	\item Constraints: $(x-1)^3 - y +1 \leq 0$ and $x+y-2 \leq 0$
% 	\item Global Maximum: $f(1.0,1.0) = 0$
% \end{itemize}

\paragraph{Rosenbrock with a disk constraint}
This is the same 2D function, but constrained to a disk. This constraint makes it a little easier than the previous function.
\begin{itemize}
	\item Objective Function: $f(x,y) = (1-x^2) + 100(y-x^2)^2$
	\item Domain: $-1.5 \leq x,y \leq 1.5$
	\item Constraint: $x^2 + y^2 \leq 2$
	\item Global Maximum: $f(1.0,1.0) = 0$
\end{itemize}

\paragraph{Townsend}
It is a highly nonlinear non convex function with multiple minima and maxima. The constraint is highly nonlinear and the y-domain is asymmetric.
\begin{itemize}
	\item Objective Function: $f(x,y) = -\left[\cos((x-0.1)y)\right]^2-x\sin(3x+y)$
	\item Domain: $-2.25 \leq x \leq 2.25$ and $-2.5 \leq y \leq 1.75$
	\item Constraint: $x^2 + y^2 \leq \left[2\cos(t)-\frac{1}{2}\cos(2t)-\frac{1}{4}\cos(3t) - \frac{1}{8}cos(4t) \right]^2+\left[2\sin(t)\right]^2 $;  \[ \text{where: } t=\arctan2(x,y) \]
	\item Global Minimum: $f(2.0052938,1.1944509) = -2.0239884$
\end{itemize}

\paragraph{Simionescu}
An exteremly simple two dimensional nonlinear function formed from 45-degrees hyperbolas. Subjected to highly nonlinear constraint.
\begin{itemize}
	\item Objective Function: $f(x,y) = 0.1xy$
	\item Domain: $-1.25 \leq x,y \leq 1.25$
	\item Constraint: $x^2 + y^2 \leq \left[r_{\Gamma}+r_S\cos\left(n \arctan(\frac{x}{y})\right)\right]^2 $; \[ \text{where: } r_{\Gamma}=1, r_S = 0.2, \text{and } n=8 \]
	\item Global Minimum: $f(\pm 0.84852813,\mp0.84852813) = -0.072$
\end{itemize}
\subsection{Discrete}
\subsubsection{Unconstrained}
\paragraph{Locally weighted sum without replacement}
This function computes the sum of the input vector components weighted by the location in the vector (i.e., component/dimension number). The input vector is sampled from a disrete uniform distribution of intergers between an upper bound $ub$, and a lower bound $lb$ without replacement (i.e., if an integer is selected for one variable (a component in the $n$-th dimensional input vector it cannot be selected for the remaining components)).
\begin{itemize}
	\item Objective Function: $f(\vec{x}) = \sum_{i=1}^{G}i \times x_i$; where $G$ is the number of variables (aka dimension of search/design space or number of Genes in Genetic algorithms)
	\item Domain: $x_i \sim \mathcal{U}^{\text{Discrete int}}_{\text{w/o replacement}} (lb,ub)$
	\item Global Minimum: $f(\vec{x}_{opt}|x_i = lb + G - i) = \sum_{i=1}^{G} (lb + G -i) \times i$
	For instance, if $lb =1$, $ub=20$ , $G=6$, $\vec{x}_{opt} = [\vec{x}]_i | x_i = lb + G - i = 1+6-i = \begin{bmatrix}
	6 & 5 & \dots & 1 \end{bmatrix}$ and the minimum will be $f_{min} = \sum_{i=1}^{G} (1 + 6 -i) \times i = \frac{G(G+1)}{6}[3lb+G-1] = 56$
	\item Global Maximum: $f(\vec{x}_{opt}|x_i = ub - G + i) = \sum_{i=1}^{G} (ub - G +i) \times i$
	For instance, if $lb =1$, $ub=20$ , $G=6$, $\vec{x}_{opt} = [\vec{x}]_i | x_i = ub - G + i = 20-6+i = \begin{bmatrix}
	15 & 16 & \dots & 20 \end{bmatrix}$ and the maximum will be $f_{max} = \sum_{i=1}^{G} (20 - 6 +i) \times i = \frac{G(G+1)}{6}[3ub-G+1] = 385$
\end{itemize}

\paragraph{Locally weighted sum with replacement}
This function computes the sum of the input vector components weighted by the location in the vector (i.e., component/dimension number). The input vector is sampled from a disrete uniform distribution of intergers between an upper bound $ub$, and a lower bound $lb$ with replacement (i.e., if an integer is selected for one variable, it can be selected again for any or all other variables).
\begin{itemize}
	\item Objective Function: $f(\vec{x}) = \sum_{i=1}^{G}i \times x_i$; where $G$ is the number of variables (aka dimension of search/design space or number of Genes in Genetic algorithms)
	\item Domain: $x_i \sim \mathcal{U}^{\text{Discrete int}}_{\text{w/ replacement}} (lb,ub)$
	\item Global Minimum: $f(\vec{x}_{opt}|x_i = lb) = \sum_{i=1}^{G} lb \times i$
	For instance, if $lb =1$, $ub=20$ , $G=6$, $\vec{x}_{opt} = [\vec{x}]_i | x_i = lb = 1 = \begin{bmatrix}
	1 & 1 & \dots & 1 \end{bmatrix}$ and the minimum will be $f_{min} = \sum_{i=1}^{G} i = \frac{G(G+1)}{2}[lb] = 21$
	\item Global Maximum: $f(\vec{x}_{opt}|x_i = ub) = \sum_{i=1}^{G} ub \times i$
	For instance, if $lb =1$, $ub=20$ , $G=6$, $\vec{x}_{opt} = [\vec{x}]_i | x_i = ub = 20 = \begin{bmatrix}
	20 & 20 & \dots & 20 \end{bmatrix}$ and the maximum will be $f_{max} = \sum_{i=1}^{G} 20 \times i = \frac{G(G+1)}{2}[ub] = 420$
\end{itemize}

\subsubsection{Constrained}
\paragraph{Locally weighted sum with replacement linearly constrained}
It uses the same function but adds a constraint.
\begin{itemize}
	\item Objective Function: $f(\vec{x}) = \sum_{i=1}^{G}i \times x_i$; where $G$ is the number of variables (aka dimension of search/design space or number of Genes in Genetic algorithms)
	\item Domain: $x_i \sim \mathcal{U}^{\text{Discrete int}}_{\text{w/o replacement}} (lb,ub)$
	\item constraint $x_2 \leq 8.5 - 0.5 * x_1$
	\item Global Minimum if $lb = 1, ub = 6, G =3$: $f(\vec{x}_{opt} = \begin{bmatrix}
	1 & 1 & 1 \end{bmatrix}) = 6.0$
	\item Global Maximum if $lb = 1, ub = 6, G =3$: $f(\vec{x}_{opt} = \begin{bmatrix}
	6 & 5 & 6 \end{bmatrix} = 34.0$
\end{itemize}

\paragraph{Locally weighted sum with replacement sphere constrained}
It uses the same function but adds a constraint.
\begin{itemize}
	\item Objective Function: $f(\vec{x}) = \sum_{i=1}^{G}i \times x_i$; where $G$ is the number of variables (aka dimension of search/design space or number of Genes in Genetic algorithms)
	\item Domain: $x_i \sim \mathcal{U}^{\text{Discrete int}}_{\text{w/o replacement}} (lb,ub)$
	\item constraint $x^{2}_{1} + x^{2}_{2} \leq 72$
	\item Global Minimum if $lb = 1, ub = 6, G =3$: $f(\vec{x}_{opt} = \begin{bmatrix}
1 & 1 & 1 \end{bmatrix})= 6.0$
\item Global Maximum if $lb = 1, ub = 6, G =3$: $f(\vec{x}_{opt} = \begin{bmatrix}
6 & 5 & 6 \end{bmatrix} = 34.0$
\end{itemize}


While first and second moments give us some idea of the performance of our sampling algorithms, we ideally would like a fuller picture.  In this section we compare the performance of algorithms using the total variation distance, Wasserstein distance and Kullback--Leibler divergence.  Using these measures, we can compare the performance to theoretical upper bounds for \texttt{ULA}.

\subsection{Statistical Distances}
Let $\mathcal{B}(\R^d)$ denote the Borel $\sigma$-algebra on $\R^d$. Let $P$ and $Q$ be probability measures on the space $(\R^d, \mathcal{B}(\R^d))$.  Then we define the total variation distance, Kullback--Leibler divergence and Wasserstein metric as follows:

\begin{defn}[Total Variation]
The total variation distance between two probability measures $P$ and $Q$ on $(\Omega, \mathcal{F})$ is defined as
$$
\norm{P - Q}_{TV} = \sup_{A \in \mathcal{F}} \abs{P(A) - Q(A)}.
$$
\end{defn}
In other words, total variation measures the greatest possible difference between the probability of an event according to $P$ and $Q$.
\begin{prop}
If the set $\Omega$ is countable then this is equivalent to half the $L^1$ norm.
$$
\norm{P - Q}_{TV} = \frac{1}{2} \norm{P-Q}_1 = \frac{1}{2} \sum_{\omega \in \Omega} \abs{P(\omega) - Q(\omega)}
$$
\end{prop}
\begin{proof}
Let $B = \{\omega: P(\omega) \geq Q(\omega)\}$ and let $A \in \mathcal{F}$ be any event.  Then
$$
P(A) - Q(A) \leq P(A \cap B) - Q(A \cap B) \leq P(B) - Q(B).
$$
The first inequality holds since $P(\omega)-Q(\omega) < 0$ for any $\omega \in A \cap B^c$, and so the difference in probability cannot be greater if these elements are excluded.  For the second inequality, we observe that including further elements of $B$ cannot decrease the difference in probability.
Similarly,
$$
Q(A) - P(A) \leq Q(B^c) - P(B^c) = P(B) - Q(B)
$$
Thus, setting $A=B$, we have that $\abs{P(A)-Q(A)}$ is equal to the upper bound in the total variation distance.  Hence,
$$
\norm{P-Q}_{TV} = \frac{1}{2} \abs{P(B)-Q(B)+Q(B^c)-P(B^c)} = \frac{1}{2} \sum_{\omega \in \Omega} \abs{P(x)-Q(x)}
$$
\end{proof}

\begin{defn}[Kullback--Leibler Divergence]
Let $P$ and $Q$ be two probability measures on $(\Omega, \mathcal{F})$.  If $P \ll Q$, the Kullback--Leibler divergence of $P$ with respect to $Q$ is defined as
$$
D_{\text{KL}}(P\,||\,Q) = \int_\Omega \od{P}{Q} \log \left(  \od{P}{Q} \right) \dif  Q.
$$
\end{defn}
The Kullback--Leibler divergence from $Q$ to $P$ measures the information lost in using $Q$ to approximate $P$  \cite{anderson2004model} and is also known as the relative entropy.  It is worth noting that, unlike the other two measures considered here, the Kullback--Leibler divergence is not a metric, and in particular is not symmetric.

Finally we consider the Wasserstein distance.  If $P$ and $Q$ are probability measures on $(\Omega, \mathcal{F})$, we say that $\gamma$ is a transport plan between two probability measures $P$ and $Q$ if it is a probability measure on $(\Omega \times \Omega, \mathcal{F} \times \mathcal{F})$ such that for any Borel set $A \subset \mathcal{F}$, $\gamma(A \times \Omega)=P(A)$ and $\gamma(\Omega \times A) = Q(A)$.  We denote the set of all such transport plans by $\Pi(P,Q)$.  In simple terms, the set of transport plans, $\Pi(P,Q)$, represents the possible ways of transporting mass distributed according to $P$ to a distribution according to $Q$, without creating or destroying mass in the process.  The `effort' to transport mass is then represented by a cost function $d:\Omega \to \Omega$, so that $d(x,y)$ is the cost of moving unit mass from $x$ to $y$.

\begin{defn}[Wasserstein distance]
For two probability measures, $P$ and $Q$, the $p$-Wasserstein distance is given by
$$
W_p(P,Q) = \left( \inf_{\gamma \in \Pi(P,Q)} \int_{\Omega \times \Omega} d(x,y)^p d \gamma(x,y) \right)^{1/p}.
$$
\end{defn}

The Wasserstein distance represents the amount of `effort' required to move mass distributed according to $P$ to $Q$.  We restrict our attention to $L^1$-Wasserstein and $L^2$-Wasserstein distances, which is to say that we choose our cost function $d$ to be the Euclidean distance, and $p=1,2$.

One particular advantage of Wasserstein distance compared to total variation or Kullback--Leibler divergence is that bounds on Wasserstein distance can be used directly to bound the accuracy of the first and second moment approximations, and so for application to statistics there is some evidence to suggest it is the most appropriate of the three measures for our purposes \cite{dalalyan2019user}.

Due to impracticality of computing higher-dimensional Wasserstein distances, we use a computationally more feasible variant, the Sliced Wasserstein distance. First proposed in \cite{rabin2011wasserstein} and further elaborated on, for example, in \cite{gswd}, the Sliced Wasserstein distance exploits the fact that the Wasserstein distance between 1-dimensional probability measures $P, Q$ can be computed with an explicit formula $\abs{F^{-1}(t)-G^{-1}(t)}^p dt$ where $F$ and $G$ are the CDFs of $P$ and $Q$ respectively \cite{ramdas2017wasserstein}.

\begin{defn}[Sliced Wasserstein distance]
For two probability measures, $P$ and $Q$, the $L^p$ Sliced Wasserstein distance is given by
$$
SW_p(P,Q) = \left(\int_{\mathbb S^{d-1} }  W_p^p\left(\mathcal{RI}_P(\cdot, \theta), \mathcal{RI}_Q(\cdot, \theta) \right) d \theta \right)^{\frac 1 p}
$$
\end{defn}

where $\mathbb S^{d-1}$ is the $(d-1)$-dimensional sphere and $\mathcal RI$ denotes the Inverse Radon transform. In the above references, it is also proved that $SW_p$ is indeed a metric. The main reason why we can use the Sliced Wasserstein distance as an approximation to the Wasserstein distance is that these two metrics are equivalent\cite{Santa}.

\subsubsection{Numerical Comparison}
In this section we present a numerical comparison of the algorithms considered, extending the results of \cite{Brosse18tULA} which only considered first and second moments.  It should be noted that the results in the aforementioned paper are in dimension 100, while here our results are in dimension 2.  This is due to the bin-filling problem when approximating the density function, which is explained in section \ref{sec:Imp}.  We do not have evidence to suggest the performance of these algorithms would be similar in higher dimensions.  Due to lack of space in this report, the results here are mostly in total variation distance.  The functionality for the experiments is available in our GitHub repository (see section \ref{sec:Imp}), and tests can easily be performed in Sliced 1-Wasserstein distance and Kullback--Leibler divergence.

\begin{note}
    It is important to state here that the accuracy of our implementation of these measures has not been thoroughly tested.  The number of bins chosen for the histogram heavily influences results, and our default choice of 100 bins for 2 dimensions is somewhat arbitrary.  As such, the results presented here should be treated with some caution.
\end{note}

The results for total variation in the case of the Gaussian (Figure \ref{fig:TVgauss}) support the claim of \cite{Brosse18tULA} that tamed algorithms have similar performance to untamed for small step size.  As expected, they do perform slightly worse than the untamed equivalent, but this difference is relatively small.  For the ill-conditioned case (Figure \ref{fig:TV_ICgauss}), the unadjusted versions of each algorithm diverge, but the tamed versions are stable.  While the performance of all Langevin-based methods is poor, coordinate-wise taming, as predicted, improves performance, and we expect this effect to be more pronounced in higher dimensions.

The double well presents a more interesting case.  For small step size (Figure \ref{fig:TVdouble02}) there is not much difference between the algorithms, with \texttt{LM} performing the poorest of all the Langevin-based methods, for higher step sizes (Figures \ref{fig:TVdouble1}, \ref{fig:TVdouble2}) it outperforms all other algorithms, even those with Metropolis--Hastings adjustments.  For step size 0.1 and higher, the matrix multiplication step in \texttt{HOLA} results in an overflow, while for step size 0.2, \texttt{ULA} diverges.  It is only for step size 0.3 that \texttt{LM} finally diverges (Figure \ref{fig:TVdouble3}).  The reasons for this behaviour is unclear, but these results seem to support the claims made in \cite{LM12}.  To verify this, the experiment for the double well with step size 0.1 was repeated in Sliced 1-Wasserstein distance (Figure \ref{fig:SWdouble1}).  The results broadly agree with those of the total variation, although Metropolised algorithms appear to be favoured by this metric, and tamed algorithms graded more harshly.

The final three plots (Figures \ref{fig:TVginz01}-\ref{fig:TVrosen01}) are included mostly for completeness.  It is interesting to note however, that on the Rosenbrock distribution (Figure \ref{fig:TVrosen001}) \texttt{LM} again outperforms \texttt{ULA} for step size 0.001.
\begin{figure}[ht!]
	\centering
		\includegraphics[height=0.43\textheight]{Figures/TV_gaussian11_step0pt02.png}
	\caption{$10^5$ samples from standard Gaussian with step size 0.02}
	\label{fig:TVgauss}
\end{figure}

\begin{figure}[ht!]
	\centering
		\includegraphics[height=0.43\textheight]{Figures/TV_ICgaussian10pt0001_step0pt02.png}
	\caption{$10^5$ samples from ill-conditioned Gaussian with step size 0.02}
	\label{fig:TV_ICgauss}
\end{figure}

\begin{figure}[ht!]
	\centering
		\includegraphics[height=0.43\textheight]{WriteUp/TV_doublewell_step0pt02.png}
	\caption{$10^5$ samples from double well with step size 0.02}
	\label{fig:TVdouble02}
\end{figure}

\begin{figure}[ht!]
	\centering
		\includegraphics[height=0.43\textheight]{WriteUp/TV_doublewell_step0pt1.png}
	\caption{$10^5$ samples from double well with step size 0.1}
	\label{fig:TVdouble1}
\end{figure}

\begin{figure}[ht!]
	\centering
		\includegraphics[height=0.43\textheight]{WriteUp/SW_doublewell_0pt1.pdf}
	\caption{SW1 distance $10^5$ samples from double well with step size 0.1}
	\label{fig:SWdouble1}
\end{figure}

\begin{figure}[ht!]
	\centering
		\includegraphics[height=0.43\textheight]{WriteUp/TV_doublewell_step0pt2.png}
	\caption{$10^5$ samples from double well with step size 0.2}
	\label{fig:TVdouble2}
\end{figure}

\begin{figure}[ht!]
	\centering
		\includegraphics[height=0.43\textheight]{WriteUp/TV_doublewell_step0pt3.png}
	\caption{$10^5$ samples from double well with step size 0.3}
	\label{fig:TVdouble3}
\end{figure}

\begin{figure}[ht!]
	\centering
		\includegraphics[height=0.43\textheight]{WriteUp/TV_ginzburg_step0pt01.png}
	\caption{$10^5$ samples from 1D Ginzburg Landau with step size 0.01}
	\label{fig:TVginz01}
\end{figure}

\begin{figure}[ht!]
	\centering
		\includegraphics[height=0.43\textheight]{WriteUp/TV_rosenbrock_step0pt001.png}
	\caption{$10^5$ samples from Rosenbrock with step size 0.001}
	\label{fig:TVrosen001}
\end{figure}

\begin{figure}[ht!]
	\centering
		\includegraphics[height=0.43\textheight]{WriteUp/TV_rosenbrock_step0pt01.png}
	\caption{$10^5$ samples from Rosenbrock with step size 0.01}
	\label{fig:TVrosen01}
\end{figure}

\newpage
\subsection{Theoretical Non-asymptotic Error Bounds}

While the asymptotic behaviour of \texttt{ULA} and \texttt{MALA} is well-understood \cite{RT96}, the results do not consider the effect of dimension on the complexity of the algorithms.  For practical purposes, an understanding of the non-asymptotic behaviour of the algorithms would be useful and, in particular, theoretical results which could determine the step size and number of iterations required to guarantee an error of no more than some acceptable value $\epsilon$.

Theoretical bounds of this nature were first provided in \cite{dalalyan2017theoretical}, which proved bounds on the total variation distance between the distribution of the $n^{\text{th}}$ iterate of the unadjusted Langevin Algorithm, under restrictive assumptions, which will be outlined shortly.  In the case of a `warm start', where the distribution of the initial value is close to $\Pi$, it was shown that \texttt{ULA} had an upper bound of $\mathcal{O}(d/\epsilon)$ iterations to achieve precision level $\epsilon$.  This result was improved by \cite{durmus2016high, durmus2017nonasymptotic} which extended the analysis to the Wasserstein distance and dispensed with the assumption of a warm start, showing that the upper bound on iterations could be reduced to $\mathcal O (d/\epsilon)$, provided the Hessian of the potential is Lipschitz continuous.  Most recently, \cite{dalalyan2019user} has provided `user-friendly' bounds on the Wasserstein distance, further improving the constants in these bounds.

It is important to note here that these results provide explicit constants for the non-asymptotic behaviour for \texttt{ULA}.  This is in contrast to the theory for \texttt{MALA} \cite{bou2013nonasymptotic}, and the tamed \cite{Brosse18tULA} and higher order algorithms \cite{Sabanis18tHOLA}, for which only the existence of a constant is proven.  For the Leimkuhler--Matthews method, no such guarantees are known.  As such the non-asymptotic theory for the unadjusted algorithm is practically much more useful.

\subsubsection{Non-asymptotic Bounds for \texttt{ULA}}
We present without proof the results of \cite{durmus2017nonasymptotic, dalalyan2019user} which to our knowledge are the best known bounds in total variation and Wasserstein distance respectively. For both results we assume that the potential $U$ is continuously differentiable on $\R^d$, and there exist positive constants $m$ and $M$ such that $U$ is $m$-strongly convex and $M$-gradient Lipschitz, i.e. for all $x$, $y \in \R^d$,
\begin{align} \label{m-convex}
    &\text{($m$-strongly convex) \ } U(x) - U(y) - \grad U(y)^\top (x - y) \geq \frac{m}{2}\norm{x-y}_2^2\\
    &\text{($M$-gradient Lipschitz) \ } \norm{\grad U(x) - \grad U(y)}_2 \leq M \norm{x-y}_2. \label{M-GradLipschitz}
\end{align}

Denote the unique minimiser of $U$ by $y=\arg \min_{x \in \R^d} U(x)$.  Let $\nu_N$ denote the distribution of the $N^{\text{th}}$ sample

\begin{theorem}[Total Variation part (i)]
    Assume $h \in (0, 2/(m+M))$ and $U$ satisfies the assumptions (\ref{m-convex}, \ref{M-GradLipschitz}) above.  Let $\kappa = \frac{2mM}{m+M}$  Then for any initial value $x_0 \in \R^d$ and $N \geq 1$,
    $$
    \norm{\pi_h - \nu_N}_{TV} \leq \left\{ 4 \pi \kappa (1-(1-\kappa h)^{N/2} \right\}^{-1/2} (1-\kappa \gamma)^{N/2} \left\{ \norm{x_0 - y}_2 + (2 \kappa^{-1}d)^{1/2} \right\}
    $$
\end{theorem}

\begin{theorem}[Total Variation part (ii)]
    Assume $h \in (0, 1/(m+M))$, $U$ satisfies the assumptions (\ref{m-convex}, \ref{M-GradLipschitz}) above, and further, assume that $U$ is three times continuously differentiable, and there exists $L$ such that for all $x,y \in \R^d$,
    $$
    \norm{\grad^2 U(x) - \grad^2 U(y)} \leq L \norm{x-y}.
    $$
    Then
    \begin{align*}
        \norm{\pi_h - \pi}_{TV} &\leq (4 \pi)^{-1/2} \left\{ h^2 E_1(h, d) + 2dh^2 E_2(h)/(\kappa m)\right\}^{1/2}\\
        &+ (4 \pi)^{-1/2} \lceil \log(h^{-1}/\log(2) \rceil \left\{ h^2 E_1(h,d) + h^2 E_2(h)(2 \kappa^{-1}d + d/m) \right\}^{1/2}\\
        &+ 2^{-3/2} M \left\{ 2d h^3 L^2/(3\kappa) + dh^2\right\}^{1/2}
    \end{align*}
    where $E_1(h,d)$ and $E_2(h)$ are defined as
    \begin{align*}
        E_1(h,d) &= 2 d \kappa^{-1} \left\{2L^2 + 4 \kappa^{-1} (dL^2/3 + h M^4 /4) + h^2 M^4/6 \right\}\\
        E_2(h) &= L^4(4\kappa^{-1}/3 + h)
    \end{align*}
\end{theorem}

The triangle inequality gives our desired bound on $\norm{\pi - \nu_N}_{TV}$.  We next present the `user-friendly' bound in Wasserstein distance.

\begin{theorem}[Wasserstein distance]
    Assume $h \in (0, 2/M)$ and $U$ satisfies the assumptions (\ref{m-convex}, \ref{M-GradLipschitz}) above.
    \begin{itemize}
        \item If $h \leq \frac{2}{m+M}$, then $W_2(\nu_N, \pi) \leq (1-mh)^N W_2(\nu_0, \pi) + 1.65 \frac{M}{m}(hp)^{1/2}.$
        \item If $h \geq \frac{2}{m+M}$, then $W_2(\nu_N, \pi) \leq (Mh-1)^N W_2(\nu_0, \pi) + \frac{1.65Mh}{2-Mh}(hp)^{1/2}.$
\end{itemize}
\end{theorem}
\begin{prop}
    If the initial value $X_0 = x_0$ is deterministic then,
    $$
    W_2(\nu_0, \pi)^2 = \int_{\R^p} \norm{x_0-x}_2^2 \pi(dx) \leq \norm{x_0-y}_2^2 + \frac{p}{m}.
    $$
\end{prop}

\begin{remark}
    If we choose $h$ and $N$ such that
    $$
    h \leq \frac{2}{m+M}, \quad e^{-mhN}W_2(\nu_0, \pi) \leq \epsilon/2, \quad 1.65\frac{M}{m}(hp)^{1/2} \leq \epsilon/2
    $$
    then $W_2(\nu_N,\pi) \leq \epsilon$.  Hence, for a deterministic initial value $X_0=x_0$, it is sufficient to choose
    $$
    h \leq \frac{m^2 \epsilon^2}{11M^2p} \wedge \frac{2}{m+M}, \quad hN \geq \frac{1}{m} \log \left( \frac{2(\norm{x_0-y}_2^2+p/m)^{1/2}}{\epsilon} \right)
    $$
    for a precision $\epsilon$ in $W_2(\nu_K, \pi)$.
\end{remark}

\begin{note}
    In practice, $\norm{x_0-y}_2$ may be difficult to calculate.  An alternative bound can easily be derived from the strong convexity of $U$ and the fact that $y$ minimises $U$:
    \begin{align*}
        m W_2(\nu_0, \pi)^2 &\leq m \norm{x_0-y}_2^2 + p\\
        &\leq 2(f(x_0) - f(y) - \nabla U(y)^{\top}(x_0-y) + p\\
        &= 2(f(x_0)-f(y))+p.
    \end{align*}
    If $U$ is bounded below by a constant, say $U \geq 0$, this provides an easily computable upper bound on $W_2(\nu_0, \pi)$.
\end{note}

\subsubsection{Numerical Tests}
Despite the `user-friendly' nature of the bounds in Wasserstein distance, it was not possible to verify them numerically, as computing the $W_2$ distance is computationally infeasible.  This problem is explained in more detail in section \ref{sec:Imp}.  We were, however, able to test the bounds in total variation.  The only distribution we have implemented which satisfies the assumptions above was the Gaussian.  For the following results, we test on a 1-dimensional $N(0,1)$ distribution.

\begin{figure}[H]
	\centering
		\includegraphics[height=0.35\textheight]{WriteUp/DM_Pih.pdf}
	\caption{Bound in $\norm{\pi_h - \pi}_{TV}$ as function of $h$.  This bound can be used to select a suitable step size to give error $\epsilon/2$}
	\label{fig:DM_Pih}
\end{figure}

\begin{figure}[H]
	\centering
		\includegraphics[height=0.35\textheight]{WriteUp/DM_EM_step0pt01.pdf}
	\caption{Bound in $\norm{\pi_h - \nu_N}_{TV}$ as function of $N$, for $h=0.01$.  Having chosen a suitable $h$ according to the bound above, a suitable $N$ can be chosen to give error $\epsilon/2$}
	\label{fig:DM_Pih}
\end{figure}

\begin{figure}[H]
	\centering
		\includegraphics[height=0.35\textheight]{Figures/DM_plot_nth_sample.png}
	\caption{$\norm{\pi - \nu_N}_{TV}$ as function of $N$, plotted against theoretical bounds.  The experiment was run with 600 chains, to produce a histogram with 80 bins.  This number of chains is insufficient to calculate the total variation to a higher level of accuracy, which may explain the difference between our calculated result and the theoretical bound.}
	\label{fig:DMnth}
\end{figure}

\begin{figure}[H]
	\centering
		\includegraphics[height=0.35\textheight]{Figures/DM_plot_whole_chain.png}
	\caption{Total Variation distance between distribution taken by pooling all $N$ samples from multiple chains and true (standard normal) distribution as function of $N$, plotted against theoretical bounds.  Although the error for the pooled distribution should be worse than that for the $N^{\text{th}}$ sample, a much higher accuracy is achieved.  This supports our claim that the higher than expected error in the above plot is a result of too few chains being run.}
	\label{fig:DMwhole}
\end{figure}


% -*- root: Main.tex -*-
\section{Essentials}
\subsection*{Derivatives}
$\frac{\partial}{\partial \mathbf{x}}(\mathbf{a}^\top \mathbf{x}) = \mathbf{a}$ \quad
$\frac{\partial}{\partial \mathbf{x}}(\mathbf{A} \mathbf{x}) = \mathbf{A}^\top$ \quad
$\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}^\top \mathbf{A}) = \mathbf{A}$ \quad
$\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}^\top \mathbf{x}) = 2\mathbf{x}$ \quad
$\frac{\partial}{\partial \mathbf{x}}(\mathbf{x}^\top \mathbf{A}\mathbf{x}) = (\mathbf{A} + \mathbf{A}^\top)\mathbf{x}$ \\
$\frac{\partial}{\partial \mathbf{X}}(\mathbf{c}^\top \mathbf{X} \mathbf{b}) = \mathbf{c}\mathbf{b}^\top$ \quad
$\frac{\partial}{\partial \mathbf{X}}(\mathbf{c}^\top \mathbf{X}^\top \mathbf{b}) = \mathbf{b}\mathbf{c}^\top$ \\
$\frac{\partial}{\partial \mathbf{X}}(\|\mathbf{X}\|_F^2) = 2\mathbf{X}$ \quad
$\frac{\partial}{\partial \mathbf{X}}(log(det(\mathbf{X}))) = \mathbf{(X^\top)^{-1}}$\\
$\frac{\partial}{\partial \mathbf{a}}(\mathbf{(x-a)^\top W(x-a)}) = -2\mathbf{W(x-a)}$ \\
$\frac{\partial}{\partial \mathbf{X}}(\mathbf{a^\top X^{-1}b}) = \mathbf{-(X^\top)^{-1}ab^\top(X^\top)^{-1}}$ \\
$\frac{\partial}{\partial \mathbf{X}}(Tr(\mathbf{AX})) = \frac{\partial}{\partial \mathbf{X}}(Tr(\mathbf{XA})) = \mathbf{A^\top}$ \\
$\frac{\partial}{\partial \mathbf{X}}(Tr(\mathbf{AX^\top})) = \frac{\partial}{\partial \mathbf{X}}(Tr(\mathbf{X^\top A})) = \mathbf{A}$ \\
$\frac{\partial}{\partial \mathbf{X}}(Tr(\mathbf{X^\top AX})) = \mathbf{(A+A^\top)X}$ \\

\subsection*{Jacobian}
$J_F= \begin{bmatrix} \nabla_x^\top F_1\\...\\\nabla_x^\top F_m\end{bmatrix}$ where $F:\mathbb{R}^n \rightarrow \mathbb{R}^m$ for $x \in \mathbb{R}^n$

\subsection*{LinAlg}
$\bullet$ A is \textbf{psd} if $v^\top Av \geq 0$, note: if $A=B^\top B$ then $A$ is psd, psd $\Leftrightarrow \lambda \geq 0$ \\
$\bullet$ \textbf{orthogonal matrix}: $U^\top U = UU^\top = \mathbb{1}$ \\
1.unit length columns/rows 2.orthogonal columns/rows 3. square matrix \\
properties: preserve norm $\|Ux\|_2^2 = \|x\|_2^2$, $det(U) = \pm1$, full rank because invertible.\\
$\bullet$ $\langle \mathbf{x}, \mathbf{y} \rangle = \|\mathbf{x}\|_2 \cdot \|\mathbf{y}\|_2 \cdot \cos(\theta)$ \\
$\bullet$ \textbf{spectral theorem}: Matrix $A$ is diagonalizable by orthogonal matrix iff $A$ is symmetric. \\
$\bullet$ $A$ is degenerate/non-invertible $\Leftrightarrow det(A)=0$. \\
$\bullet$ \textbf{rank-nullity theorem}: $dim(kernel(A))+dim(range(A)) = n$ where $n$ is input dimension.
$\bullet$ \textbf{determinant}: $det(diag(a_1,...,a_n))=a_1*...*a_n$ \quad $det(\begin{bmatrix}a & b \\c&d\end{bmatrix})=ad-bc$ \\
$\bullet$ \textbf{cauchy-schwarz}: $x^\top y \leq \|x\|_2 \|y\|_2$ \\
$\bullet$ \textbf{trace} of a matrix is equal to the sum of its eigenvalues. Trace operator is linear and cyclic. \\
$\bullet$ eigenvectors corresponding to distinct eigenvalues of a symmetric matrix are orthogonal to each other. \\
$\bullet$ a function is convex iff its Hessian $\nabla^2$ is psd.\\
$\bullet$ $f$ convex $\Leftrightarrow \forall \lambda \in [0,1], x,y \in X. f(\lambda x + (1- \lambda)y) \leq \lambda f(x) + (1 - \lambda) f(y)$\\
$\bullet$ \textbf{jensens inequality}: $E[f(X)] \geq f(E[X])$ for $f$ convex ($\leq$ for $f$ concave). \\

\subsection*{Eigendecomposition}
$\Sigma = U\Lambda U^\top$ where $\Lambda=diag(\lambda_1,...,\lambda_m)$ with eigenvalues $\lambda_1 \geq...\geq\lambda_m$, $U=(u_1|...|u_m)$ eigenvectors orthogonal. Only exists if $\Sigma$ is symmetric.

\subsection*{Eigenvalue / -vectors}
Eigenvalue Problem: $\mathbf{Ax} = \lambda \mathbf{x}$\\
1. solve $\operatorname{det}(\mathbf{A} - \lambda \mathbb{1}) \overset{!}{=} 0$ resulting in $\{\lambda_i\}_i$\\
2. $\forall \lambda_i$:
solve $(\mathbf{A} - \lambda_i \mathbb{1}) \mathbf{x}_i = \mathbf{0}$, for $\mathbf{x}_i$.

\subsection*{Norms}
$\|\mathbf{A}\|_F =\allowbreak \sqrt{\sum_{i=1}^{m} \sum_{j=1}^{n}\mathbf{a}_{i,j}^2} = \sqrt{trace(A^\top A)} = \sqrt{trace(AA^\top)} =\allowbreak \sqrt{\sum_{i=1}^{\min\{m, n\}} \sigma_i^2}$\\
$\|\mathbf{A}\|_2 = sup\{\|Ax\|: \|x\|=1\} = \sigma_1$ \\
$\|\mathbf{M}\|_\star = \sum_{i=1}^{\min(m, n)} \sigma_i$

\subsection*{Probability / Statistics}
\begin{inparaitem}
	\item $Var(X)=E[(X-\mu)^2]$
	\item sample variance: $\frac{1}{N}\sum_{i=1}^{N}(x_i-\bar{x})^2$
\end{inparaitem}


\lab{Algorithms}{Numerical Derivatives}{Numerical Derivatives}
\label{Ch:Numerical Derivatives}

\objective{Understand and implement finite difference approximations of the derivative. Then explore Image Filters. In particular, we will explore the Sobel Filter, which uses
numerical derivatives to find edges in images.}

\section*{One Dimension}

The derivative of a function at a point is formally defined as

\begin{equation}
\label{eqn:deriv}
f'(x) = \lim_{h\rightarrow 0} \frac{f(x + h)-f(x)}{h}.
\end{equation}

In most real world applications we will be solving problems using computers. How does a computer calculate a limit? In short it can't. Computers can only approximate functions at specific points, and the notion of a limit graces infinity in a way that a computer never can.

So how can we use a computer to find the derivative of a function, particularly when we can't differentiate the function by hand? We use methods known as finite difference methods. For example suppose that in equation \ref{eqn:deriv}, instead of taking a limit we just pick a particularly small value for h. Then we have

\begin{equation*}
f'(x) \approx \frac{f(x + h)-f(x)}{h}
\end{equation*}

This is known as the first order forward difference approximation of the derivative.

How do we know the quality of this approximation? We can use Taylor's formula to find

\begin{equation*}
f(x_0 + h) = f(x_0) + hf'(x_0) + h^2/2 f''(\xi),\hspace{5mm} \xi \in (x_0,x_0 + h)
\end{equation*}

Which can be also expressed as

\begin{equation*}
f'(x_0) = \frac{f(x_0 + h) - f(x)}{h} + \frac{h}{2}f''(\xi) = \frac{f(x_0 + h) - f(x)}{h} + O(h)
\end{equation*}

Here we use the big-O notation to denote that the errors are bounded by some constant multiplied by $h$.

We can use Taylor expansions to find approximations that have different big-O error bounds, up to any polynomial of arbitrary degree. Tables \ref{Table:CDiff} and \ref{Table:FDiff} offer the coefficients for centered and forward difference schemes.

\begin{table}
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
\hline
Derivative & Accuracy & -3 & -2 & -1 & 0 & 1 & 2 & 3 \\ \hline
 & 2 & & & -1/2 & 0 & 1/2 & & \\ \cline{2-9}
 1 & 4 & & 1/12 & -2/3 &  0 & 2/3 & -1/12 & \\ \cline{2-9}
  & 6 & -1/60 & 3/20 & -3/4 & 0 & 3/4 & -3/20 & 1/60 \\ \hline
  & 2 & & & 1 & -2 & 1 & & \\ \cline{2-9}
 2 & 4 & & -1/12 & 4/3 &  -5/2 & 4/3 & -1/12 & \\ \cline{2-9}
  & 6 & 1/90 & -3/20 & 3/2 & -49/18 & 3/2 & -3/20 & 1/90 \\ \hline
\end{tabular}
\caption{Centered Difference Coefficients}
\label{Table:CDiff}
\end{center}
\end{table}

\begin{table}
\begin{center}
\begin{tabular}{|c|c|c|c|c|c|c|}
\hline
Derivative & Accuracy & 0 & 1 & 2 & 3 & 4 \\ \hline
 & 1 & -1 & 1 &  & &  \\ \cline{2-7}
 1 & 2 & -3/2 & 2 & -1/2 & &  \\ \cline{2-7}
  & 3 & -11/6 & 3 & -3/2 & 1/3 &  \\ \hline
  & 1 & 1 & -2 & 1 &  & \\ \cline{2-7}
 2 & 2 & 2 & -5 & 4 &  -1 &  \\ \cline{2-7}
  & 3 & 35/12 & -26/3 & 19/2 & -14/3 & 11/12 \\ \hline
\end{tabular}
\caption{Forward Difference Coefficients}
\label{Table:FDiff}
\end{center}
\end{table}

These tables can be used by simply summing the function evaluations (the number at the top represents how many times $h$ is added to $x$), and then dividing by $h^n$, where $n$ is the degree of the derivative.

So, for example, the centered difference estimate of the second derivative that is $O(h^4)$ is
\begin{equation*}
f''(x) \approx \frac{-1/12(f(x-2h) + f(x+2h)) + 4/3(f(x-h) + f(x+h)) -5/2f(x)}{h^2}
\end{equation*}

Or, the forward difference estimate for the first derivative that is $O(h^2)$ is

\begin{equation*}
f'(x) \approx \frac{-3/2f(x) + 2f(x+h) - 1/2 f(x+2h)}{h}
\end{equation*}

It should be noted that we can convert a forward difference estimate to a backwards difference estimate by using $-h$. So the backwards difference estimate for the first derivative that is $O(h^2)$ is

\begin{equation*}
f'(x) \approx \frac{3/2f(x) - 2f(x-h) + 1/2 f(x-2h)}{h}
\end{equation*}

There are two important observations that you should make about these tables. First, in order to get higher order approximations we need to evaluate the function at more points. This should not be surprising. Second, you should notice that centered difference formulas require less function evaluations to get higher order approximations. However, in certain applications it is not possible to use centered difference formulas, so the backwards and forwards formulas are still very applicable.

One important aspect of this method is selecting an appropriate $h$. The natural temptation is to pick a very very small value. However, this is not always advisable. Note the values in table \ref{Table:FloatingError}, which approximates the derivative of $e(x)$ at $x = 1$:

\begin{table}[h!]
\begin{center}
\begin{tabular}{|cc|}
\hline
h & Error  = $|f'(1)-f'_{app}(1)|$ \\ \hline
1e-1 & 4.5e-3 \\
1e-3 & 4.5305e-7 \\
1e-7 & 5.8587e-11 \\
1e-10 & 6.7274e-7 \\ \hline
\end{tabular}
\caption{Error in numerical derivative, using double precision floating point arithmetic}
\label{Table:FloatingError}
\end{center}
\end{table}

As you can see, the error actually increases as $h$ becomes very small. Why is this? Division by small numbers causes errors in floating point arithmetic. So, be aware that usually the optimal $h$ is of moderately small size. However, in the framework of double floating point arithmetic, this is usually less of a concern.

As a matter of reference, calculating numerical derivatives is an unstable operation. An unstable operation, informally, is one where errors are magnified by the operation. This usually is not an issue, but it's important to know that taking derivatives can amplify errors.

\begin{problem}
Write a function \li{numDer1} that accepts as inputs: a callable function object \li{f} and
a keyword argument \li{h} giving the step size (default \li{h = 1e-5}). Have the function return an array of the approximate 1st order derivative with accuracy 1 of \li{f} at each of the points in \li{pts}, using the centered  coefficients.
\end{problem}

\begin{problem}
Write a function \li{numDer2} that accepts as inputs: a callable function object \li{f} and
a keyword argument \li{h} giving the step size (default \li{h = 1e-5}). Have the function return an array of the approximate 2nd order derivative with accuracy 1 of \li{f} at each of the points in \li{pts}, using the centered  coefficients.
\end{problem}

\begin{comment}
\begin{problem}
Write a function \li{numDer} that accepts as inputs: a callable function object \li{f}, an
array of numbers \li{pts}, a keyword argument \li{mode} (taking one of the values
\li{'centered'}, \li{'forward'}, or \li{'backward'}), a keyword argument \li{d} (taking one of
the values \li{1} or \li{2}), a keyword argument \li{o} (taking an integer value in
\li{[2, 4, 6]} if \li{mode = 'centered'}, and otherwise taking a value in \li{[1, 2, 3]}), and
a keyword argument \li{h} giving the step size. The default settings of the keyword
arguments should be \li{mode = 'centered', d = 1, o = 2, h = 1e-5}.

Have the function return an array of the approximate derivative of order \li{d} with accuracy
 \li{o} of \li{f} at each of the points in \li{pts}, using the coefficients indicated by \li{mode}.
\end{problem}
\end{comment}

We note that higher order approximations of the derivative can be derived using the Taylor series and Lagrange polynomials, but generally higher-order approximations are not practically useful as they can often be ill-conditioned.

For any numerical approximation method, it is important to be able to empirically calculate
the order of convergence. We will do so for our numerical derivative approximation.
For an $m$-th order approximation of the first derivative, we have that the error is in
$O(h^m)$, so that
$$
err(h) \approx Ch^m
$$
for some constant $C$.
Hence, taking the log of both sides, we obtain
$$
\log err(h) \approx \log C + m\log h,
$$
which means that if we plot the log of the errors against the log of the $h$ values, we
ought to see a linear relationship whose slope gives the rate of convergence.

We do this in python as follows (The \li{numDer} function in this code can use any of the coefficients from \ref{Table:CDiff} and \ref{Table:FDiff}):
\begin{lstlisting}
# assume that the function numDer has already been written
import numpy as np
from matplotlib import pyplot as plt

# approximate the derivative of cosine at x = 3
# create a callable function object
def myCosine(x):
    return np.cos(x)
f = myCosine

# calculate the actual derivative
actual = -np.sin(3.0)

# initialize array of h values at which to calculate the error
hvals = np.linspace(1e-5, 1e-1)
err1 = np.zeros(hvals.shape)
err2 = np.zeros(hvals.shape)

# calculate the errors for order 1 and order 2 approximations for the forward coeffiects
for i in xrange(len(hvals)):
    err1[i] = np.abs(actual - numDer(f, np.array([3.0]), mode = 'forward', h = hvals[i], o=1))
    err2[i] = np.abs(actual - numDer(f, np.array([3.0]), mode = 'forward', h = hvals[i], o=2))

# plot the log of the h values against the log of the errors
plt.subplot(121)
plt.loglog(hvals, err1)
plt.ylim((1e-11, 1e-1))
plt.subplot(122)
plt.loglog(hvals, err2)
plt.ylim((1e-11, 1e-1))
plt.show()
\end{lstlisting}

The generated plot is shown in Figure \ref{fig:convergence}. Note that the slope
of the line in the left plot is about 1, and the slope of the line in the right
plot is about 2. Further, the log of the errors for the order 2 approximations
are much lower than the log of the errors of the order 1 approximations.

\begin{figure}[t]
    \includegraphics[width=0.8\textwidth]{convergence.pdf}
    \caption{Convergence plots for our numerical derivative approximations.
    The left plot shows the convergence for order 1 approximations.
    The right plot shows the convergence for order 2 approximations.}
    \label{fig:convergence}
\end{figure}

\begin{comment}
\begin{problem}
Explore the convergence properties for different orders of approximation,
and for the second derivative as well. You may need to adjust your $h$
values, as they may be too small for some of the calculations.
\end{problem}
\end{comment}

\section*{Multi Dimensions}

The Jacobian is a generalization of the derivative in many dimensions. We can think of it,
intuitively, as defining a linear approximation to the function in a neighborhood of some given
point. If the function happens to be real-valued, the Jacobian defines a tangent plane to the graph of
the function at a point.
The Jacobian is of critical importance in a variety of areas, and we will use it in lab \ref{lab:NewtonsMethod}
to find zeros of multivariate functions.

Formally, the Jacobian of a function $f:\mathbb{R}^n \rightarrow \mathbb{R}^m$ is an $m \times n$ matrix. It is defined by the formula:
\begin{equation*}
J_{ij} = \frac{\partial f_i}{\partial x_j}
\end{equation*}

We can use finite difference approximations to find partial derivatives in the natural manner:
\begin{equation*}
\frac{\partial f}{\partial x_i} (x) \approx \frac{f(x+h e_i)-f(x)}{h}
\end{equation*}
where $e_i$ is a unit vector in the $i$th coordinate (the direction $x_i$). Higher order approximations and centered and backwards differences follow by naturally extending this definition.

\begin{problem}
Write a function \li{Jacobian} that accepts a function handle, an integer giving the dimension of the
range of the function, an integer giving the dimension of the domain of the function, a NumPy array giving
the point at which to approximate the Jacobian, and an optional argument giving the step size $h$. Use center approximation of order 1.
Return a NumPy array giving the approximated Jacobian matrix at the given point.

If the function under consideration maps from $\mathbb{R}^n$ to $\mathbb{R}^m$, the function handle should
accept as input an array of shape \li{(n,)} and return an array of shape \li{(m,)}.

Test your function on the following $f: \mathbb{R}^2 \to \mathbb{R}^2$:
\begin{equation*}
f(x, y) =
\begin{pmatrix}
e^{x} \sin(y) + y^3 \\
3y - \cos(x)
\end{pmatrix}
\end{equation*}

Compare your \li{Jacobian} function against the analytically computed derivative on the square $[-1,1] \times [-1,1]$ using ten thousand grid points (100 per side). Which method is faster? What is the maximum error of your function?
\end{problem}

Given a function from $\mathbb{R}^n \to \mathbb{R}$, sometimes the mixed partial derivatives are useful. In particular, the mixed partials will be useful when we study optimization in Volume 2. This information is contained in the Hessian matrix, which is defined as

\begin{equation*}
H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}
\end{equation*}

We can use the following formula to approximate mixed partial derivatives
\small
\begin{equation*}
\frac{\partial^2 f}{\partial x_i \partial x_j} = \frac{f(x + (e_i + e_j)h) - f(x + (e_i-e_j)h) -f(x + (e_j-e_i)h) + f(x - (e_i + e_j)h)}{4h^2}
\end{equation*}
\normalsize

\begin{problem}
Write a Python function that numerically calculates the Hessian of a given function.
The function should be named \li{Hessian}, and should accept as inputs a function handle,
an integer giving the dimension of the domain of the function, a NumPy
array giving the point at which to approximate the Hessian, and an optional argument giving the
step size.
Return the approximated Hessian matrix at the given point.

Test it on the following function
\begin{equation*}
f(x,y) = (1-x)^2 + 100(y-x^2)^2
\end{equation*}
This function is known as the Rosenbrock Banana function, or Rosenbrock's Valley. It is a common test function for optimization algorithms because it is non-convex and the global minimum is hard to find from certain starting points. A graph is shown in figure \ref{Fig:Rosenbrock}. Compare the output of your function with the analytic solution on the region $[-2,2] \times [0,2]$, using ten thousand points. What is the maximum error of your function?
\end{problem}
\begin{figure}
\begin{center}
\includegraphics[width = \textwidth]{Rosenbrock}
\caption{The Rosenbrock Banana Function, a common test function in optimization algorithms}
\label{Fig:Rosenbrock}
\end{center}
\end{figure}

\section*{Application: Image Filters}
One of the primary tools in image processing is the use of filters to identify important information
in images. In this section we will implement our own filters and explore applications.

One of the easiest filters to implement is matrix based. In this implementation, the filter that we
select is a matrix, which represents how much we want certain pixels to contribute to the output image.
Suppose that we choose our filter to be:

\[
A = \begin{pmatrix}
a_{-1,-1}&a_{-1,0}&a_{-1,1}\\
a_{0,-1}&a_{0,0}&a_{0,1}\\
a_{1,-1}&a_{1,0}&a_{1,1}
\end{pmatrix}.
\]

We will call our input image $B$ and our output image $C$ (both images are represented as matrices).
Our matrix filter operates according to the following formula:
\[
C_{ij} = \sum_{k=-1}^1 \sum_{m=-1}^1 a_{km}B_{i+k,j+m}.
\]

Essentially this computes each pixel of the output image as the weighted sum of the surrounding
pixels in the input image. The technical term for this type of operation is a convolution.

Suppose that our image $B$ is an $m\times n$ matrix. Note that the definition for
$C_{ij}$ given above does not make sense when $i = 0, m-1$ or $j = 0, n-1$, since we would be trying
to access matrix elements $B_{-1,j}, B_{m,j}, B_{i,-1},$ or $B_{i,n}$, which are out of bounds.
%One fix is to treat these out-of-bounds elements as copies of the nearest in-bounds element.
%That is, define
%\begin{align*}
%B_{-1,j} &:= B_{0,j},\\
%B_{m,j} &:= B_{m-1,j},\\
%B_{i,-1} &:= B_{i,0},\\
%B_{i,n} &:= B_{i,n-1}.
%\end{align*}
%For the corners, define
%\begin{align*}
%B_{-1,-1} &:=B_{0,0},\\
%B_{m,-1}&:= B_{m-1,0},\\
%B_{-1,n}&:=B_{0,n-1},\\
%B_{m,n}&:=B_{m-1,n-1}.
%\end{align*}
One fix is to assign the value of 0 to these out-of-bounds elements. This process is know as \emph{padding}
the image with zeros. The amount of padding necessary is dependent on the size of the filter.
In the following code, we create a padded version of the image $B$ with respect to a filter $f$ of shape
$h\times k$, where we assume that $h$ and $k$ are odd integers.

\begin{lstlisting}
>>> import numpy as np
>>> # assume that the matrix B and filter f already exist.
>>> # create padded version of B.
>>> m, n = B.shape
>>> h, k = f.shape
>>> B_pad = np.zeros((m + h - 1, n + k - 1))
>>> B_pad[h/2:h/2 + m, k/2:k/2 + n] = B
\end{lstlisting}

We can now easily calculate a given element of the filtered image $C$ as follows:
\begin{lstlisting}
>>> C[i,j] = (f*B_pad[i:i + h, j:j + k]).sum()
\end{lstlisting}

Make sure that you understand why this Python code does what we want it to do.
\begin{problem}
Write a function \li{Filter} that takes an image and an arbitrary filter matrix, and outputs the
filtered image. Note that for these filters to be meaningful, the number of columns and number
of rows need to be odd (there are ways to deal with even size filters, but we won't handle them here).
\end{problem}

So what can we do with these filters? We will show an example of how to apply a Gaussian blur
using this type of filter. You will also be able to test your code with this example.
To begin, load the following picture:

\begin{lstlisting}
>>> import numpy as np
>>> from matplotlib import pyplot as plt
>>> K = plt.imread('cameraman.tif')
>>> plt.imshow(K, cmap = plt.cm.Greys_r)
>>> plt.show()
\end{lstlisting}

This code should display an image like the one shown in Figure \ref{imfil:camclean} (provided
that you have the correct image file, of course).

\begin{figure}
\includegraphics{cameramanClean.pdf}
\caption{An example image.}
\label{imfil:camclean}
\end{figure}

Now that you have a simple filter function, try the following filter on this image:

\[
A = \frac{1}{159}\begin{pmatrix}
2&4&5&4&2\\
4&9&12&9&4\\
5&12&15&12&5\\
4&9&12&9&4\\
2&4&5&4&2
\end{pmatrix}
\]

\begin{lstlisting}
>>> A = np.array([[2,4,5,4,2],
                 [4,9,12,9,4],
                 [5,12,15,12,5],
                 [4,9,12,9,4],
                 [2,4,5,4,2]])/159.
>>> C = Filter(K, A)
>>> plt.imshow(C, cmap = plt.cm.Greys_r)
>>> plt.show()
\end{lstlisting}
You should get an output that looks something like Figure \ref{imfil:camblur}.
\begin{figure}
\includegraphics{cameramanBlur.pdf}
\caption{A blurred version of Figure \ref{imfil:camclean}.}
\label{imfil:camblur}
\end{figure}
You can see that the filter blurred the image. This can be very important in trying to
wash out images that are ``noisy.''

We now turn our attention to the problem of edge detection. Automatically detecting edges in an image
can be useful in segmenting the image, detecting and extracting certain features in the image, and
sharpening contrast in the image. There are many approaches to this problem, and in this lab, we will
design a filter that numerically approximates the gradient of the image at each pixel. The magnitude
of the gradient tells us the local rate of change of the pixel values, and so large magnitudes
correspond to regions of high contrast in the image. Since there is high contrast at edges within
the image, we have reason to hope that this approach will be effective.

The filter we will use is called the \emph{Sobel Filter}. To find the gradient
in the vertical direction, the filter is given by:
\[
A = \frac{1}{8}\begin{pmatrix}
-1&-2&-1\\
0&0&0\\
1&2&1
\end{pmatrix}
\]
The filter for the horizontal gradient is simply the transpose of the above matrix.

For an image $B$, suppose that filtered images using the Sobel filter in the vertical and
horizontal directions are given by $B_y$ and $B_x$, respectively. These two matrices give
the $y$ and $x$ components of the gradient, respectively, at each pixel. To obtain the
magnitude of the gradient (i.e. the length of the gradient vector), we use the formula
$$
B_{grad} = \sqrt{B_x^2 + B_y^2},
$$
where the square root and squaring operations are performed pointwise on the matrices.
The matrix $B_{grad}$ now gives the magnitude of the gradient at each pixel, and so we
can threshold this matrix to isolate the pixels with the largest gradient, setting all
other pixel values to 0.
In Python, this can be done as follows:
\begin{lstlisting}
>>> B_edges = B_grad > thresh
\end{lstlisting}
where \li{thresh} is some positive threshold value of our choosing. In general, a reasonable
value for this threshold is $4\mu(B_{grad})$, where $\mu(B_{grad})$ is the mean value of the
matrix $B_{grad}$.
If we now plot the matrix \li{B_edges}, we will see a black-and-white image with the edges
shown as white lines.

\begin{problem}
Write a function \li{plotEdges} that finds and plots the edges of an image using the Sobel filter.
The function should take an image, and the last line of code should be a call
to \li{plt.show}. The function should not return anything. For your threshold value, use the
suggestion given above.

For the cameraman example, you should obtain an output similar to that displayed in
Figure \ref{imfil:edges}.
\end{problem}

\begin{figure}[h!]
\includegraphics{edges.pdf}
\caption{A filtered version of Figure \ref{imfil:camclean}.}
\label{imfil:edges}
\end{figure}


\section{Maxima and Minima}\label{sec:MultivariateMaximaMinima}

Suppose a surface given by $f(x,y)$ has a local maximum at
$(x_0,y_0,z_0)$; geometrically, this point on the surface looks like
the top of a hill. If we look at the cross-section in the plane
$y=y_0$, we will see a local maximum on the curve at $(x_0,z_0)$, and
we know from single-variable calculus that ${\partial z\over\partial x}=0$
at this point. Likewise, in the plane $x=x_0$, ${\partial
  z\over\partial y}=0$. So if there is a local maximum at
$(x_0,y_0,z_0)$, both partial derivatives at the point must be zero,
and likewise for a local minimum. Thus, to find local maximum and
minimum points, we need only consider those points at which both
partial derivatives are 0. As in the single-variable case, it is
possible for the derivatives to be 0 at a point that is neither a
maximum or a minimum, so we need to test these points further.

You will recall that in the single variable case, we examined three
methods to identify maximum and minimum points; the most useful is the
second derivative test, though it does not always work. For functions
of two variables there is also a second derivative test; again it is
by far the most useful test, though it doesn't always work.

\begin{theorem}{Extrema Test for Multivariate Functions}{}
Suppose that the second partial derivatives of $f(x,y)$ are
continuous near $(x_0,y_0)$, and $f_x(x_0,y_0)=f_y(x_0,y_0)=0$.
We denote by $D$ the \dfont{discriminant}\index{discriminant}
$D(x_0,y_0)=f_{xx}(x_0,y_0)f_{yy}(x_0,y_0)-f_{xy}(x_0,y_0)^2$.
If $D>0$ and $f_{xx}(x_0,y_0)<0$ there is a local maximum at $(x_0,y_0)$;
if $D>0$ and $f_{xx}(x_0,y_0)>0$ there is a local minimum at $(x_0,y_0)$;
if $D<0$ there is neither a maximum nor a minimum at $(x_0,y_0)$;
if $D=0$, the test fails.\index{local extrema!multivariate}
\end{theorem}

\begin{example}{Extrema on an Elliptic Paraboloid}{}
Verify that $f(x,y)=x^2+y^2$ has a minimum at $(0,0)$.
\end{example}
\begin{solution}
First, we compute all the needed derivatives:
$$f_x=2x \qquad f_y=2y \qquad f_{xx}=2 \qquad f_{yy}=2 \qquad
f_{xy}=0.$$
The derivatives $f_x$ and $f_y$ are zero only at $(0,0)$. Applying the
second derivative test there:
$$D(0,0)=f_{xx}(0,0)f_{yy}(0,0)-f_{xy}(0,0)^2=
2\cdot2-0=4>0,$$
so there is a local minimum at $(0,0)$, and there are no other
possibilities.
\end{solution}

\begin{example}{Extrema on a Hyperbolic Paraboloid}{}
Find all local maxima and minima for $f(x,y)=x^2-y^2$.
\end{example}
\begin{solution}
The derivatives:
$$f_x=2x \qquad f_y=-2y \qquad f_{xx}=2 \qquad f_{yy}=-2 \qquad
f_{xy}=0.$$
Again there is a single critical point, at $(0,0)$, and
$$D(0,0)=f_{xx}(0,0)f_{yy}(0,0)-f_{xy}(0,0)^2=
2\cdot-2-0=-4<0,$$
so there is neither a maximum nor minimum there, and so there are no
local maxima or minima. The surface is shown in
Figure~\ref{fig:saddle}.
\end{solution}

\begin{figure}[H]
%\texonly
\centerline{
\vbox{\beginpicture
\normalgraphs
%\ninepoint
\setcoordinatesystem units <3truecm,3truecm>
\setplotarea x from -1 to 1, y from 0 to 1
\put {\hbox{\epsfxsize8cm\epsfbox{images/saddle.eps}}} at 0 -0.15
\endpicture}}
%\endtexonly
\caption{A saddle point, neither a maximum nor a minimum.}
\label{fig:saddle}
\end{figure}

\begin{example}{Finding Extrema}{}
Find all local maxima and minima for $f(x,y)=x^4+y^4$.
\end{example}
\begin{solution}
The derivatives:
$$f_x=4x^3 \qquad f_y=4y^3 \qquad f_{xx}=12x^2 \qquad f_{yy}=12y^2 \qquad
f_{xy}=0.$$
Again there is a single critical point, at $(0,0)$, and
$$D(0,0)=f_{xx}(0,0)f_{yy}(0,0)-f_{xy}(0,0)^2=
0\cdot0-0=0,$$
so we get no information. However, in this case it is easy to see that
there is a minimum at $(0,0)$, because $f(0,0)=0$ and
at all other points $f(x,y)>0$.
\end{solution}

\begin{example}{Finding Extrema}{}
Find all local maxima and minima for $f(x,y)=x^3+y^3$.
\end{example}
\begin{solution}
The derivatives:
$$f_x=3x^2 \qquad f_y=3y^2 \qquad f_{xx}=6x^2 \qquad f_{yy}=6y^2 \qquad
f_{xy}=0.$$
Again there is a single critical point, at $(0,0)$, and
$$D(0,0)=f_{xx}(0,0)f_{yy}(0,0)-f_{xy}(0,0)^2=
0\cdot0-0=0,$$
so we get no information. In this case, a little thought shows there
is neither a maximum nor a minimum at $(0,0)$: when $x$ and $y$ are
both positive, $f(x,y)>0$, and when $x$ and $y$ are
both negative, $f(x,y)<0$, and there are points of both kinds
arbitrarily close to $(0,0)$. Alternately, if we look at the
cross-section when $y=0$, we get $f(x,0)=x^3$, which does not have
either a maximum or minimum at $x=0$.
\end{solution}

\begin{example}{Optimizing Dimensions of a Box}{}
Suppose a box with no top is to hold a certain volume $V$. Find
the dimensions for the box that result in the minimum surface area.
\end{example}
\begin{solution}
The area of the box is $A=2hw+2hl+lw$, and the volume is $V=lwh$, so
we can write the area as a function of two variables,
$$A(l,w)={2V\over l}+{2V\over w}+lw.$$
Then
$$A_l=-{2V\over l^2}+w \hbox{\quad and\quad} A_w=-{2V\over w^2}+l.$$
If we set these equal to zero and solve, we find
$\ds w=(2V)^{1/3}$ and $\ds l=(2V)^{1/3}$, and the corresponding
height is $h=V/(2V)^{2/3}$.

The second derivatives are
$$A_{ll}={4V\over l^3}\qquad A_{ww}={4V\over w^3}\qquad
A_{lw}=1,$$
so the discriminant is
$$D={4V\over l^3}{4V\over w^3}-1=4-1=3>0.$$
Since $A_{ll}$ is 2, there is a local minimum at the critical point.
Is this a global minimum? It is, but it is difficult to see this
analytically; physically and graphically it is clear that there is a
minimum, in which case it must be at the single critical point.
%\texonly
%%\expandafter\url\expandafter{\sageurl critical_point_on_surface}Here\endurl\
%\expandafter\url\expandafter{\liveurl jmol_surface_minimum}Here\endurl\
%is the graph as rendered by Sage, as an example.
%\endtexonly
%\htmlonly
%Here is the graph as rendered by Sage, as an example.
%\endhtmlonly
%\htmlfigure{Partial_Differentiation_minimum_point_on_surface.html}
%Note that we must
%choose a value for $V$ in order to graph it.
\end{solution}

Recall that when we did single variable global maximum and minimum
problems, the easiest cases were those for which the variable could be
limited to a finite closed interval, for then we simply had to check
all critical values and the endpoints. The previous example is
difficult because there is no finite boundary to the domain of the
problem---both $w$ and $l$ can be in $(0,\infty)$. As in the single
variable case, the problem is often simpler when there is a finite
boundary.

\begin{theorem}{Multivariate Absolute Extrema}{}
If $f(x,y)$ is continuous on a closed and bounded subset of
$\R^2$, then it has both a maximum and minimum value.\index{absolute extrema!multivariate}
\end{theorem}

As in the case of single variable functions, this means that the
maximum and minimum values must occur at a critical point or on the
boundary; in the two variable case, however, the boundary is a curve,
not merely two endpoints.

\begin{example}{Optimizing Volume of a Box}{box diagonal}
The length of the diagonal of a box is to be 1 meter; find the
maximum possible volume.
\end{example}
\begin{solution}
If the box is placed with one corner at the origin, and sides along
the axes, the length of the diagonal is $\ds\sqrt{x^2+y^2+z^2}$, and
the volume is
$$V=xyz=xy\sqrt{1-x^2-y^2}.$$
Clearly, $x^2+y^2\le 1$, so the domain we are interested in
is the quarter of the unit disk in the first quadrant.
Computing derivatives:
\begin{align*}
V_x&={y-2yx^2-y^3\over\sqrt{1-x^2-y^2}}	\\
V_y&={x-2xy^2-x^3\over\sqrt{1-x^2-y^2}}	\\
\end{align*}
If these are both 0, then $x=0$ or $y=0$, or $x=y=1/\sqrt3$. The boundary of
the domain is composed of three curves: $x=0$ for $y\in[0,1]$; $y=0$
for $x\in[0,1]$; and $x^2+y^2=1$, where $x\ge0$ and $y\ge0$. In all
three cases, the volume  $xy\sqrt{1-x^2-y^2}$ is 0, so the maximum
occurs at the only critical point $(1/\sqrt3,1/\sqrt3,1/\sqrt3)$. See
Figure~\ref{fig:max volume}.
\end{solution}

\begin{figure}[H]
%\texonly
\centerline{
\vbox{\beginpicture
\normalgraphs
%\ninepoint
\setcoordinatesystem units <3truecm,3truecm>
\setplotarea x from -1 to 1, y from 0 to 1
\put {\hbox{\epsfxsize8cm\epsfbox{images/max_volume.eps}}} at 0 0
\endpicture}}
%\endtexonly
\caption{The volume of a box with fixed length diagonal.}
\label{fig:max volume}
\end{figure}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\Opensolutionfile{solutions}[ex]
\section*{Exercises for \ref{sec:MultivariateMaximaMinima}}

\begin{enumialphparenastyle}

\begin{ex}
Find all local maximum and minimum points of
$f=x^2+4y^2-2x+8y-1$.
\begin{sol}
minimum at $(1,-1)$
\end{sol}
\end{ex}

\begin{ex}
Find all local maximum and minimum points of
$f=x^2-y^2+6x-10y+2$.
\begin{sol}
none
\end{sol}
\end{ex}

\begin{ex}
Find all local maximum and minimum points of
$f=xy$.
\begin{sol}
none
\end{sol}
\end{ex}

\begin{ex}
Find all local maximum and minimum points of
$f=9+4x-y-2x^2-3y^2$.
\begin{sol}
maximum at $(1,-1/6)$
\end{sol}
\end{ex}

\begin{ex}
Find all local maximum and minimum points of
$f=x^2+4xy+y^2-6y+1$.
\begin{sol}
none
\end{sol}
\end{ex}

\begin{ex}
Find all local maximum and minimum points of
$f=x^2-xy+2y^2-5x+6y-9$.
\begin{sol}
minimum at $(2,-1)$
\end{sol}
\end{ex}

\begin{ex}
Find the absolute maximum and minimum points of
$f=x^2+3y-3xy$ over the region bounded by
$y=x$, $y=0$, and $x=2$.
\begin{sol}
$f(2,2)=-2$, $f(2,0)=4$
\end{sol}
\end{ex}

\begin{ex}
A six-sided rectangular box is to hold $1/2$ cubic meter;
what shape should the box be to minimize surface area?
\begin{sol}
a cube $1/\root 3 \of {2}$ on a side
\end{sol}
\end{ex}

\begin{ex}
The post office will accept packages whose combined length
and girth is at most 130 inches. (Girth is the maximum distance around
the package perpendicular to the length; for a rectangular box, the
length is the largest of the three dimensions.) What is the largest volume
that can be sent in a rectangular box?
\begin{sol}
$65/3\times 65/3\times 130/3$
\end{sol}
\end{ex}

\begin{ex}
The bottom of a rectangular box costs twice as much per unit
area as the sides and top. Find the shape for a given volume that will
minimize cost.
\begin{sol}
It has a square base, and is one and one half times as tall as wide.
If the volume is $V$ the dimensions are $\root 3 \of {2V/3}\times
\root 3 \of {2V/3}\times \root 3\of {9V/4}$.
\end{sol}
\end{ex}

\begin{ex}
Using the methods of this section, find the shortest
distance from the origin to the plane $x+y+z=10$.
\begin{sol}
$\sqrt{100/3}$
\end{sol}
\end{ex}

\begin{ex}
Using the methods of this section, find the shortest
distance from the point $(x_0,y_0,z_0)$ to the plane $ax+by+cz=d$.
You may assume that $c\not=0$; use of Sage or similar software
is recommended.
\begin{sol}
$|ax_0+by_0+cz_0-d|/\sqrt{a^2+b^2+c^2}$
\end{sol}
\end{ex}

\begin{ex}
A trough is to be formed by bending up two sides of a long
metal rectangle
so that the cross-section of the trough is an isosceles trapezoid.
If the width of the metal sheet is 2
meters, how should it be bent to maximize the volume of the trough?
\begin{sol}
The sides and bottom should all be $2/3$ meter, and the sides
should be bent up at angle $\pi/3$.
\end{sol}
\end{ex}

\begin{ex}
Given the three points $(1,4)$, $(5,2)$, and $(3,-2)$,
$\ds(x-1)^2+(y-4)^2+(x-5)^2+(y-2)^2+(x-3)^2+(y+2)^2$
is the sum of the squares of the distances from point $(x,y)$ to the
three points. Find $x$ and $y$ so that this quantity is minimized.
\begin{sol}
$(3,4/3)$
\end{sol}
\end{ex}

\begin{ex}
Suppose that $f(x,y)=x^2+y^2+kxy$. Find and classify the
  critical points, and discuss how they change when $k$ takes on
  different values.
\end{ex}

\begin{ex}
Find the shortest distance from the point $(0,b)$ to the
  parabola $y=x^2$.
\begin{sol}
$|b|$ if $b\le1/2$, otherwise $\ds\sqrt{b-1/4}$
\end{sol}
\end{ex}

\begin{ex}
Find the shortest distance from the point $(0,0,b)$ to the
  paraboloid $z=x^2+y^2$.
\begin{sol}
$|b|$ if $b\le1/2$, otherwise $\ds\sqrt{b-1/4}$
\end{sol}
\end{ex}

\begin{ex}
Consider the function $f(x,y)=x^3-3x^2y+y^3$.

\begin{enumerate}
	\item Show that $(0,0)$ is the only critical point of $f$.
	\item Show that the discriminant test is inconclusive for $f$.
	\item Determine the cross-sections of $f$ obtained by setting $y=kx$ for
	  various values of $k$.
	\item What kind of critical point is $(0,0)$?
\end{enumerate}
\end{ex}

\begin{ex}
Find the volume of the largest rectangular box with edges
  parallel to the axes that can be inscribed in the ellipsoid
  $2x^2+72y^2+18z^2=288$.
\begin{sol}
$\ds 1024/\sqrt3$
\end{sol}
\end{ex}

\end{enumialphparenastyle}


\section{The Hopf fibration}

Our goal in this section is to construct the \define{Hopf fibration}. The Hopf fibration is a fiber sequence
\begin{equation*}
\sphere{1}\hookrightarrow\sphere{3}\twoheadrightarrow\sphere{2}.
\end{equation*}
More generally, we show that for any type $A$ equipped with a multiplicative operation $\mu:A\to (A\to A)$ for which $\mu(x,\blank)$ and $\mu(\blank,x)$ are equivalences, there is a fiber sequence
\begin{equation*}
  A\hookrightarrow \join{A}{A}\twoheadrightarrow \susp A.
\end{equation*}
The construction of this fiber sequence is known as the \define{Hopf construction}. We then get the Hopf fibration from the Hopf construction by using the multiplication on $\sphere{1}$ constructed in \cref{sec:mulcircle} after we show that $\eqv{\join{\sphere{1}}{\sphere{1}}}{\sphere{3}}$.

We then introduce the long exact sequence of homotopy groups. The long exact sequence is an important tool to compute homotopy groups which applies to any fiber sequence
\begin{equation*}
  F \hookrightarrow E \twoheadrightarrow B.
\end{equation*}
In the case of the Hopf fibration, we will use the long exact sequence to show that
\begin{equation*}
  \pi_k(\sphere{3})=\pi_k(\sphere{2})
\end{equation*}
for any $k\geq 3$.

Since the Hopf fibration is closely related to the multiplication operation of the complex numbers on the unit circle, the Hopf fibration is sometimes also called the \emph{complex} Hopf fibration. Indeed, there is also a \emph{real} Hopf fibration
\begin{equation*}
  \sphere{0}\hookrightarrow\sphere{1}\twoheadrightarrow\sphere{1}.
\end{equation*}
This is just the double cover of the circle. There is even a \emph{quaternionic} Hopf fibration
\begin{equation*}
  \sphere{3}\hookrightarrow\sphere{7}\twoheadrightarrow\sphere{4},
\end{equation*}
which uses the multiplication of the quaternionic numbers on the unit sphere. The main difficulty in defining the quaternionic Hopf fibration in homotopy type theory is to define the quaternionic multiplication
\begin{equation*}
  \mulsphere{3} : \sphere{3}\to(\sphere{3}\to\sphere{3}).
\end{equation*}
The construction of the octonionic Hopf fibration
\begin{equation*}
  \sphere{7}\hookrightarrow\sphere{15}\twoheadrightarrow\sphere{8}
\end{equation*}
in homotopy type theory is still an open problem. Another open problem is to formalize Adams' theorem \cite{Adams58} in homotopy type theory, that there are \emph{no} further fiber sequences of the form
\begin{equation*}
  \sphere{k}\hookrightarrow\sphere{l}\twoheadrightarrow\sphere{m},
\end{equation*}
for $k,l,m\geq 0$.

\subsection{Fiber sequences}

\begin{defn}
  A \define{short sequence} of maps into a pointed type $B$ with base point $b$ consists of maps
  \begin{equation*}
    \begin{tikzcd}
      F \arrow[r,"i"] & E \arrow[r,"p"] & B
    \end{tikzcd}
  \end{equation*}
  equipped with a homotopy $p\circ i\htpy \const_b$. We say that a short sequence as above is an \define{unpointed fiber sequence} if the commuting square
  \begin{equation*}
    \begin{tikzcd}[column sep=large]
      F \arrow[r,"i"] \arrow[d,swap,"\const_\star"] & E \arrow[d,"p"] \\
      \unit \arrow[r,swap,"\const_b"] & B
    \end{tikzcd}
  \end{equation*}
  is a pullback square.
\end{defn}

\begin{defn}
  A \define{short sequence} of pointed maps into a pointed type $B$ with base point $b$ consists of pointed maps
    \begin{equation*}
    \begin{tikzcd}[column sep=large]
      F \arrow[r,"i"] & E \arrow[r,"p"] & B
    \end{tikzcd}
  \end{equation*}
  equipped with a pointed homotopy $p\circ i\htpy_\ast \const_b$. We say that a short sequence as above is an \define{fiber sequence} if the commuting square
  \begin{equation*}
    \begin{tikzcd}
      F \arrow[r,"i"] \arrow[d,swap,"\const_\star"] & E \arrow[d,"p"] \\
      \unit \arrow[r,swap,"\const_b"] & B
    \end{tikzcd}
  \end{equation*}
  is a pullback square.
\end{defn}

\subsection{The Hopf construction}

The Hopf construction is a general construction of a fiber sequence
\begin{equation*}
  A \hookrightarrow \join{A}{A}\twoheadrightarrow \susp A,
\end{equation*}
that applies to any H-space $A$. Our definition of an H-space is chosen such that it provides only the necessary structure to apply the Hopf construction. We give an unpointed and a pointed variant, and moreover we give a coherent variant that is more closely related to the traditional definition of an H-space.

\begin{defn}
  ~
  \begin{enumerate}
  \item An \define{unpointed H-space} structure on a type $A$ consists of a multiplicative operation
  \begin{equation*}
    \mu:A\to(A\to A)
  \end{equation*}
  such that $\mu(x,\blank)$ and $\mu(\blank,x)$ are equivalences, for each $x:A$.
  \item If $A$ is a pointed type with base point $e:A$, then an \define{H-space} structure on $A$ is an unpointed H-space structure on $A$ equipped with an identification $\mu(e,e)=e$.
  \item A \define{coherent H-space} structure on a pointed type $A$ with base point $e:A$ consists of an unpointed H-space structure $\mu$ on $A$ that satisfies the unit laws, i.e., $\mu$ comes equipped with identifications
  \begin{align*}
    \leftunit_\mu & : \mu(e,a) = a \\
    \rightunit_\mu & : \mu(a,e) = a \\
    \cohunit_\mu & : \leftunit_\mu(e)=\rightunit_\mu(e).
  \end{align*}
  \end{enumerate}
\end{defn}

\begin{eg}
  The loop space $\loopspace{A}$ of any pointed type is a coherent H-space, where the multiplication is given by path concatenation.
\end{eg}

By an unpointed fiber sequence, we mean a sequence
\begin{equation*}
  \begin{tikzcd}
    F \arrow[r,hook,"i"] & E \arrow[r,->>,"p"] & B
  \end{tikzcd}
\end{equation*}
where only the type $B$ is assumed to be pointed (with base point $b$), and the square
\begin{equation*}
  \begin{tikzcd}
    F \arrow[r,"i"] \arrow[d,swap,"\const_{\star}"] & E \arrow[d,"p"] \\
    \unit \arrow[r,swap,"\const_{b}"] & B
  \end{tikzcd}
\end{equation*}
is a pullback square.

\begin{thm}[The Hopf construction]\label{thm:hopf-construction}
  Consider a type $A$ equipped with an H-space structure $\mu$. Then there is an unpointed fiber sequence
  \begin{equation*}
    A \hookrightarrow \join{A}{A} \twoheadrightarrow \susp A.
  \end{equation*}
  If $A$ and the $H$-space structure are pointed, then this unpointed fiber sequence is an fiber sequence.
\end{thm}

\begin{proof}
  Note that there is a unique map $h:\join{A}{A}\to \susp A$ such that the cube
  \begin{equation*}
    \begin{tikzcd}
      & A \times A \arrow[d,swap,"\mu"] \arrow[dl,swap,"\proj 1"] \arrow[dr,"\proj 2"] \\
      A \arrow[d] & A \arrow[dl] \arrow[dr] & A \arrow[dl,crossing over] \arrow[d] \\
      \unit \arrow[dr] & \join{A}{A} \arrow[from=ul,crossing over] \arrow[d,swap,densely dotted,"h"] & \unit \arrow[dl] \\
      & \susp A
    \end{tikzcd}
  \end{equation*}
  commutes. Thus we see that we obtain a fiber sequence $A\hookrightarrow\join{A}{A}\twoheadrightarrow\susp A$ if we show that the front two squares are pullback squares. By the descent theorem, \cref{thm:descent}, it suffices to show that the two squares in the back
    \begin{equation*}
    \begin{tikzcd}
      A\times A \arrow[d,swap,"\proj 1"] \arrow[r,"\mu"] & A \arrow[d] &[2em] A\times A \arrow[r,"\mu"] \arrow[d,swap,"\proj 2"] & A \arrow[d] \\
      A \arrow[r] & \unit & A \arrow[r] & \unit
    \end{tikzcd}
  \end{equation*}
  are pullback squares. We claim that in both squares, the multiplicative operation $\mu$ induces equivalences on the fibers, and hence both squares are pullbacks by \cref{cor:pb_fibequiv}. To see this, note that the induced map on fibers fit in commuting squares
  \begin{equation*}
    \begin{tikzcd}
      \fib{\proj 1}{x} \arrow[r,densely dotted] \arrow[d,swap,"\simeq"] & \fib{\const_{\star}}{\star} \arrow[d,"\simeq"] & \fib{\proj 2}{x} \arrow[r,densely dotted] \arrow[d,swap,"\simeq"] & \fib{\const_{\star}}{\star} \arrow[d,"\simeq"] \\
      A \arrow[r,swap,"{\mu(x,\blank)}"] & A & A \arrow[r,swap,"{\mu(\blank,x)}"] & A.
    \end{tikzcd}
  \end{equation*}
  The claim now follows, since we have assumed that $\mu(x,\blank)$ and $\mu(\blank,x)$ are equivalences for each $x:X$.
\end{proof}

\begin{rmk}
  The Hopf map $h$ constructed in \cref{thm:hopf-construction} is the unique map $\join{A}{A}\to\susp A$ equipped with identifications
  \begin{align*}
    p & : \north = h(\inl(x)) \\
    p' & : \south = h(\inr(x'))
  \end{align*}
  and an identification $q$ witnessing that the square
  \begin{equation*}
    \begin{tikzcd}
      \north \arrow[d,equals,swap,"{\merid(\mu(x,x'))}"] \arrow[r,equals,"p"] & h(\inl(x)) \arrow[d,equals,"\ap{h}{\glue(x,x')}"] \\
      \south \arrow[r,equals,swap,"{p'}"] & h(\inr(x'))
    \end{tikzcd}
  \end{equation*}
  commutes.
\end{rmk}

\begin{cor}
There is a fiber sequence
\begin{equation*}
\sphere{1}\hookrightarrow \join{\sphere{1}}{\sphere{1}} \twoheadrightarrow \sphere{2}.
\end{equation*}
\end{cor}

\begin{proof}
  By \cref{thm:hopf-construction} it suffices to construct an H-space structure on $\sphere{1}$. This H-space structure $\sphere{1}\times\sphere{1}\to\sphere{1}$ is determined by the complex multiplication operation constructed in \cref{defn:mul-circle}.
\end{proof}

\begin{lem}
The join operation is associative
\end{lem}

\begin{proof}
\begin{equation*}
\begin{tikzcd}
A & A\times C \arrow[l] \arrow[r] & A\times C \\
A\times B \arrow[u] \arrow[d] & A\times B \times C \arrow[r] \arrow[d] \arrow[l] \arrow[u] & A\times C \arrow[u] \arrow[d] \\
B & B\times C \arrow[l] \arrow[r] & C
\end{tikzcd}
\end{equation*}
\end{proof}

\begin{cor}
There is an equivalence $\eqv{\join{\sphere{1}}{\sphere{1}}}{\sphere{3}}$.
\end{cor}

\begin{thm}
There is a fiber sequence $\sphere{1}\hookrightarrow\sphere{3}\twoheadrightarrow\sphere{2}$.
\end{thm}

\begin{lem}
Suppose $f:G\to H$ is a group homomorphism, such that the sequence
\begin{equation*}
\begin{tikzcd}
0 \arrow[r] & G \arrow[r,"f"] & H \arrow[r] & 0
\end{tikzcd}
\end{equation*}
is exact at $G$ and $H$, where we write $0$ for the trivial group consisting of just the unit element. Then $f$ is a group isomorphism.
\end{lem}

\begin{cor}
We have $\pi_2(\sphere{2})=\Z$, and for $k>2$ we have $\pi_k(\sphere{2})=\pi_k(\sphere{3})$.
\end{cor}

\subsection{The long exact sequence}

\begin{defn}
A fiber sequence $F\hookrightarrow E \twoheadrightarrow B$ consists of:
\begin{enumerate}
\item Pointed types $F$, $E$, and $B$, with base points $x_0$, $y_0$, and $b_0$ respectively,
\item Base point preserving maps $i:F\to_\ast E$ and $p:E\to_\ast B$, with $\alpha:i(x_0)=y_0$ and $\beta:p(y_0)=b_0$,
\item A pointed homotopy $H:\mathsf{const}_{b_0}\htpy_\ast p\circ_\ast i$ witnessing that the square
\begin{equation*}
\begin{tikzcd}
F \arrow[r,"i"] \arrow[d] & E \arrow[d,"p"] \\
\unit \arrow[r,swap,"\mathsf{const}_{b_0}"] & B,
\end{tikzcd}
\end{equation*}
commutes and is a pullback square.
\end{enumerate}
\end{defn}

\begin{lem}
Any fiber sequence $F\hookrightarrow E\twoheadrightarrow B$ induces a sequence of pointed maps
\begin{equation*}
\begin{tikzcd}
\loopspace{F} \arrow[r,"\loopspace{i}"] & \loopspace{E} \arrow[r,"\loopspace{p}"] & \loopspace{B} \arrow[r,"\partial"] & F \arrow[r,"i"] & E \arrow[r,"p"] & B,
\end{tikzcd}
\end{equation*}
in which every two consecutive maps form a fiber sequence.
\end{lem}

\begin{proof}
By taking pullback squares repeatedly, we obtain the diagram
\begin{equation*}
\begin{gathered}[b]
\begin{tikzcd}[column sep=large]
\loopspace{F} \arrow[d,swap,"\loopspace{i}"] \arrow[r] & \unit \arrow[d,"\mathsf{const}_{\refl{b_0}}"] \\
\loopspace{E} \arrow[r,"\loopspace{p}"] \arrow[d] & \loopspace{B} \arrow[r] \arrow[d,swap,"\partial"] & \unit \arrow[d,"\mathsf{const}_{y_0}"] \\
\unit \arrow[r,swap,"\mathsf{const}_{x_0}"] & F \arrow[r,"i"] \arrow[d] & E \arrow[d,"p"] \\
& \unit \arrow[r,swap,"\mathsf{const}_{b_0}"] & B.
\end{tikzcd}\\[-\dp\strutbox]
\end{gathered}\qedhere
\end{equation*}
\end{proof}

\begin{defn}
We say that a consecutive pair of pointed maps between pointed sets
\begin{equation*}
\begin{tikzcd}
A \arrow[r,"f"] & B \arrow[r,"g"] & C
\end{tikzcd}
\end{equation*}
is \define{exact} at $B$ if we have
\begin{equation*}
\Big(\exis{a:A}f(a)=b\Big)\leftrightarrow (g(b)=c)
\end{equation*}
for any $b:B$.
\end{defn}

\begin{rmk}
If a pair of consecutive pointed maps between pointed sets
\begin{equation*}
\begin{tikzcd}
A \arrow[r,"f"] & B \arrow[r,"g"] & C
\end{tikzcd}
\end{equation*}
is exact at $B$, it directly that $\im(f)=\fib{g}{c}$. Indeed, such a pair of pointed maps is exact at $B$ if and only if there is an equivalence $e:\im(f)\eqvsym \fib{g}{c}$ such that the triangle
\begin{equation*}
\begin{tikzcd}[column sep=tiny]
\im(f) \arrow[dr] \arrow[rr,"e"] & & \fib{g}{c} \arrow[dl] \\
& B
\end{tikzcd}
\end{equation*}
commutes. In other words, $\im(f)$ and $\fib{g}{c}$ are equal \emph{as subsets of $B$}.
\end{rmk}

\begin{lem}
Suppose $F\hookrightarrow E \twoheadrightarrow B$ is a fiber sequence. Then the sequence
\begin{equation*}
\begin{tikzcd}
\trunc{0}{F} \arrow[r,"\trunc{0}{i}"] & \trunc{0}{E} \arrow[r,"\trunc{0}{p}"] & \trunc{0}{B}
\end{tikzcd}
\end{equation*}
is exact at $\trunc{0}{E}$.
\end{lem}

\begin{proof}
To show that the image $\im\trunc{0}{i}$ is the fiber $\fib{\trunc{0}{p}}{\tproj{0}{b_0}}$, it suffices to construct a fiberwise equivalence
\begin{equation*}
\prd{x:\trunc{0}{E}} \trunc{-1}{\fib{\trunc{0}{i}}{x}} \eqvsym \trunc{0}{p}(x)=\tproj{0}{b_0}.
\end{equation*}
By the universal property of $0$-truncation it suffices to show that
\begin{equation*}
\prd{x:E} \trunc{-1}{\fib{\trunc{0}{i}}{\tproj{0}{x}}} \eqvsym \trunc{0}{p}(\tproj{0}{x})=\tproj{0}{b_0}.
\end{equation*}
First we note that
\begin{align*}
\trunc{0}{p}(\tproj{0}{x})=\tproj{0}{b_0} & \eqvsym \tproj{0}{p(x)} = \tproj{0}{b_0} \\
& \eqvsym \trunc{-1}{p(x)=b_0}.
\end{align*}
Next, we note that
\begin{align*}
\fib{\trunc{0}{i}}{\tproj{0}{x}} & \eqvsym \sm{y:\trunc{0}{F}}\trunc{0}{i}(y)=\tproj{0}{x} \\
& \eqvsym \trunc{0}{\sm{y:F}\trunc{0}{i}(\tproj{0}{y})=\tproj{0}{x}} \\
& \eqvsym \trunc{0}{\sm{y:F}\tproj{0}{i(y)}=\tproj{0}{x}} \\
& \eqvsym \trunc{0}{\sm{y:F}\trunc{-1}{i(y)=x}}.
\end{align*}
Therefore it follows that
\begin{align*}
\trunc{-1}{\fib{\trunc{0}{i}}{\tproj{0}{x}}} & \eqvsym \trunc{-1}{\sm{y:F}\trunc{-1}{i(y)=x}} \\
& \eqvsym \trunc{-1}{\sm{y:F}i(y)=x} \\
\end{align*}
Now it suffices to show that $\eqv{\big(\sm{y:F}i(y)=x\big)}{p(x)=b_0}$. This follows by the pasting lemma of pullbacks
\begin{equation*}
\begin{tikzcd}
(p(x)=b_0) \arrow[r] \arrow[d] & \unit \arrow[d] \\
F \arrow[r] \arrow[d] & E \arrow[d] \\
\unit \arrow[r] & B
\end{tikzcd}
\end{equation*}
\end{proof}

\begin{thm}
Any fiber sequence $F\hookrightarrow E\twoheadrightarrow B$ induces a long exact sequence on homotopy groups
\begin{equation*}
\begin{tikzcd}
& & \cdots \arrow[out=355,in=175,dll] \\
\pi_n(F) \arrow[r,"\pi_n(i)"] & \pi_n(E) \arrow[r,"\pi_n(p)"] & \pi_n(B) \arrow[out=355,in=175,dll,densely dotted] \\
\pi_1(F) \arrow[r,"\pi_1(i)"] & \pi_1(E) \arrow[r,"\pi_1(p)"] & \pi_1(B) \arrow[out=355,in=175,dll] \\
\pi_0(F) \arrow[r,"\pi_0(i)"] & \pi_0(E) \arrow[r,"\pi_0(p)"] & \pi_0(B)
\end{tikzcd}
\end{equation*}
\end{thm}

\subsection{The universal complex line bundle}

\begin{defn}
  A \define{coherently associative unpointed H-space} structure on a type $X$ consists of
\end{defn}

\subsection{The finite dimensional complex projective spaces}

\begin{rmk}
  The universe of types that are merely equal to the circle does not classify complex line bundles.
\end{rmk}

\begin{exercises}
  \exercise Consider an unpointed H-space $X$ of which the multiplication is associative, and consider $x:X$. Construct a unit for the multiplication, and show that it satisfies the coherent unit laws.
  \exercise
  \begin{subexenum}
  \item Show that the type of associative unpointed H-space structures on $\bool$ is equivalent to $\bool$.
  \item Show that the type of associative (pointed) H-space structures on $(\bool,\btrue)$ is contractible.
  \end{subexenum}
  \exercise Show that any fiber sequence
\begin{equation*}
  F \hookrightarrow E \twoheadrightarrow B
\end{equation*}
where the base points are $x_0:B$, $y_0:F$, and $z_0:E$ induces a fiber sequence of connected components
\begin{equation*}
  \mathsf{BAut}(y_0) \hookrightarrow \mathsf{BAut}(z_0) \twoheadrightarrow \mathsf{BAut}(x_0).
\end{equation*}
  \exercise Show that there is a fiber sequence
\begin{equation*}
  \sphere{3}\hookrightarrow\sphere{2}\twoheadrightarrow\trunc{2}{\sphere{2}},
\end{equation*}
where the map $\sphere{2}\to\trunc{2}{\sphere{2}}$ is the unit of the $2$-truncation.
  \exercise Show that $\cprojective{\infty}$ is a coherent H-space. Note: the $2$-sphere is not an H-space, and yet its $2$-truncation is!
  \exercise Construct for every group $G$ of order $n+1$ a fiber sequence
  \begin{equation*}
    \begin{tikzcd}
      G \arrow[r,hook] & \bigvee_{(i:\Fin(n^2))}\sphere{1}\arrow[r,->>] & \bigvee_{(i:\Fin(n))}\sphere{1}
    \end{tikzcd}
  \end{equation*}
  \exercise Show that there is a fiber sequence
\begin{equation*}
  \rprojective{\infty}\hookrightarrow\cprojective{\infty}\twoheadrightarrow\cprojective{\infty}.
\end{equation*}
\exercise Show that the type of (small) fiber sequences is equivalent to the type of quadruples $(B,P,b_0,x_0)$, consisting of
\begin{align*}
B & : \UU \\
P & : B \to \UU \\
b_0 & : B \\
x_0 & : P(b_0).
\end{align*}
\end{exercises}


% !TEX root = /Users/hzd88688126com/Desktop/USFD_Academic-_Report_LaTeX-Template/main.tex
\allowdisplaybreaks[4]
\section{The Least Mean Square (LMS) Algorithm}
\subsection{Correlation matrix}
The autocorrelation matrix is defined \cite{Mandic2019} as $\mathbf{R}_{xx}=\mathbb{E}\{\mathbf{x}(n)\mathbf{x}^T(n)\}$ and $\mathbf{x}(n)$ is $[ x(n-1), x(n-2)]^T$, thus $\mathbf{R}_{xx}$ is
\begin{equation}
\mathbf{R}_{xx}=\mathbb{E}\left \{ \left[
	\begin{matrix}
	x(n-1)x(n-1) & x(n-1)x(n-2)\\
	x(n-2)x(n-1) & x(n-2)x(n-2)
	\end{matrix}
		\right]
\right \}=
\left[
	\begin{matrix}
	\mathbf{r}_{xx}(0) & \mathbf{r}_{xx}(1)\\
	\mathbf{r}_{xx}(1) & \mathbf{r}_{xx}(0)
	\end{matrix}
		\right]
\end{equation}
Given by the $x(n)$ is a second-order auto-regressive process as
\begin{equation}
x(n)=a_1x(n-1)+a_2x(n-2)+\eta(n) \quad where \quad \eta(n)\sim \mathcal{N}(0,\sigma_{\eta}^2)
\end{equation}
the $\mathbf{r}_{xx}(0)$ and $\mathbf{r}_{xx}(1)$ can be calculated by following steps due to the correlation matrix $\mathbf{r}_{xx}(k)=\mathbb{E}\{x(n)x(n-k)\}$
\begin{align}
\mathbf{r}_{xx}(0)
	&=\mathbb{E}\{x(n)x(n)\} \notag \\
	&=\mathbb{E}\{ a_1^2x^2(n-1)+ a_2^2x^2(n-2)+2 a_1a_2 x(n-1) x(n-2)\notag\\
	&\quad+\eta(n)[a_1x(n-1)+a_2x(n-2)]\}+\sigma^2_\eta \notag\\
	&=a_1^2\mathbf{r}_{xx}(0)+a_2^2\mathbf{r}_{xx}(0)+2a_1a_2\mathbf{r}_{xx}(1)+\sigma^2_\eta	\label{equation:r0}
\end{align}
Cause the noise subspace is orthogonal with signal subspace, the product term $\mathbb{E}\{\eta(n)[a_1x(n-1)+a_2x(n-2)]\}$ is zero. In the same way, the $\mathbf{r}_{xx}(1)$ can be calculated as well as shown in Eq.\ref{equation:r1}.
\begin{align}
\mathbf{r}_{xx}(1)
	&=\mathbb{E}\{x(n)x(n-1)\} \notag \\
	&=\mathbb{E}\{ a_1x^2(n-1)+ a_2x(n-1)x(n-2)+\eta(n)x(n-1)\}\notag\\
	&=a_1\mathbf{r}_{xx}(0)+a_2\mathbf{r}_{xx}(1)	\label{equation:r1}
\end{align}
Therefore, substituting Eq.\ref{equation:r0} to Eq.\ref{equation:r1}, the unique solutions of $\mathbf{r}_{xx}(0)$ and $\mathbf{r}_{xx}(1)$ are $\frac{25}{27}$ and $\frac{25}{54}$ respectively. The autocorrelation matrix $\mathbf{R}_{xx}$ is
\begin{equation}
\mathbf{R}_{xx}=\left[
\begin{matrix}
\frac{25}{27}&\frac{25}{54}\\
\frac{25}{54}&\frac{25}{27}
\end{matrix}\right]
\end{equation}
As to LMS algorithm, the convergence step $\mu$ is defined in the range of $0<\mu<\frac{2}{\lambda_{max}}$., where $\lambda_{max}$ is the maximum eigenvalue of $\mathbf{R}_{xx}$. Applying eigendecomposition to autocorrelation matrix, the eigenvalues are approximate 0.463 and 1.3889. Taking the large one into account, the range of step $\mu$ is
\begin{equation}
0<\mu<\frac{2}{1.3889}=1.44
\label{eq:range}
\end{equation}
\subsection{LMS filter}
The LMS adaptive filter is implemented to estimate AR model's weights using $N=1000$ samples with 100 realizations. As a comparison of convergence, tow different steps $\mu=0.01,\text{ }0.05$ are considered within the limited range in Eq.\ref{eq:range}. Fig.\ref{fig:2_1_b} shows the estimated errors of single realization and mean error with two steps.\\
As to a single realization, the impact of descent step is not significant, since the white noise signal is randomly. However, taking account into mean error of 100 realizations, the convergence speed of large steps is faster than the smaller one. Specifically, the learning curve converges approximately after 100 samples at $\mu=0.05$, while it is 250 samples at $\mu=0.01$. Nevertheless, large step size causes significant fluctuation as well which should be trade-off.
\begin{figure}[htp]
     \centering
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/21/21b1.eps}
     \end{subfigure}
     ~
     \begin{subfigure}[b]{0.4\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/21/21b2.eps}
     \end{subfigure}
        \caption{LMS estimated error with different realizations and steps}
        \label{fig:2_1_b}
\end{figure}
\subsection{Misadjustment of LMS}
The theoretical misadjustent of the LMS is approximated as $\mathcal{M}_{LMS} \approx \frac{\mu}{2}\mathrm{Tr}\{\mathbf{R}_{xx}\}$, where the autocorrelation matrix $\mathbf{R}_{xx}$ is calculated. However, the estimated misadjustment is the ratio of excess MSE and minimum MSE, which is $\mathcal{M}=\frac{MSE-\sigma^2}{\sigma^2}$. In addition, in order to guarantee the error calculated during steady state, the samples are selected after 400. Table.\ref{tab:M} shows the values of misadjustment. As shown in table, the measured MSE and misadjustment is slightly larger than theoretical values. However, small step size introduces a small misadjustment, whereas the convergence speed is slow..
\begin{table}[htp]
\centering
\caption{Theoretical and Actual Misadjustment values for step sizes}
\begin{tabular}{ |c|c|c| }
 \hline
 $\mu$ & $\mathcal{M}_{LMS}$ & $\mathcal{M}$ \\
 \hline
 0.01 & 0.0093 & 0.0134
 \\
 \hline
 0.05 & 0.0463 & 0.0534
 \\
 \hline
\end{tabular}
\label{tab:M}
\end{table}
\subsection{LMS: estimated weights}
Fig.\ref{fig:2_1_d} depicts the estimated weights $a_1=0.1$ and $a_2=0.8$ with steps $\mu=0.01$ and $\mu=0.05$. Compared these estimated weights with true values, small step size provides an acceptable steady state error which is closer to actual weights. However, the convergence speed is relatively slow compared with large step size. On the contrast, when $\mu=0.05$, the convergence speed is significantly improved, resulting in a large difference with actual values. Thus, there is a trade-off between steady state error and convergence speed.
\begin{figure}[htp]
     \centering
     \includegraphics[width=0.4\textwidth]{fig/21/21d.eps}
     \caption{LMS estimated weights with different steps}
     \label{fig:2_1_d}
\end{figure}
\subsection{Leaky LMS}
For the leaky LMS, the cost function is
\begin{align}
\mathcal J_2(n)&=\frac{1}{2}(e^2(n)+\gamma ||\mathbf {w}(n)||_2^2)\label{eq:cost2}\\
where \quad e(n)&=d(n)-\mathbf{w}_T(n)\mathbf {x}(n)\notag
\end{align}
To get the minimum squared error, the gradient with weight $\mathbf{w}$ of Eq.\ref{eq:cost2} is taken,
\begin{align}
\nabla \mathcal J_2(n)&=\frac{1}{2}\partial (e^2(n)+\gamma ||\mathbf {w}(n)||_2^2)/\partial \mathbf {w}\notag\\
&=\frac{1}{2}(\frac{\partial(e^2(n))}{e(n)}\frac{\partial(e(n))}{\mathbf {w}}+2\gamma ||\mathbf {w}(n)||_2)\notag\\
&=-e(n)\mathbf{x}(n)+\gamma \mathbf {w}(n)
\label{eq:J2}
\end{align}
Therefore, the updated weight is
\begin{align}
\mathbf {w}(n+1)&=\mathbf {w}(n)+\mu (-\nabla \mathcal J_2(n))\notag\\
				&=\mathbf {w}(n)+\mu (e(n)\mathbf{x}(n)-\gamma \mathbf {w}(n))\notag\\
				&=(1-\mu \gamma)\mathbf {w}(n)+\mu e(n)\mathbf{x}(n)
\label{eq:llms}
\end{align}
\subsection{Leaky LMS: estimated weights}
Fig.\ref{fig:2_1_f} illustrated the estimated weights using leaky LMS filter with $\gamma=$ 0.2, 0.4, 0.6 and $\mu=$ 0.01, 0.05. However, the estimated values converges to a certain value with a large difference with actual coefficient. With the incremental of $\gamma$, the steady state error significantly rises up. Observing weights curves, increasing $\gamma$ introduces a larger bias for lager weight than small weight. Thus, Leaky LMS causes large penalty for the large weight estimation.
\begin{figure}[htb]
     \centering
      \hspace{-0.4cm}
     \begin{subfigure}[b]{0.33\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/21/21f1.eps}
     \end{subfigure}
     \hspace{-0.4cm}
     \begin{subfigure}[b]{0.33\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/21/21f2.eps}
     \end{subfigure}
      \hspace{-0.4cm}
     \begin{subfigure}[b]{0.33\textwidth}
         \centering
         \includegraphics[width=\textwidth]{fig/21/21f3.eps}
     \end{subfigure}
        \caption{Leaky LMS estimated weights with different $\gamma$ and $\mu$}
        \label{fig:2_1_f}
\end{figure}\\
As to the general LMS algorithm, the optimal weight is
\begin{align}
 \mathbf {w}_{opt}=\mathbf{R}_{xx}^{-1} \mathbf{p}\quad \text{where} \quad \mathbf{p}= \mathbb{E}\{\mathbf{x}(n)d(n)\}
\end{align}
The autocorrelation matrix is semi-positive definite and can be invertible. However, the gradient vanishing problem is suffered when the eigenvalues of $\mathbf{R}_{xx}$ are zeros, resulting in divergence to expected values. By adding a small number $\gamma$ of $\mathbf{R}_{xx}$, the weights can converge, which is the purpose of Leaky LMS. Thus, the optimal weight is
\begin{equation}
 \mathbf {w}_{opt}=(\mathbf{R}_{xx}+\gamma \mathbf{I})^{-1} \mathbf{p}
\end{equation}
Nevertheless, the eigenvalues of autocorrelation matrix are non-zeros for this experiment. Hence, adding large value of $\gamma$ leads to an obvious bias and the estimated weights converge to incorrect coefficients.