\documentclass[a4paper, 12pt]{article}

\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage[]{amsfonts}
\usepackage[]{graphicx}
\usepackage[]{amsthm}

\title{CS231A Course Notes 2: Single View Metrology}
\author{Kenji Hata and Silvio Savarese}
\date{}

\renewcommand\emph{\textbf}

\begin{document}

\maketitle

\section{Introduction}
In the previous lecture notes, we discussed how we can transform points from the real, 3D world into digital images using the extrinsic and intrinsic properties of cameras. We looked at how we can use known structure in a calibration rig and its corresponding image to deduce these camera properties. This time, we will look at a related problem: can we recover known structure of the 3D world if we have a single image and know the properties of the camera that took the image? We will then more generally discuss what information can be deduced from a single image.

\section{Transformations in 2D}
To better understand how we can learn from images, we should be able to first know about the various transformations in 2D space.

\emph{Isometric transformations} are transformations that preserve distances. In its most basic form, an isometry can be described as a rotation $R$  and translation $t$. Therefore, mathematically, they are defined as
\begin{equation*}
    \begin{bmatrix}x'\\y'\\1\end{bmatrix} = \begin{bmatrix}R & t\\ 0 & 1\end{bmatrix}\begin{bmatrix}x\\y\\1\end{bmatrix}
\end{equation*}
where $\begin{bmatrix}x'&y'&1\end{bmatrix}^T$ is the point achieved after the isometric transformation. A useful property of rotation matrices is that $RR^T = I$, which can be used to quickly check if a matrix is a rotation.

\emph{Similarity transformations} are transformations that preserve shape. Intuitively, they can do everything that isometric transformations can plus scaling. Mathematically, they are denoted as
\begin{equation*}
    \begin{bmatrix}x'\\y'\\1\end{bmatrix} = \begin{bmatrix}SR & t\\ 0 & 1\end{bmatrix}\begin{bmatrix}x\\y\\1\end{bmatrix},\ S = \begin{bmatrix}s & 0\\ 0 & s\end{bmatrix}
\end{equation*}
Since they preserve shapes, they also preserve ratio of lengths and angles. Note that every isometric transformation is a specific form of a similarity transformation when $s=1$. The converse does not hold true however.

\emph{Affine transformations} are transformations that preserve points, straight lines, and parallelism. For some vector $v$, an affine transformation $T$ is defined as
\begin{equation*}
    T(v) = Av + t
\end{equation*}
where $A$ is a linear transformation of $\mathbb{R}^n$. In homogeneous coordinates, affine transformations are often written as
\begin{equation*}
    \begin{bmatrix}x'\\y'\\1\end{bmatrix} = \begin{bmatrix}A & t\\ 0 & 1\end{bmatrix}\begin{bmatrix}x\\y\\1\end{bmatrix}
\end{equation*}
From the above equation, it is easy to see that all similarities (and thus isometries) are a specific case of affinities.

\emph{Projective transformations} or \emph{homographies} are any transformations that maps lines to lines, but does not necessarily preserve parallelism. In homogeneous coordinates, projective transformations are represented as
\begin{equation*}
    \begin{bmatrix}x'\\y'\\1\end{bmatrix} = \begin{bmatrix}A & t\\ v & b\end{bmatrix}\begin{bmatrix}x\\y\\1\end{bmatrix}
\end{equation*}
We see that the form is a further generalization of affine transformations, as extra degrees of freedom are added with the addition of $v$.

Despite not preserving parallelism, projective transformations does preserve collinearity of points, as it maps lines to lines. Furthermore, we prove that the cross ratio of four collinear points remains invariant under projective transformations. The cross ratio takes four points $P_1, P_2, P_3, P_4$ on a line and computes
\begin{equation}
    \mathrm{cross\ ratio} = \frac{\|P_3-P_1\|\|P_4-P_2\|}{\|P_3-P_2\|\|P_4-P_1\|}
\end{equation}
We leave proving the invariance of the cross ratio under projective transformation as a class exercise.

\section{Points and Lines at Infinity}
Lines are important for determining structure in images, so knowing their definitions in both 2D and 3D is essential. A line in 2D can be represented with the homogeneous vector $\ell = \begin{bmatrix}a & b & c \end{bmatrix}^T$. The ratio $-\frac{a}{b}$ captures the slope of the line and the ratio $-\frac{c}{b}$ captures the y-intercept. Formally, the relationship between 2D lines and points on them are defined by:
\begin{equation}
\forall p = \begin{bmatrix}x\\y\end{bmatrix} \in \ell,\ \  \begin{bmatrix}a & b & c\end{bmatrix}\begin{bmatrix}x\\y\\1\end{bmatrix} = 0
\end{equation}

In general, two lines $\ell$ and $\ell'$ will intersect at a point $x$. This point is defined as the cross product between $\ell$ and $\ell'$.

\begin{proof}
    Given two intersecting lines $\ell$ and $\ell'$, the intersection point $x$ should lie on both lines $\ell$ and $\ell'$. Therefore, $x^T \ell = 0$ and $x^T\ell' = 0$. If we set $x = \ell \times \ell'$, then by definition of cross product, the vector $x$ is orthogonal to both vectors $\ell$ and $\ell'$. By the definition of orthogonality, $x^T \ell = 0$ and $x^T\ell' = 0$. Thus, this definition of $x$ satisfies the constraints.
\end{proof}

What about the case of parallel lines? Everyday knowledge expects these lines to never intersect. However, this definition could be rewritten to say that these lines intersect at infinity. In homogeneous coordinates, a point at infinity is written as $\begin{bmatrix}x & y & 0\end{bmatrix}^T$. Recall the Euclidean coordinates are gathered by dividing all coordinates by the last coordinate. In this case, the coordinate is zero, achieving a point at infinity. Therefore, homogeneous coordinates give a good formulation of determining intersections, even in cases of parallel lines.

Now, let's consider two parallel lines $\ell$ and $\ell'$. When two lines are parallel, their slope is equal and thus $\frac{a}{b} = \frac{a'}{b'}$. If we compute the point of intersection using homogenoous coordinates, then we verify that
\begin{equation}
\ell \times \ell' \propto \begin{bmatrix}b \\ -a \\ 0  \end{bmatrix}= x_\infty
\end{equation}

Thus, we confirmed our intuition that two parallel lines intersect at infinity. The point of intersection at infinity of two parallel lines is also called an \emph{ideal point}. One interesting property of a point at infinity is that all parallel lines with the same slope $-\frac{a}{b}$ pass through the ideal point as shown below:
\begin{equation}
\ell ^T x_\infty = \begin{bmatrix}a & b & c\end{bmatrix} \begin{bmatrix}b \\ -a \\ 0\end{bmatrix} = 0
\end{equation}

\begin{figure}[h!]
\centering
\includegraphics[width=0.5\textwidth]{figures/line_infinity.png}
\caption{Points at infinity form lines at infinity.}
\label{fig:line_infinity}
\end{figure}

The concept of points of infinity can be extended to define \emph{lines at infinity}. Consider two or more pairs of parallel lines. Each pair of parallel lines intersects at a point at infinity $\{x_{\infty,1},...,x_{\infty,n}\}$. The line $\ell_\infty$ that passes through all these points at infinity must satisfy $\forall i, \ell_\infty^T x_{\infty,i}= 0$. This means that $\ell_\infty = \begin{bmatrix}0 & 0 & c\end{bmatrix}^T$. Since $c$ is an arbitrary value, we can simple define $\ell_\infty = \begin{bmatrix}0 & 0 & 1\end{bmatrix}^T$.

If we apply a generic projective transformation $H$ to a point at infinity $p_\infty$, what will happen?
\begin{equation}
p' =Hp_\infty = \begin{bmatrix} A &t \\ v&b \end{bmatrix}\begin{bmatrix}1\\1\\0\end{bmatrix} = \begin{bmatrix}p'_x\\ p'_y\\p'_z\end{bmatrix}
\end{equation}

Notice that the last element of $p'$ may become non-zero, which suggests that a projective transformation generally maps points at infinity to points that are no longer at infinity. However, this is not the case for affine transformations, which map points at infinity to points at infinity:
\begin{equation}
p' =Hp_\infty = \begin{bmatrix} A &t \\ 0&1 \end{bmatrix}\begin{bmatrix}1\\1\\0\end{bmatrix} = \begin{bmatrix}p'_x\\ p'_y\\0\end{bmatrix}
\end{equation}

Now let's apply a projective transformation $H$ to a line $\ell$ to get a new line $\ell'$. All points $x$ that pass through a line must satisfy the property $x^T\ell = 0$. In the transformed space, we know that lines still map to lines, which means that $x'\ell' = 0$. We can use the identity property to get \[x^TI\ell = x^TH^T H^{-T} \ell = 0\]If we apply a projective transformation to the line, then all the points become transformed as well, giving $x' = Hx$. Thus we get $x^TH^T H^{-T} \ell = x'^T\ell'$, and we find the projective transformation of a line is $\ell' = H^{-T}\ell$. Similar to our observations with points at infinity, we find that the projective transformation of a line at infinity does not necessarily map to another line at infinity. Additionally, affine transformations still map lines at infinity to lines at infinity.

\section{Vanishing Points and Lines}
So far, we have introduced the concepts of lines and points at infinity in 2D. Let us now introduce the equivalent concepts for 3D in its corresponding homogeneous coordinates.

In the 3D world, we are now introduced to the concepts of planes. We can represent a plane as a vector $\begin{bmatrix}a&b&c&d\end{bmatrix} ^T$, where $(a,b,c)$ form a normal vector and $d$ is the distance from the origin to the plane in that normal vector's direction. Formally, a plane is defined as all the points $x$ which satisfy
\begin{equation}
x^T\begin{bmatrix}a\\b\\c\\d\end{bmatrix} = ax_1 + bx_2 + cx_3 + d = 0
\end{equation}
Lines in 3D are defined as the intersection of two planes. Since they have four degrees of freedom (a defined intercept location and slopes in each of the three dimensions), they are difficult to represent nicely in 3D space. Please see Section 3.2.2 of the Hartley \& Zisserman textbook for more details.

Points, however, are defined similarly in 3D as they are in 2D. Points at infinity in 3D are again defined as the intersection point of parallel lines in 3D. Furthermore, if we apply a projective transformation to one of these points at infinity $x_\infty$, then we obtain a point $p_\infty$ in the image plane, which is no longer at infinity in homogeneous coordinates. This point $p_\infty$ is known as a \emph{vanishing point}. But, what can we do with vanishing points?

We can derive a useful relationship between parallel lines in 3D, their corresponding vanishing point in the image, and the camera parameters $K,R,T$. Let us define $d = (a,b,c)$ as the direction of a set of 3D parallel lines in the camera reference system. These lines intersect to a point at infinity and the projection of such a point in the image returns the vanishing point $v$, which is defined by
\begin{equation}
v = Kd
\end{equation}
We leave the derivation of the above equation as an exercise. This equation can be rewritten to extract the direction $d$:
\begin{equation}
d = \frac{K^{-1}v}{\|K^{-1}v\|}
\end{equation}

If we consider a plane $\Pi$ as a superset of parallel lines, each set of parallel lines intersects at a point at infinity. The line that passes through such set of points at infinity is the line at infinity $\ell_\infty$ associated to $\Pi$. A line at infinity is also defined as the line where two parallel planes intersect. The projective transformation of $\ell_\infty$ to the image plane is no longer a line at infinity and is called the vanishing line or the \emph{horizon line} $\ell_{\mathrm{horiz}}$. The horizon line is a line that passes through the corresponding vanishing points in the image. The horizon line can be computed as
\begin{equation}
\ell_{\mathrm{horiz}} = H_P^{-T} \ell_\infty
\end{equation}

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/horizon.png}
\caption{The computed horizon line from a set of vanishing points.}
\label{fig:horizon}
\end{figure}

The concept of a horizon line allows us to answer as humans to intuitively deduce properties about the image that may not be easily apparent mathematically. For example, in Figure~\ref{fig:horizon}, although the lines on the ground are not parallel in image coordinates, we have a natural understanding that they are parallel in the 3D world.

Furthermore, the horizon line allows us to compute useful properties about the world. For example, we can derive an interesting relationship between the the normal $n$ of a plane in 3D with the corresponding horizon line $\ell_{\mathrm{horiz}}$ in an image:
\begin{equation}
n = K^T \ell_{\mathrm{horiz}}
\end{equation}
This means that if we can recognize the horizon line associated with a plane, and if our camera is calibrated, then we can estimate the orientation of that plane.
\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/plane_infinity.png}
\caption{A set of two or more vanishing lines (the blue lines) defines the plane at infinity $\Pi_\infty$ (the yellow plane).}
\label{fig:plane_infinity}
\end{figure}
Before introducing the last property that relates vanishing points and lines, we first need to define the plane at infinity $\Pi_\infty$. This plane is defined by a set of 2 or more vanishing lines and is described by the vector $\begin{bmatrix} 0 & 0 & 0 &1\end{bmatrix}^T$ in homogeneous coordinates.

\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/angles.png}
\caption{Deriving the angle between two lines.}
\label{fig:angles}
\end{figure}

The last property we introduce relates lines and planes in 3D with the corresponding vanishing points and lines in the image plane. Suppose that two pairs of parallel lines in 3D have directions $d_1$ and $d_2$, and are associated with the points at infinity $x_{1,\infty}$ and $x_{2,\infty}$. Let $v_1$ and $v_2$ be the corresponding vanishing points. Then, we find that the angle $\theta$ between $d_1$ and $d_2$ is given by using the cosine rule:
\begin{equation}
    \begin{split}
        \cos\theta &= \frac{d_1 \cdot d_2}{\|d_1\| \|d_2\|}\\
        &= \frac{v_1^T\omega v_2}{\sqrt{v_1^T \omega v_1}\sqrt{v_2^T \omega v_2}}
    \end{split}
    \label{eq:angles}
\end{equation}
where $\omega = (KK^T)^{-1}$.

We can extend this idea further to the 3D planar case, in which we want to relate different planes in 3D. Recall that for any plane, we can compute its associated vanishing line $\ell_\mathrm{{horiz}}$ and its normal $K^T \ell_{\mathrm{horiz}}$. Therefore, we can determine the angle $\theta$ between two planes by computing the angle between each of the planes' normal vectors $n_1$ and $n_2$. We derive the angle $\theta$ between two planes with a vanishing lines $\ell_1$ and $\ell_2$ respectively:
\begin{equation}
    \begin{split}
        \cos\theta &= \frac{n_1 \cdot n_2}{\|n_1\| \|n_2\|}\\
        &= \frac{\ell_1^T\omega^{-1} \ell_2}{\sqrt{\ell_1^T \omega^{-1} \ell_1}\sqrt{\ell_2^T \omega^{-1} \ell_2}}
    \end{split}
\end{equation}

\section{A Single View Metrology Example}
\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/example1.png}
\caption{The example setup with two vanishing points for a pair of perpendicular planes.}
\label{fig:example1}
\end{figure}
Suppose that we can identify two planes in an image of the 3D world. Additionally, let's suppose that we can identify a pair of parallel lines on each of these planes. This allows us to estimate two vanishing points $v_1$ and $v_2$ in the image. Finally, let's suppose that we know that these planes are perpendicular in 3D. In this case, we know that from Equation~\ref{eq:angles}, that $v_1\omega v_2 = 0$.

But recall that $\omega$ depends on the camera matrix $K$, which is potentially unknown at this time. Therefore, is knowing these two vanishing points sufficient for accurately estimating the camera parameters? Considering that $K$ has 5 degrees of freedom and that $v_1 \omega v_2 = 0$ provides only one constraint, we do not have enough information to calculate $K$.
\begin{figure}[h!]
\centering
\includegraphics[width=0.8\textwidth]{figures/example2.png}
\caption{The example setup with three vanishing points for a set of mutually perpendicular planes.}
\label{fig:example2}
\end{figure}
What if we are able to find another vanishing $v_3$ for another mutually orthogonal plane? Then we know that $v_1\omega v_2 = v_1\omega v_3 = v_2\omega v_3 = 0$. Since each pair gives a constraint, we only end up with 3 out of the 5 constraints needed to compute $K$. However, if we make the assumption that the camera has zero-skew and square pixels, then we can add the additional two constraints needed. By these assumptions, then we know that $\omega$ takes on the form
\begin{equation}
    \omega = \begin{bmatrix}\omega_1 & 0 & \omega_4 \\ 0 & \omega_1 & \omega_5 \\ \omega_4 & \omega_5 &\omega_6 \end{bmatrix}
\end{equation}

If you noticed carefully, there are four variables in the definition of $\omega$. However, we can only know $\omega$ up to scale, which reduces the amount of actual variables to three, allowing it to be solved. Once we have $\omega$, then we can use Cholesky decomposition to compute $K$. Thus, we have managed to calibrate the camera using a single image!

Once $K$ is known, then we can reconstruct the 3D geometry of the scene. For example, we can compute the orientation of all the planes identified above. Therefore, a single image can be readily used to uncover a wealth of information about the scene it captures.

\end{document}


\section*{VALIDATION}\label{validation}
The accuracy of the fitted models have been estimated using two test
data sets, containing 20\% and 30\% of the data. The results from this
evaluation in seen in Tab.\ref{tab:test_validataion} and
Fig.\ref{fig:test_evaluation}. The actual predictions on the
test dataset is also plotted in Fig.\ref{fig:predictions}. The
Support Vector Regressor seems to be the model that performs the best.
However, none of the models seem to perform very well on this dataset
and the accuracy changes very much with the test size, which means that
the ranking between the models is very unreliable. So one might as well
use the simple polynomial regression. The explicit formula from this
regression is shown in Eq.\ref{eq:model_polynomial}. It can be
seen from this expression that the $Power$ depends on the ship draught
$T$ and one of the wind components $U_{wind}$. There is however also
a very high interception term, which means that most of the $Power$ is
not explained by this model, if we assume that $Power$ is zero when
the ship is at rest in the harbour.
\begin{table}[H]
\scriptsize
\center
\caption{Models evaluated with the test sets}
\label{tab:test_validataion}
\begin{tabular}{|l|l|l|l|}
\hline\addlinespace
model name & test size & r2 score & mean absolute error\\
&  &  & \\
\hlineSVR & 0.2 & 0.64 & 687.07\\
XGBoost & 0.2 & 0.27 & 976.96\\
lasso & 0.2 & 0.29 & 999.13\\
polynomial & 0.2 & 0.3 & 997.36\\
ridge & 0.2 & 0.3 & 997.54\\
SVR & 0.3 & 0.83 & 510.07\\
XGBoost & 0.3 & 0.55 & 856.03\\
lasso & 0.3 & 0.45 & 1063.0\\
polynomial & 0.3 & 0.45 & 1060.51\\
ridge & 0.3 & 0.45 & 1061.05\\
\hline
\end{tabular}
\end{table}
\begin{figure}[H]
\begin{center}\includegraphics[width = 0.95\textwidth]{figures/test_evaluation.pdf}\end{center}
\vspace{-0.7cm}
\caption{Evaluation of the fitted models accuracy}
\label{fig:test_evaluation}
\end{figure}
\begin{figure}[H]
\begin{center}\includegraphics[width = 0.95\textwidth]{figures/predictions.pdf}\end{center}
\vspace{-0.7cm}
\caption{Predictions on the test data sets}
\label{fig:predictions}
\end{figure}
\begin{equation}
y = - 944.923049 T + 244.319815 U_{wind} + 2964.727832
\label{eq:model_polynomial}
\end{equation}


\section{Arithmetic task}

The aim of the ``Arithmetic task'' is to directly test arithmetic models ability to extrapolate beyond the training range. Additionally, our generalized version provides a high degree of flexibility in how the input is shaped, sampled, and the problem complexity.

Our ``arithmetic task'' is identical to the ``simple function task'' in the NALU paper \cite{trask-nalu}. However, as they do not describe their setup in details, we use the setup from \citet{maep-madsen-johansen-2019}, which provide Algorithm \ref{tab:simple-function-task-defaults}, an evaluation-criterion to if and when the model has converged, the sparsity error, as well as methods for computing confidence intervals for success-rate and the sparsity error.

\begin{figure}[h]
\centering
\includegraphics[scale=0.7]{graphics/function_task_static_problem.pdf}
\caption{Shows how the dataset is parameterized.}
\label{fig:simple-function-task-problem}
\end{figure}

\subsection{Dataset generation}
\label{sec:appendix:simple-function-task:data-generation}

The goal is to sum two random subsets of a vector $\mathbf{x}$ ($a$ and $b$), and perform an arithmetic operation on these ($a \circ b$).

\begin{equation}
    a = \sum_{i=s_{1,\mathrm{start}}}^{s_{1,\mathrm{end}}} x_i, \quad b = \sum_{i=s_{2,\mathrm{start}}}^{s_{2,\mathrm{end}}} x_i, \quad t = a \circ b
\end{equation}

Algorithm \ref{alg:simple-function-task-generator} defines the exact procedure to generate the data, where an interpolation range will be used for training and validation and an extrapolation range will be used for testing. Default values are defined in table \ref{tab:simple-function-task-defaults}.

\begin{table}[h]
\caption{Default dataset parameters for ``Arithmetic task''}
\label{tab:simple-function-task-defaults}
\centering
\begin{tabular}{cc}
\begin{minipage}{.4\linewidth}
\begin{tabular}{r l}
\toprule
 Parameter name & Default value \\
 \midrule
 Input size & 100 \\
 Subset ratio & 0.25 \\
 Overlap ratio & 0.5 \\
 \bottomrule
\end{tabular}
\end{minipage} &
\begin{minipage}{.4\linewidth}
\begin{tabular}{r l}
\toprule
 Parameter name & Default value \\
 \midrule
 Interpolation range & $U[1,2]$ \\
 Extrapolation range & $U[2,6]$ \\
 \\
 \bottomrule
\end{tabular}
\end{minipage}
\end{tabular}
\end{table}

\begin{algorithm}[h]
  \caption{Dataset generation algorithm for ``Arithmetic task''}
  \begin{algorithmic}[1]
    \Function{Dataset}{${\Call{Op}{\cdot, \cdot}: \mathrm{Operation}}$, ${i: \mathrm{Input Size}}$, ${s: \mathrm{Subset Ratio}}$, ${o: \mathrm{Overlap Ratio}}$, ${\hspace{3cm}R: \mathrm{Range}}$}
      \Let{$\mathbf{x}$}{\Call{Uniform}{$R_{lower}, R_{upper}, i$}} \Comment{Sample $i$ elements uniformly}
      \Let{$k$}{\Call{Uniform}{$0, 1 - 2s - o$}} \Comment{Sample offset}
      \Let{$a$}{\Call{Sum}{$\mathbf{x}[ik:i(k+s)]$}} \Comment{Create sum $a$ from subset}
      \Let{$b$}{\Call{Sum}{$\mathbf{x}[i(k+s-o):i (k+2s-0)]$}} \Comment{Create sum $b$ from subset}
      \Let{$t$}{\Call{Op}{$a, b$}} \Comment{Perform operation on $a$ and $b$}
      \State \Return{$x, t$}
    \EndFunction
  \end{algorithmic}
  \label{alg:simple-function-task-generator}
\end{algorithm}

\subsection{Model defintions and setup}

Models are defined in table \ref{tab:simple-function-task-model-defintions} and are all optimized with Adam optimization \cite{adam-optimization} using default parameters, and trained over $5 \cdot 10^6$ iterations. Training takes about 8 hours on a single CPU core(\text{8-Core Intel Xeon E5-2665 2.4GHz}). We run 19150 experiments on a HPC cluster.

The training dataset is continuously sampled from the interpolation range where a different seed is used for each experiment, all experiments use a mini-batch size of 128 observations, a fixed validation dataset with $1 \cdot 10^4$ observations sampled from the interpolation range, and a fixed test dataset with $1 \cdot 10^4$ observations sampled from the extrapolation range.

\begin{table}[h]
\caption{Model definitions}
\label{tab:simple-function-task-model-defintions}
\centering
\begin{tabular}{r l l l l l}
\toprule
 Model & Layer 1 & Layer 2 & $\hat{\lambda}_{\mathrm{sparse}}$ & $\lambda_{\mathrm{start}}$ & $\lambda_{\mathrm{end}}$ \\
 \midrule
 NMU & NAU & NMU & 10 & $10^6$ & $2 \cdot 10^6$ \\
 NAU & NAU & NAU & 0.01 & $5 \cdot 10^3$ & $5 \cdot 10^4$ \\
 $\mathrm{NAC}_{\bullet}$ & $\mathrm{NAC}_{+}$ & $\mathrm{NAC}_{\bullet}$ & -- & -- & -- \\
 $\mathrm{NAC}_{\bullet,\sigma}$ & $\mathrm{NAC}_{+}$ & $\mathrm{NAC}_{\bullet,\sigma}$ & -- & -- & -- \\
 $\mathrm{NAC}_{\bullet,\mathrm{NMU}}$ & $\mathrm{NAC}_{+}$ & $\mathrm{NAC}_{\bullet,\mathrm{NMU}}$ & 10 & $10^6$ & $2 \cdot 10^6$ \\
 $\mathrm{NAC}_{+}$ & $\mathrm{NAC}_{+}$ & $\mathrm{NAC}_{+}$ & -- & -- & -- \\
 NALU & NALU & NALU & -- & -- & -- \\
 Linear & Linear & Linear & -- & -- & -- \\
 ReLU & ReLU & ReLU & -- & -- & -- \\
 ReLU6 & ReLU6 & ReLU6 & -- & -- & -- \\
 \bottomrule
\end{tabular}
\end{table}

\subsection{Ablation study}
\label{sec:appendix:ablation-study}
To validate our model, we perform an ablation on the multiplication problem. Some noteworthy observations:

\begin{enumerate}
    \item None of the $W$ constraints, such as $\mathcal{R}_{sparse}$ and clamping W to be in $[0, 1]$, are necessary when the hidden size is just $2$.
    \item Removing the $\mathcal{R}_{sparse}$ causes the NMU to immediately fail for larger hidden sizes.
    \item Removing the clamping of W does not cause much difference. This is because $\mathcal{R}_{sparse}$ also constrains $W$ outside of $[0, 1]$. The regularizer used here is $\mathcal{R}_{sparse} = \min(|W|, |1 - W|)$, which is identical to the one used in other experiments in $[0, 1]$, but is also valid outside $[0, 1]$. Doing this gives only a slightly slower convergence. Although, this can not be guaranteed in general, as the regularizer is omitted during the initial optimization.
    \item Removing both constraints, gives a somewhat satisfying solution, but with a lower success-rate, slower convergence, and higher sparsity error.
\end{enumerate}

In conclusion both constraints are valuable, as they provide faster convergence and a sparser solution, but they are not critical to the success-rate of the NMU.

\begin{figure}[h]
\centering
\includegraphics[width=\linewidth]{results/simple_function_static_mul_hidden_size_ablation.pdf}
\caption{Ablation study where $\mathcal{R}_{sparse}$ is removed and the clamping of W is removed. There are 50 experiments with different seeds, for each configuration.}
\label{fig:simple-function-static-ablation}
\end{figure}

\subsection{Effect of dataset parameter}
\label{sec:appendix-simple-function-task:dataset-parameter-effect}

To stress test the models on the multiplication task, we vary the dataset parameters one at a time while keeping the others at their default value (default values in table \ref{tab:simple-function-task-defaults}). Each runs for 50 experiments with different seeds. The results, are visualized in figure \ref{fig:simple-function-static-dataset-parameters-boundary}.

In figure \ref{fig:simple-function-static-theoreical-claims-experiment}, the interpolation-range is changed, therefore the extrapolation-range needs to be changed such it doesn't overlap. For each interpolation-range the following extrapolation-range is used: ${\mathrm{U}[-2,-1] \text{ uses } \mathrm{U}[-6,-2]}$, ${\mathrm{U}[-2,2] \text{ uses } \mathrm{U}[-6,-2] \cup \mathrm{U}[2,6]}$, ${\mathrm{U}[0,1] \text{ uses } \mathrm{U}[1,5]}$, ${\mathrm{U}[0.1,0.2] \text{ uses } \mathrm{U}[0.2,2]}$, ${\mathrm{U}[1.1,1.2] \text{ uses } \mathrm{U}[1.2,6]}$, ${\mathrm{U}[1,2] \text{ uses } \mathrm{U}[2,6]}$, ${\mathrm{U}[10, 20] \text{ uses } \mathrm{U}[20, 40]}$.

\begin{figure}[h]
\centering
\includegraphics[width=\linewidth,trim={0 1.3cm 0 0},clip]{results/simple_function_static_mul_input_size.pdf}
\includegraphics[width=\linewidth,trim={0 1.3cm 0 0.809cm},clip]{results/simple_function_static_mul_overlap.pdf}
\includegraphics[width=\linewidth,trim={0 0 0 0.809cm},clip]{results/simple_function_static_mul_subset.pdf}
\caption{Shows the effect of the dataset parameters.}
\label{fig:simple-function-static-dataset-parameters-boundary}
\end{figure}

\subsection{Gating convergence experiment}
\label{sec:appendix:nalu-gate-experiment}

In the interest of adding some understand of what goes wrong in the NALU gate, and the shared weight choice that NALU employs to remedy this, we introduce the following experiment.

We train two models to fit the arithmetic task. Both uses the $\mathrm{NAC}_{+}$ in the first layer and NALU in the second layer. The only difference is that one model shares the weight between $\mathrm{NAC}_{+}$ and $\mathrm{NAC}_{\bullet}$ in the NALU, and the other treat them as two separate units with separate weights. In both cases NALU should gate between $\mathrm{NAC}_{+}$ and $\mathrm{NAC}_{\bullet}$ and choose the appropriate operation. Note that this NALU model is different from the one presented elsewhere in this paper, including the original NALU paper \cite{trask-nalu}. The typical NALU model is just two NALU layers with shared weights.

Furthermore, we also introduce a new gated unit that simply gates between our proposed NMU and NAU, using the same sigmoid gating-mechanism as in the NALU. This combination is done with seperate weights, as NMU and NAU use different weight constrains and can therefore not be shared.

The models are trained and evaluated over 100 different seeds on the multiplication and addition task. A histogram of the gate-value for all seeds is presented in figure \ref{fig:simple-function-static-nalu-gate-graph} and table \ref{tab:simple-function-static-nalu-gate-table} contains a summary. Some noteworthy observations:

\vspace{-0.3cm}\begin{enumerate}
    \item When the NALU weights are separated far more trials converge to select $\mathrm{NAC}_{+}$ for both the addition and multiplication task. Sharing the weights between $\mathrm{NAC}_{+}$ and $\mathrm{NAC}_{\bullet}$ makes the gating less likely to converge for addition.
    \item The performance of the addition task is dependent on NALU selecting the right operation. In the multiplication task, when the right gate is selected, $\mathrm{NAC}_{\bullet}$ do not converge consistently, unlike our NMU that converges more consistently.
    \item Which operation the gate converges to appears to be mostly random and independent of the task. These issues are caused by the sigmoid gating-mechanism and thus exists independent of the used sub-units.
\end{enumerate}

\vspace{-0.2cm}These observations validates that the NALU gating-mechanism does not converge as intended. This becomes a critical issues when more gates are present, as is normally the case. E.g. when stacking multiple NALU layers together.

\begin{figure}[h]
\centering
\includegraphics[width=0.93\linewidth]{results/function_task_static_nalu.pdf}
\vspace{-0.2cm}\caption{Shows the gating-value in the NALU layer and a variant that uses NAU/NMU instead of $\mathrm{NAC}_{+}$/$\mathrm{NAC}_{\bullet}$. Separate/shared refers to the weights in $\mathrm{NAC}_{+}$/$\mathrm{NAC}_{\bullet}$ used in NALU.}
\label{fig:simple-function-static-nalu-gate-graph}
\end{figure}

\input{results/function_task_static_nalu.tex}

\subsection{Regularization}
\label{sec:appendix:simple-function-task:regualization}

The $\lambda_{start}$ and $\lambda_{end}$ are simply selected based on how much time it takes for the model to converge. The sparsity regularizer should not be used during early optimization as this part of the optimization is exploratory and concerns finding the right solution by getting each weight on the right side of $\pm 0.5$.

In figure \ref{fig:simple-fnction-static-regularizer-add}, \ref{fig:simple-fnction-static-regularizer-sub} and \ref{fig:simple-fnction-static-regularizer-mul} the scaling factor $\hat{\lambda}_{\mathrm{sparse}}$ is optimized.
\begin{equation}
\lambda_{\mathrm{sparse}} = \hat{\lambda}_{\mathrm{sparse}} \max(\min(\frac{t - \lambda_{\mathrm{start}}}{\lambda_{\mathrm{end}} - \lambda_{\mathrm{start}}}, 1), 0)
\end{equation}

\begin{figure}[H]
\centering
\includegraphics[width=\linewidth,trim={0 1.3cm 0 0},clip]{results/simple_function_static_regualization_add.pdf}
\caption{Shows effect of $\hat{\lambda}_{\mathrm{sparse}}$ in NAU on the arithmetic dataset for the $\bm{+}$ operation.}
\label{fig:simple-fnction-static-regularizer-add}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=\linewidth,trim={0 1.3cm 0 0},clip]{results/simple_function_static_regualization_sub.pdf}
\caption{Shows effect of $\hat{\lambda}_{\mathrm{sparse}}$ in NAU on the arithmetic dataset for the $\bm{-}$ operation.}
\label{fig:simple-fnction-static-regularizer-sub}
\end{figure}

\begin{figure}[H]
\centering
\includegraphics[width=\linewidth,trim={0 1.3cm 0 0},clip]{results/simple_function_static_regualization_mul.pdf}
\caption{Shows effect of $\hat{\lambda}_{\mathrm{sparse}}$ in NMU on the arithmetic dataset for the $\bm{\times}$ operation.}
\label{fig:simple-fnction-static-regularizer-mul}
\end{figure}

\subsection{Comparing all models}
\label{sec:appendix:comparison-all-models}

Table \ref{tab:function-task-static-defaults-all} compares all models on all operations used in NALU \cite{trask-nalu}. All variations of models and operations are trained for 100 different seeds to build confidence intervals. Some noteworthy observations are:

\begin{enumerate}
    \item Division does not work for any model, including the $\mathrm{NAC}_{\bullet}$ and NALU models. This may seem surprising but is actually in line with the results from the NALU paper (\citet{trask-nalu}, table 1) where there is a large error given the interpolation range. The extrapolation range has a smaller error, but this is an artifact of their evaluation method where they normalize with a random baseline. Since a random baseline will have a higher error for the extrapolation range, errors just appear to be smaller. A correct solution to division should have both a small interpolation and extrapolation error.
    \item $\mathrm{NAC}_{\bullet}$ and NALU are barely able to learn $\sqrt{z}$, with just 2\% success-rate for NALU and 7\% success-rate for $\mathrm{NAC}_{\bullet}$.
    \item NMU is fully capable of learning $z^2$. It learn this by learning the same subset twice in the NAU layer, this is also how $\mathrm{NAC}_{\bullet}$ learn $z^2$.
    \item The Gated NAU/NMU (discussed in section \ref{sec:appendix:nalu-gate-experiment}) works very poorly, because the NMU initialization assumes that $E[z_{h_{\ell-1}}] = 0$. This is usually true, as discussed in section \ref{sec:methods:moments-and-initialization}, but not in this case for the first layer. In the recommended NMU model, the NMU layer appears after NAU, which causes that assumption to be satisfied.
\end{enumerate}

\input{results/function_task_static_all.tex}


%%%%%%%%%%%%%%%%%%%%%%%%
%
%   Thesis template by Youssif Al-Nashif
%
%   May 2020
%
%%%%%%%%%%%%%%%%%%%%%%%%

\section{Alternate Clustering Method with KDE}

As an alternative to popular clustering methods that were used in the preceding section, another clustering method was attempted that features use of kernel density estimation. By using a kernel density estimation (KDE) on the kernel values, we can cluster the documents into similar groups, defined by local maxima. \\
First, the graph kernel matrix, $K$ is taken, and we extract a row, $i$, and we compute a KDE using R's \texttt{density()} function. Now, the default value for bandwidth will likely produce a smooth, unimodal or bimodal distribution, but this is not what the goal is. The goal is to use the KDE to find clusters through their value appearing in a local maxima. So, through producing a KDE with few local maxima, we produce very few clusters. If the number of clusters needs to increase, we can essentially overfit the KDE and abuse the use of the bandwidth parameter to create a KDE with many more local maxima and minima.\\

%% INSERT GRAPHIC HERE.
**GRAPHIC**\\

Once a KDE with a sufficient number of local maxima, which is determined by the user, then cluster breaks are located. If we consider the estimated KDE to be a function $k(x)$, where $x$ is a kernel value, and $k(x)$ is the estimated density at a value $x$, then we can do calculus to locate the break points.


\documentclass[11pt,a4paper]{article}
\usepackage[utf8]{inputenc}
\usepackage[english]{babel}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage[left=2cm,right=2cm,top=2cm,bottom=2cm]{geometry}
\usepackage{hyperref}
\title{Three-Dimensional Idealized Model for Tidal Motion in Tidal Estuaries. Part II: An Efficient Finite-Element Multigrid Approach}
\author{Matthias M\"oller}
\begin{document}
\maketitle

\begin{abstract}
...
\end{abstract}

\section{Introduction}
...

\section{Mathematical Model}

The linear elliptic partial differential equation for the sea surface elevation as derived in [?] reads
\begin{alignat}{2}
\nabla\cdot(\mathbb{A}\nabla N)+i\omega N&=0  \quad && \text{in } \Omega,
\label{eq:sse_scalar1}\\
N&=A_{M_2} \quad && \text{on } \Gamma_D,
\label{eq:sse_scalar2}\\
(\mathbb{A}\nabla N)\cdot\mathbf{n}&=0 \quad && \text{on } \Gamma_N,
\label{eq:sse_scalar3}
\end{alignat}
where $\Gamma_D$ and $\Gamma_N$ denote the Dirichlet and Neumann boundary part and $A_{M_2}$ is the amplitude of sea surface elevation at the Dirichlet boundary. Both the sea surface elevation $N\in\mathbb{C}$ and the matrix
$$
\mathbb{A}=
\begin{bmatrix}
C_1 & C_2\\
C_3 & C_4
\end{bmatrix},\quad\text{where}\quad C_k\in\mathbb{C}, k=1,\dots,4
$$
consist of complex values, where $i$ denotes the imaginary number, i.e. $i^2=-1$.

Let us define the $z$-dependent function $C:\mathbb{C}\mapsto\mathbb{C}$ as
\begin{equation}
C(\alpha,z)=\frac{g}{\alpha^3A_v}\left[\frac{\alpha^2A_vz\sinh(\alpha h)-s\sinh(\alpha z)+\alpha z s \cosh(\alpha h)}{\alpha A_v\sinh(\alpha h)+s\cosh(\alpha h)}\right]
\end{equation}
and introduce the auxiliary real-valued parameters
$$
\alpha_1=\sqrt{i\frac{\omega+f}{A_\nu}},\qquad
\alpha_2=\sqrt{i\frac{\omega-f}{A_\nu}}.
$$
Then the complex-valued coefficients $C_1,\dots,C_4$ are given by
\begin{alignat}{4}
C_1&=&\frac{C(\alpha_1,-h)+C(\alpha_2,-h)}{2},&& \qquad
C_2&=&i\frac{C(\alpha_1,-h)-C(\alpha_2,-h)}{2},\\
C_3&=&-i\frac{C(\alpha_1,-h)-C(\alpha_2,-h)}{2},&&\qquad
C_4&=&\frac{C(\alpha_1,-h)+C(\alpha_2,-h)}{2}.
\end{alignat}
Note that $C_1=C_4$ and $C_2=-C_3$. The other parameters are given in the following table

\begin{center}
\begin{tabular}{|l|l|l|l|}
\hline
Symbol & Value & Description & Units\\
\hline
$(x,y,z)$ & & Cartesian coordinates & $m$\\
$t$ & & time & $s$\\
$L$ & & length scale & $m$\\
$H$ & & mean depth & $m$\\
$h(x,y)$ & 10 & bed profile & $m$\\
$\eta(x,y,t)$ & & sea surface elevation above $z=0$ & $m$\\
$\mathbf{u}(x,y,z,t)$ & & velocity vector & $m s^{-1}$\\
$f(x,y)$ & 0 & Coriolis acceleration (Earth rotation) & $s^{-1}$\\
$g$ & 10 & gravitational acceleration & $ms^{-2}$\\
$A_h(x,y)$ & & horizontal eddy viscosity & $m^2 s^{-1}$\\
$A_\nu(x,y)$ & 0.012 & vertical eddy viscosity & $m^2 s^{-1}$\\
$s(x,y)$ & 0.049 & bottom stress & $m s^{-1}$\\
$A_{M_2}(x,y)$ & 1.0 & $M_2$ amplitude at seaward side & $m$\\
$\omega$ & $1.4\times 10^{-4}$ &tidal frequency of $M_2$ tide & $s^{-1}$\\
\hline
\end{tabular}
\end{center}

\subsection{Variational formulation}
The variational formulation reads: find $N\in V_{A_{M_2}}$ such that
\begin{equation}
\int_\Omega\nabla \varphi\cdot(-\mathbb{A}\nabla N)+ i\omega \varphi N\,\mathrm{d}\mathbf{x}=0\qquad \forall \varphi\in V_0,
\end{equation}
The essential boundary conditions have been built into the trial and test spaces
\begin{align}
V_{A_{M_2}}&=\{v\in H^1(\Omega)\,:\, v=A_{M_2}\text{ on }\Gamma_D\},\\
V_0&=\{\varphi\in H^1(\Omega)\,:\, \varphi=0\text{ on }\Gamma_D\}.
\end{align}

\subsection{Finite element approximation}
Let the variable for the sea surface elevation be approximated by finite elements as follows
\begin{equation}
N(\mathbf{x})=\sum_{j}\varphi_j(\mathbf{x})N_j,
\end{equation}
where $\{\varphi_j(\mathbf{x})\}_j$ denotes a basis of \emph{real} valued functions and complex valued coefficients $N_j\in\mathbb{C}$.

Instead of working with complex values directly we split them into their real and imaginary parts, i.e. $N_j=N_j^\mathrm{Re}+iN_j^\mathrm{Im}$ and use separate scalar finite element fields $N^\mathrm{Re}$ and $N^\mathrm{Im}$ for discretizing each one of them. Here and below superscript $(\cdot)^\mathrm{Re}$ and $(\cdot)^\mathrm{Im}$ denote the real and imaginary part, respectively. Then the problem at hand reads: find $(N^\mathrm{Re},N^\mathrm{Im})\in V_{A_{M_2}^\mathrm{Re}}\times V_{A_{M_2}^\mathrm{Im}}$ such that
\begin{equation}
\sum_j\int_\Omega \nabla\varphi_i\cdot\left(-\mathbb{A}^\mathrm{Re}\nabla N_j^\mathrm{Re}+\mathbb{A}^\mathrm{Im}\nabla N_j^\mathrm{Im}\right)-\omega \varphi_i N_j^\mathrm{Im}\,\mathrm{d}\mathbf{x}=0
\end{equation}
and
\begin{equation}
\sum_ji\int_\Omega \nabla\varphi_i\cdot\left(-\mathbb{A}^\mathrm{Im}\nabla N_j^\mathrm{Re}-\mathbb{A}^\mathrm{Re}\nabla N_j^\mathrm{Im}\right)+\omega \varphi_i N_j^\mathrm{Re}\,\mathrm{d}\mathbf{x}=0
\end{equation}
for all admissible real valued test functions $\varphi_i\in V_{0}$.
To simplify the notation let us define the following auxiliary stiffness matrices and the standard consistent mass matrix
\begin{alignat}{2}
S^\mathrm{Re}&=\{s_{ij}^\mathrm{Re}\} \qquad s_{ij}^\mathrm{Re}&=&\int_\Omega \nabla\varphi_i\cdot(-\mathbb{A}^\mathrm{Re}\nabla\varphi_j)\,\mathrm{d}\mathbf{x},\\
S^\mathrm{Im}&=\{s_{ij}^\mathrm{Im}\} \qquad s_{ij}^\mathrm{Im}&=&\int_\Omega \nabla\varphi_i\cdot(-\mathbb{A}^\mathrm{Im}\nabla\varphi_j)\,\mathrm{d}\mathbf{x},\\
M&=\{m_{ij}\} \qquad m_{ij}&=&\int_\Omega \varphi_i\varphi_j\,\mathrm{d}\mathbf{x}.
\end{alignat}
Then the problem at hand can be written in compact matrix notation as follows:
\begin{equation}
\begin{bmatrix}
S^\mathrm{Re} & -S^\mathrm{Im}-\omega M\\
S^\mathrm{Im}+\omega M &  S^\mathrm{Re}
\end{bmatrix}
\begin{bmatrix}
N^\mathrm{Re}\\
N^\mathrm{Im}
\end{bmatrix}
=
\begin{bmatrix}
0\\
0
\end{bmatrix}.
\end{equation}

\section{First-order system formulation}

In what follows, we present two alternative reformulations of the second-order scalar model problem as a first order system. In essence, the two formulations vary in the treatment of the anisotropic diffusion operator $\mathbb{A}$.

\subsection{Model problems}
Let $\boldsymbol{\sigma}=\mathbb{A}\nabla N$ in order to convert the scalar problem equation \eqref{eq:sse_scalar1}--\eqref{eq:sse_scalar3} into the first-order system
\begin{alignat}{2}
\nabla\cdot\boldsymbol{\sigma}+i\omega N&=0  \quad && \text{in } \Omega,
\label{eq:sse_system1}\\
\boldsymbol{\sigma}-\mathbb{A}\nabla N&=0 \quad && \text{in } \Omega,
\label{eq:sse_system2}\\
N&=A_{M_2} \quad && \text{on } \Gamma_D,
\label{eq:sse_system3}\\
\boldsymbol{\sigma}\cdot\mathbf{n}&=0 \quad && \text{on } \Gamma_N.
\label{eq:sse_system4}
\end{alignat}

It is also possible to define $\boldsymbol{\tau}=\nabla N$ as the gradient of the sea surface elevation which yields the alternative first-order system resulting from the scalar problem equation \eqref{eq:sse_scalar1}--\eqref{eq:sse_scalar3}
\begin{alignat}{2}
\nabla\cdot(\mathbb{A}\boldsymbol{\tau})+i\omega N&=0  \quad && \text{in } \Omega,
\label{eq:sse_system5}\\
\boldsymbol{\tau}-\nabla N&=0 \quad && \text{in } \Omega,
\label{eq:sse_system6}\\
N&=A_{M_2} \quad && \text{on } \Gamma_D,
\label{eq:sse_system7}\\
(\mathbb{A}\boldsymbol{\tau})\cdot\mathbf{n}&=0 \quad && \text{on } \Gamma_N.
\label{eq:sse_system8}
\end{alignat}
Both approaches lead to slightly different variational formulations and, consequently, system matrices.

\subsection{Variational formulation I}
The variational formulation associated with \eqref{eq:sse_system1}--\eqref{eq:sse_system4} reads: find $N\in W$ and $\boldsymbol{\sigma}\in\mathbf{Q}_0$ such that
\begin{alignat}{2}
\int_\Omega\varphi\left(\nabla\cdot\boldsymbol{\sigma}+i\omega N\right)\,\mathrm{d}\mathbf{x}&=0 \qquad && \forall \varphi\in W,
\label{eq:weak1}\\
\int_\Omega\boldsymbol{\psi}\cdot\left(\boldsymbol{\sigma}-\mathbb{A}\nabla N\right)\,\mathrm{d}\mathbf{x}&=0 \qquad && \forall \boldsymbol{\psi}\in \mathbf{Q}_0,
\label{eq:weak2}
\end{alignat}
where the test and trial spaces are defined as follows:
\begin{equation}
W=L^2(\Omega),\quad
\mathbf{Q}_0=\{\mathbf{q}\in H(\text{div};\Omega)\,:\,\mathbf{q}\cdot\mathbf{n}=0 \text{ on }\Gamma_N\}.
\end{equation}
Note that the Neumann boundary condition \eqref{eq:sse_scalar2} of the second-order scalar problem becomes a Dirichlet boundary condition \eqref{eq:sse_system3} in the first-order system formulation, and hence, it has been implemented into the test and trial spaces. Since $\mathbb{A}^*=\overline{\mathbb{A}}^T$ and $\overline{\boldsymbol{\psi}}=\boldsymbol{\psi}$, it is clear that
\begin{equation}
\boldsymbol{\psi}\cdot(\mathbb{A}\nabla N)
=\boldsymbol{\psi}^T(\mathbb{A}\nabla N)
=(\boldsymbol{\psi}^T\mathbb{A})\nabla N
=(\mathbb{A}^*\boldsymbol{\psi})^*\nabla N
=\overline{\left(\overline{\mathbb{A}}^T\boldsymbol{\psi}\right)}^T\nabla N
=(\mathbb{A}^T\boldsymbol{\psi})\cdot\nabla N.
\end{equation}
Integration by parts is applied to \eqref{eq:weak2} to include the boundary condition \eqref{eq:sse_system3} as a natural one
\begin{align*}
\int_\Omega\boldsymbol{\psi}\cdot(\boldsymbol{\sigma}-\mathbb{A}\nabla N)\,\mathrm{d}\mathbf{x}
&=\int_\Omega\boldsymbol{\sigma}\cdot\boldsymbol{\psi}-(\mathbb{A}^T\boldsymbol{\psi})\cdot\nabla N\,\mathrm{d}\mathbf{x}\\
&=\int_\Omega\boldsymbol{\psi}\cdot\boldsymbol{\sigma}+\nabla\cdot(\mathbb{A}^T\boldsymbol{\psi})N\,\mathrm{d}\mathbf{x}-\int_\Gamma N(\mathbb{A}^T\boldsymbol{\psi})\cdot\mathbf{n}\,\mathrm{d}\mathbf{x}\\
&=\int_\Omega\boldsymbol{\psi}\cdot\boldsymbol{\sigma}+\nabla\cdot(\mathbb{A}^T\boldsymbol{\psi})N\,\mathrm{d}\mathbf{x}-\int_{\Gamma_D} A_{M_2}(\mathbb{A}^T\boldsymbol{\psi})\cdot\mathbf{n}\,\mathrm{d}s-\int_{\Gamma_N} N(\mathbb{A}^T\boldsymbol{\psi})\cdot\mathbf{n}\,\mathrm{d}s
\end{align*}
Thus, the variational formulation for the problem at hand reads: find $(N,\boldsymbol{\sigma})\in W\times\mathbf{Q}_0$ such that
\begin{align}
&\int_\Omega\varphi\left(\nabla\cdot\boldsymbol{\sigma}+i\omega N\right)+\boldsymbol{\psi}\cdot\boldsymbol{\sigma}+\nabla\cdot(\mathbb{A}^T\boldsymbol{\psi})N\,\mathrm{d}\mathbf{x}\\
=&\int_{\Gamma_D} A_{M_2}(\mathbb{A}^T\boldsymbol{\psi})\cdot\mathbf{n}\,\mathrm{d}s+\int_{\Gamma_N} N(\mathbb{A}^T\boldsymbol{\psi})\cdot\mathbf{n}\,\mathrm{d}s \qquad \forall (\varphi,\boldsymbol{\psi})\in W\times\mathbf{Q}_0.
\end{align}

\subsection{Variational formulation II}
The variational formulation associated with \eqref{eq:sse_system5}--\eqref{eq:sse_system8} reads: find $(N,\boldsymbol{\sigma})\in W\times\mathbf{Q}_0$ such that
\begin{alignat}{2}
\int_\Omega\varphi\left(\nabla\cdot(\mathbb{A}\boldsymbol{\tau})+i\omega N\right)\,\mathrm{d}\mathbf{x}&=0 \qquad && \forall \varphi\in W,
\label{eq:weak3}\\
\int_\Omega\boldsymbol{\psi}\cdot\left(\boldsymbol{\tau}-\nabla N\right)\,\mathrm{d}\mathbf{x}&=0 \qquad && \forall \boldsymbol{\psi}\in \mathbf{Q}_0.
\label{eq:weak4}
\end{alignat}
In order to impose the natural boundary conditions \eqref{eq:sse_system7}, we perform integration by parts in \eqref{eq:weak4}
\begin{equation}
\int_\Omega\boldsymbol{\psi}\cdot\boldsymbol{\tau}+
\nabla\cdot\boldsymbol{\psi}N\,\mathrm{d}\mathbf{x}=
\int_{\Gamma_D}A_{M_2}\boldsymbol{\psi}\cdot\mathbf{n}\,\mathrm{d}s+
\int_{\Gamma_N}N\boldsymbol{\psi}\cdot\mathbf{n}\,\mathrm{d}s.
\end{equation}
Thus, the variational formulation for the problem at hand reads: find $(N,\boldsymbol{\sigma})\in W\times\mathbf{Q}_0$ such that
\begin{align}
&\int_\Omega\varphi\left(\nabla\cdot(\mathbb{A}\boldsymbol{\tau})+i\omega N\right)+\boldsymbol{\psi}\cdot\boldsymbol{\tau}+\nabla\cdot\boldsymbol{\psi}N\,\mathrm{d}\mathbf{x}\\
=& \int_{\Gamma_D}A_{M_2}\boldsymbol{\psi}\cdot\mathbf{n}\,\mathrm{d}s+
\int_{\Gamma_N}N\boldsymbol{\psi}\cdot\mathbf{n}\,\mathrm{d}s
\qquad \forall (\varphi,\boldsymbol{\psi})\in W\times\mathbf{Q}_0.
\end{align}

\subsection{Finite element approximation}
Let us approximate the sea surface elevation and its derivative by finite elements again separating the real and imaginary parts as in the scalar case. That is,
\begin{alignat}{3}
N(\mathbf{x})&=\sum_{j}\varphi_j(\mathbf{x})N_j && \qquad N_j&=N_j^\mathrm{Re}+iN_j^\mathrm{Im}\\
\boldsymbol{\sigma}(\mathbf{x})=(\sigma_1(\mathbf{x}),\sigma_2(\mathbf{x}))^T&=\sum_{j}\psi_j(\mathbf{x})\boldsymbol{\sigma}_j && \qquad \boldsymbol{\sigma}_j&=\boldsymbol{\sigma}_j^\mathrm{Re}+i\boldsymbol{\sigma}_j^\mathrm{Im}\\
\boldsymbol{\tau}(\mathbf{x})=(\tau_1(\mathbf{x}),\tau_2(\mathbf{x}))^T&=\sum_{j}\psi_j(\mathbf{x})\boldsymbol{\tau}_j && \qquad \boldsymbol{\tau}_j&=\boldsymbol{\tau}_j^\mathrm{Re}+i\boldsymbol{\tau}_j^\mathrm{Im}
\end{alignat}
Note that $\{\psi_j\}$ are scalar basis functions which are multiplied by vector-valued coefficients in order to obtain a vector-valued finite element functions. Here, we do not use vector-valued finite elements such as Raviar-Thomas or Brezzi-Douglas-Marini finite elements.

\paragraph{First formulation.} The problem reads: find $(N^\mathrm{Re},N^\mathrm{Im},\boldsymbol{\sigma}^\mathrm{Re},\boldsymbol{\sigma}^\mathrm{Im})\in [W]^2\times [\mathbf{Q}_0]^2$ such that
\begin{align}
&\sum_j\int_\Omega \varphi_i\left(\nabla\cdot\boldsymbol{\sigma}_j^\mathrm{Re}-\omega N^\mathrm{Im}_j\right)+
\boldsymbol{\psi}_i\cdot\boldsymbol{\sigma}^\mathrm{Re}_j+
\nabla\cdot({(\mathbb{A}^\mathrm{Re})}^T\boldsymbol{\psi}_i)N^\mathrm{Re}_j-
\nabla\cdot({(\mathbb{A}^\mathrm{Im})}^T\boldsymbol{\psi}_i)N^\mathrm{Im}_j
\,\mathrm{d}\mathbf{x}\\
=&
\int_{\Gamma_D}\left(A^\mathrm{Re}_{M_2}({(\mathbb{A}^\mathrm{Re})}^T\boldsymbol{\psi}_i)-
A^\mathrm{Im}_{M_2}({(\mathbb{A}^\mathrm{Im})}^T\boldsymbol{\psi}_i)
\right)\cdot\mathbf{n}\,\mathrm{d}s
+
\int_{\Gamma_N}\left(N^\mathrm{Re}_j({(\mathbb{A}^\mathrm{Re})}^T\boldsymbol{\psi}_i)-
N^\mathrm{Im}_j({(\mathbb{A}^\mathrm{Im})}^T\boldsymbol{\psi}_i)
\right)\cdot\mathbf{n}\,\mathrm{d}s
\end{align}
and
\begin{align}
&\sum_ji\int_\Omega \varphi_i\left(\nabla\cdot\boldsymbol{\sigma}_j^\mathrm{Im}+\omega N^\mathrm{Re}_j\right)+
\boldsymbol{\psi}_i\cdot\boldsymbol{\sigma}^\mathrm{Im}_j+
\nabla\cdot({(\mathbb{A}^\mathrm{Re})}^T\boldsymbol{\psi}_i)N^\mathrm{Im}_j+
\nabla\cdot({(\mathbb{A}^\mathrm{Im})}^T\boldsymbol{\psi}_i)N^\mathrm{Re}_j
\,\mathrm{d}\mathbf{x}\\
&=
i\int_{\Gamma_D}\left(A^\mathrm{Re}_{M_2}({(\mathbb{A}^\mathrm{Im})}^T\boldsymbol{\psi}_i)+
A^\mathrm{Im}_{M_2}({(\mathbb{A}^\mathrm{Re})}^T\boldsymbol{\psi}_i)
\right)\cdot\mathbf{n}\,\mathrm{d}s
+
i\int_{\Gamma_N}\left(N^\mathrm{Re}_j({(\mathbb{A}^\mathrm{Im})}^T\boldsymbol{\psi}_i)+
N^\mathrm{Im}_j({(\mathbb{A}^\mathrm{Re})}^T\boldsymbol{\psi}_i)
\right)\cdot\mathbf{n}\,\mathrm{d}s
\end{align}
for all admissible pairs of real valued test functions $(\varphi_i,\boldsymbol{\psi}_i)\in W\times\mathbf{Q}_0$.

Let us define the following auxiliary matrices and vectors
\begin{alignat}{2}
A&=\{a_{ij}\} \qquad a_{ij}&=&\int_\Omega \psi_i\psi_j\,\mathrm{d}\mathbf{x}\\
B_x&=\{b_{ij,x}\} \qquad b_{ij,x}&=&\int_\Omega \partial_x\psi_i\varphi_j\,\mathrm{d}\mathbf{x}\\
B_y&=\{b_{ij,y}\} \qquad b_{ij,x}&=&\int_\Omega \partial_y\psi_i\varphi_j\,\mathrm{d}\mathbf{x}
\end{alignat}

\begin{equation}
\begin{bmatrix}
0 & -\omega M & B_x^T & 0 & B_y^T & 0\\
\omega M & 0 & 0 & B_x^T & 0 & B_y^T\\
B_x & 0 & A & 0 & 0 & 0\\
0 & B_x & 0 & A & 0 & 0\\
B_y & 0 & 0 & 0 & A & 0\\
0 & B_y & 0 & 0 & 0 & A
\end{bmatrix}
\begin{bmatrix}
N^\mathrm{Re}\\
N^\mathrm{Im}\\
\sigma_1^\mathrm{Re}\\
\sigma_1^\mathrm{Im}\\
\sigma_2^\mathrm{Re}\\
\sigma_2^\mathrm{Im}
\end{bmatrix}
=
\begin{bmatrix}
0\\
0\\
\phantom{i}\int_{\Gamma_D}A^\mathrm{Re}_{M_2}\partial_x\psi^x_i\,\mathrm{d}s\\
i\int_{\Gamma_D}A^\mathrm{Im}_{M_2}\partial_x\psi^x_i\,\mathrm{d}s\\
\phantom{i}\int_{\Gamma_D}A^\mathrm{Re}_{M_2}\partial_y\psi^y_i\,\mathrm{d}s\\
i\int_{\Gamma_D}A^\mathrm{Im}_{M_2}\partial_y\psi^y_i\,\mathrm{d}s
\end{bmatrix}
\end{equation}

\paragraph{Second formulation.} The problem reads: find $(N^\mathrm{Re},N^\mathrm{Im},\boldsymbol{\tau}^\mathrm{Re},\boldsymbol{\tau}^\mathrm{Im})\in [W]^2\times [\mathbf{Q}_0]^2$ such that
\begin{align}
&\sum_j\int_\Omega \varphi_i\left(\nabla\cdot(\mathbb{A}^\mathrm{Re}\boldsymbol{\tau}_j^\mathrm{Re}-\mathbb{A}^\mathrm{Im}\boldsymbol{\tau}_j^\mathrm{Im})-\omega N^\mathrm{Im}_j\right)+
\boldsymbol{\psi}_i\cdot\boldsymbol{\tau}^\mathrm{Re}_j+
\nabla\cdot\boldsymbol{\psi}_iN^\mathrm{Re}_j
\,\mathrm{d}\mathbf{x}\\
&=
\int_{\Gamma_D}(\boldsymbol{\psi}_i\cdot\mathbf{n})A^\mathrm{Re}_{M_2}\,\mathrm{d}s
+
\sum_j\int_{\Gamma_N}(\boldsymbol{\psi}_i\cdot\mathbf{n})N^\mathrm{Re}_j\,\mathrm{d}s
\end{align}
and
\begin{align}
&\sum_ji\int_\Omega \varphi_i\left(\nabla\cdot(\mathbb{A}^\mathrm{Re}\boldsymbol{\tau}_j^\mathrm{Im}+\mathbb{A}^\mathrm{Im}\boldsymbol{\tau}_j^\mathrm{Re})+\omega N^\mathrm{Re}_j\right)+
\boldsymbol{\psi}_i\cdot\boldsymbol{\tau}^\mathrm{Im}_j+
\nabla\cdot\boldsymbol{\psi}_iN^\mathrm{Im}_j
\,\mathrm{d}\mathbf{x}\\
&=
i\int_{\Gamma_D}(\boldsymbol{\psi}_i\cdot\mathbf{n})A^\mathrm{Im}_{M_2}\,\mathrm{d}s
+
\sum_ji\int_{\Gamma_N}(\boldsymbol{\psi}_i\cdot\mathbf{n})N^\mathrm{Im}_j\,\mathrm{d}s
\end{align}
for all admissible pairs of real valued test functions $(\varphi_i,\boldsymbol{\psi}_i)\in W\times\mathbf{Q}_0$.
In addition to the mass matrices $A$ and $M$ introduced above, let us define the following auxiliary matrices and vectors
\begin{alignat}{2}
C^\mathrm{Re}_k&=\{c^\mathrm{Re}_{ij,k}\} \qquad c^\mathrm{Re}_{ij,k}&=&
\int_\Omega \varphi_i\nabla\cdot(\mathbf{a}^\mathrm{Re}_k\psi_j)\,\mathrm{d}\mathbf{x}
=
\int_\Omega \varphi_i\left(\partial_x(a^\mathrm{Re}_{1k}\psi_j)+\partial_y(a^\mathrm{Re}_{2k}\psi_j)\right)\,\mathrm{d}\mathbf{x}\\
C^\mathrm{Im}_k&=\{c^\mathrm{Im}_{ij,k}\} \qquad c^\mathrm{Im}_{ij,k}&=&
\int_\Omega \varphi_i\nabla\cdot(\mathbf{a}^\mathrm{Im}_k\psi_j)\,\mathrm{d}\mathbf{x}
=
\int_\Omega \varphi_i\left(\partial_x(a^\mathrm{Im}_{1k}\psi_j)+\partial_y(a^\mathrm{Im}_{2k}\psi_j)\right)\,\mathrm{d}\mathbf{x}\\
D_k&=\{d_{ij,k}\} \qquad d_{ij,k}&=&
\int_{\Gamma_N}\psi_i\varphi_jn_k\,\mathrm{d}s\\
b^\mathrm{Re}_k&=\{b^\mathrm{Re}_{i,k}\} \qquad b^\mathrm{Re}_{i,k}&=&
\int_{\Gamma_D}A^\mathrm{Re}_{M_2}\psi_in_k\,\mathrm{d}s\\
b^\mathrm{Im}_k&=\{b^\mathrm{Im}_{i,k}\} \qquad b^\mathrm{Im}_{i,k}&=&
\int_{\Gamma_D}A^\mathrm{Im}_{M_2}\psi_in_k\,\mathrm{d}s
\end{alignat}
where $\mathbf{a}^\mathrm{Re}_k$ and $\mathbf{a}^\mathrm{Im}_k$ denotes the real and imaginary parts of the $k$-th column of matrix $\mathbb{A}$ and $n_k$ represents the $k$-th component of the outward unit normal vector $\mathbf{n}=(n_1,n_2)$.

Then, the matrix form reads
\begin{equation}
\begin{bmatrix}
0 & -\omega M & C_1^\mathrm{Re} & -C_1^\mathrm{Im} & C_2^\mathrm{Re} & -C_2^\mathrm{Im}\\[1ex]
\omega M & 0 & C_1^\mathrm{Im} & C_1^\mathrm{Re} & C_2^\mathrm{Im} & C_2^\mathrm{Re}\\[1ex]
B_x+D_1 & 0 & A & 0 & 0 & 0\\[1ex]
0 & B_x+D_1 & 0 & A & 0 & 0\\[1ex]
B_y+D_2 & 0 & 0 & 0 & A & 0\\[1ex]
0 & B_y+D_2 & 0 & 0 & 0 & A
\end{bmatrix}
\begin{bmatrix}
N^\mathrm{Re}\\[1ex]
N^\mathrm{Im}\\[1ex]
\tau_1^\mathrm{Re}\\[1ex]
\tau_1^\mathrm{Im}\\[1ex]
\tau_2^\mathrm{Re}\\[1ex]
\tau_2^\mathrm{Im}
\end{bmatrix}
=
\begin{bmatrix}
0\\[1ex]
0\\[1ex]
b_1^\mathrm{Re}\\[1ex]
b_1^\mathrm{Im}\\[1ex]
b_2^\mathrm{Re}\\[1ex]
b_2^\mathrm{Im}
\end{bmatrix}
\end{equation}

\bibliographystyle{apalike}
\bibliography{mybib.bib}
\end{document}


% declare document class and geometry
\documentclass[12pt]{article} % use larger type; default would be 10pt
\usepackage[margin=1in]{geometry} % handle page geometry

% import packages and commands
\input{../header2.tex}

\title{Phys 220A -- Classical Mechanics -- HW06}
\author{UCLA, Fall 2014}
\date{\formatdate{12}{11}{2014}} % Activate to display a given date or no date (if empty),
         % otherwise the current date is printed

\begin{document}
\maketitle

\section*{Problem 1 (15 pts)}
\textit{
A particle with mass $m$ and charge $e$ moves in a magnetic field $\v B$.
}

\begin{enumproblem}

% part A
\item \textit{
Show that the following Poisson brackets hold
\begin{eqn}
\cbr{m \dot x_i, m \dot x_j} = e \epsilon_{ijk} B_k, \qquad
\cbr{m \dot x_i, x_j} = - \delta_{ij}.
\end{eqn}
}

% part B
\item \textit{
Consider the magnetic field of a magnetic monopole with
\begin{eqn}
\v B = g_m \frac{\uv x}{\abs{x}^2}
\end{eqn}
where $\uv x$ is the unit vector in the $\v x$ direction. Show that the following generalization of the angular momentum
\begin{eqn}
\v J = m \v x \times \vd x - eg \uv x
\end{eqn}
commutes with the Hamiltonian $H$.
}

% part C
\item \textit{
Can you interpret the fact that the usual angular momentum is not conserved but the new $\v J$ defined above is?
}

\end{enumproblem}

\section*{Problem 2 (15 pts)}
\textit{
Show that the following coordinate transformations are canonical.
}

\begin{enumproblem}

% part A
\item \textit{
$P = \frac{1}{2}(p^2 + q^2), \quad Q = \arctan(q/p)$.
}

% part B
\item \textit{
$P = 1/q, \quad Q = pq^2$.
}

% part C
\item \textit{
$P = 2 \sqrt{q} (1 + \sqrt{q} \cos p) \sin p, \quad Q = \log(1 + \sqrt{q} \cos p)$.
}

\end{enumproblem}

\section*{Problem 3 (10 pts)}
\textit{
For a particle moving in one dimension in a uniform force field the Lagrangian is given by
\begin{eqn}
L(q, \dot q) = \frac{1}{2} m \dot q^2 + m a q.
\end{eqn}
}

\begin{enumproblem}

% part A
\item \textit{
Solve the Hamilton equations and find the map
\begin{eqn}
g_t : \set{q(0), p(0)} \mapsto \set{q(t), p(t)}.
\end{eqn}
}

% part B
\item \textit{
Determine the shape of a square in phase space given by $\set{q,p} = \set{\set{0,0}, \set{\alpha,0}, \set{0,\beta}, \set{\alpha,\beta}}$ and verify that the volume is preserved.
}

\end{enumproblem}

\section*{Problem 4 (10 pts)}

\begin{enumproblem}

% part A
\item \textit{
Verify that the symplectic tensor $J$ expressed in new coordinates a transformation $Q_i = Q_i(p_j, q_k), P_i = P_i(p_j, q_k)$ takes the form
\begin{eqn}
J_{IJ} =
\begin{pmatrix}
\set{Q_i, Q_j} & \set{Q_i, P_j} \\
\set{P_j, Q_j} & \set{P_i, P_j}
\end{pmatrix}.
\end{eqn}
}

% part B
\item \textit{
Verify the Jacobi-identity
\begin{eqn}
\set{A, \set{B,C}} + \set{B,\set{C,A}} + \set{C,\set{A,B}} = 0.
\end{eqn}
}

\end{enumproblem}

\end{document}


\chapter{Multivariable calculus done correctly}
As I have ranted about before, linear algebra is done wrong
by the extensive use of matrices to obscure the structure of a linear map.
Similar problems occur with multivariable calculus, so here I would like to set
the record straight.

Since we are doing this chapter using morally correct linear algebra,
it's imperative you're comfortable with linear maps,
and in particular the dual space $V^\vee$ which we will repeatedly use.

In this chapter, all vector spaces have norms and
are finite-dimensional over $\RR$.
So in particular every vector space is also a metric space
(with metric given by the norm), and we can talk about open sets as usual.

\section{The total derivative}
\prototype{If $f(x,y) = x^2+y^2$, then $(Df)_{(x,y)} = 2x\ee_1^\vee + 2y\ee_2^\vee$.}
First, let $f : [a,b] \to \RR$.
You might recall from high school calculus that for every point $p \in \RR$,
we defined $f'(p)$ as the derivative at the point $p$ (if it existed), which we interpreted as the \emph{slope} of
the ``tangent line''.

\begin{center}
	\begin{asy}
		import graph;
		size(150,0);

		real f(real x) {return 3-2/(x+2.5);}
		graph.xaxis("$x$");
		graph.yaxis();
		draw(graph(f,-2,2,operator ..), blue, Arrows);

		real p = -1;
		real h = 1000 * (f(p+0.001)-f(p));
		real r = 0.9;
		draw( (p+r,f(p)+r*h)--(p-r,f(p)-r*h), red);
		dot( (p, f(p)), red );
		draw( (p, f(p))--(p,0), dashed);
		dot("$p$", (p, 0), dir(-90));
		label("$f'(p)$", (p+r/2, f(p) + h*r/2), dir(115), red);
	\end{asy}
\end{center}

That's fine, but I claim that the ``better'' way to interpret
the derivative at that point is as a \emph{linear map},
that is, as a \emph{function}.
If $f'(p) = 1.5$,
then the derivative tells me that if I move $\eps$ away from $p$
then I should expect $f$ to change by about $1.5\eps$.
In other words,
\begin{moral}
The derivative of $f$ at $p$ approximates $f$ near $p$ by a \emph{linear function}.
\end{moral}

What about more generally?
Suppose I have a function like $f : \RR^2 \to \RR$, say
\[ f(x,y) = x^2+y^2 \]
for concreteness or something.
For a point $p \in \RR^2$, the ``derivative'' of $f$ at $p$ ought to represent a linear map
that approximates $f$ at that point $p$.
That means I want a linear map $T : \RR^2 \to \RR$ such that
\[ f(p + v) \approx f(p) + T(v) \]
for small displacements $v \in \RR^2$.

Even more generally, if $f : U \to W$ with $U \subseteq V$ open
(in the $\norm{\bullet}_V$ metric as usual),
then the derivative at $p \in U$ ought to be so that
\[ f(p + v) \approx f(p) + T(v) \in W. \]
(We need $U$ open so that for small enough $v$, $p+v \in U$ as well.)
In fact this is exactly what we were doing earlier with $f'(p)$ in high school.

\begin{center}
	\includegraphics{media/tangent.pdf}
	\\ \scriptsize Image derived from \cite{img:tangentplane}
\end{center}

The only difference is that, by an unfortunate coincidence,
a linear map $\RR \to \RR$ can be represented by just its slope.
And in the unending quest to make everything a number so that it can be AP tested,
we immediately forgot all about what we were trying to do in the first place
and just defined the derivative of $f$ to be a \emph{number} instead of a \emph{function}.

\begin{moral}
	The fundamental idea of Calculus is the local approximation of functions by linear functions.
	The derivative does exactly this.
\end{moral}
Jean Dieudonn\'e as quoted in \cite{ref:pugh} continues:
\begin{quote}
	In the classical teaching of Calculus, this idea is immediately obscured
	by the accidental fact that, on a one-dimensional vector space,
	there is a one-to-one correspondence between linear forms and numbers,
	and therefore the derivative at a point is defined as a number instead of a linear form.
	This \textbf{slavish subservience to the shibboleth of numerical interpretation at any cost}
	becomes much worse . . .
\end{quote}

So let's do this right.
The only thing that we have to do is say what ``$\approx$'' means, and for
this we use the norm of the vector space.
\begin{definition}
	Let $U \subseteq V$ be open.
	Let $f : U \to W$ be a continuous function, and $p \in U$.
	Suppose there exists a linear map $T : V \to W$ such that
	\[
		\lim_{\norm{v}_V \to 0}
		\frac{\norm{f(p + v) - f(p) - T(v)}_W}{\norm{v}_V} = 0.
	\]
	Then $T$ is the \vocab{total derivative} of $f$ at $p$.
	We denote this by $(Df)_p$, and say $f$ is \vocab{differentiable at $p$}.

	If $(Df)_p$ exists at every point, we say $f$ is \vocab{differentiable}.
\end{definition}

\begin{ques}
	Check if that $V = W = \RR$, this is equivalent to the single-variable definition.
	(What are the linear maps from $V$ to $W$?)
\end{ques}
\begin{example}[Total derivative of $f(x,y) = x^2+y^2$]
	Let $V = \RR^2$ with standard basis $\ee_1$, $\ee_2$ and let $W = \RR$,
	and let $f\left( x \ee_1 + y \ee_2 \right) = x^2+y^2$.  Let $p = a\ee_1 + b\ee_2$.
	Then, we claim that \[ (Df)_p : \RR^2 \to \RR \quad\text{by}\quad
	v \mapsto 2a \cdot \ee_1^\vee(v) + 2b \cdot \ee_2^\vee(v). \]
\end{example}
Here, the notation $\ee_1^\vee$ and $\ee_2^\vee$ makes sense,
because by definition $(Df)_p \in V^\vee$: these are functions from $V$ to $\RR$!

Let's check this manually with the limit definition.
Set $v = xe_1 + ye_2$, and note that the norm on $V$ is $\norm{(x,y)}_V = \sqrt{x^2+y^2}$
while the norm on $W$ is just the absolute value $\norm{c}_W = \left\lvert c \right\rvert$.
Then we compute
\begin{align*}
	\frac{\norm{f(p + v) - f(p) - T(v)}_W}{\norm{v}_V}
	&= \frac{\left\lvert (a+x)^2 + (b+y)^2 - (a^2+b^2) - (2ax+2by) \right\rvert}{\sqrt{x^2+y^2}} \\
	&= \frac{x^2+y^2}{\sqrt{x^2+y^2}} = \sqrt{x^2+y^2} \\
	&\to 0
\end{align*}
as $\norm{v} \to 0$.
Thus, for $p = ae_1 + be_2$ we indeed have $(Df)_p = 2a \cdot e_1^\vee + 2b \cdot e_2^\vee$.

\begin{remark}
	As usual, differentiability implies continuity.
\end{remark}
\begin{remark}
	Although $U \subseteq V$, it might be helpful to think of vectors from $U$ and $V$
	as different types of objects (in particular, note that it's possible for $0_V \notin U$).
	The vectors in $U$ are ``inputs'' on our space
	while the vectors coming from $V$ are ``small displacements''.
	For this reason, I deliberately try to use $p \in U$ and $v \in V$ when possible.
\end{remark}

\section{The projection principle}
Before proceeding I need to say something really important.
\begin{theorem}[Projection principle]
	\label{thm:project_principle}
	Let $U$ be an open subset of the vector space $V$.
	Let $W$ be an $n$-dimensional real vector space with basis $w_1, \dots, w_n$.
	Then there is a bijection between continuous functions $f : U \to W$ and
	$n$-tuples of continuous $f_1, f_2, \dots, f_n : U \to \RR$
	by projection onto the $i$th basis element, i.e.\
	\[ f(v) = f_1(v)w_1 + \dots + f_n(v)w_n. \]
\end{theorem}
\begin{proof}
	Obvious.
\end{proof}
The theorem remains true if one replaces ``continuous'' by ``differentiable'', ``smooth'', ``arbitrary'',
or most other reasonable words. Translation:
\begin{moral}
To think about a function $f : U \to \RR^{n}$,
it suffices to think about each coordinate separately.
\end{moral}
For this reason, we'll most often be interested in functions $f : U \to \RR$.
That's why the dual space $V^\vee$ is so important.

\section{Total and partial derivatives}
\prototype{If $f(x,y) = x^2+y^2$, then
$(Df) : (x,y) \mapsto 2x \cdot \ee_1^\vee + 2y \cdot \ee_2^\vee$, and
$\fpartial fx = 2x$, $\fpartial fy = 2y$.}
Let $U \subseteq V$ be open and let $V$ have a basis $e_1$, \dots, $e_n$.
Suppose $f : U \to \RR$ is a function which is differentiable everywhere,
meaning $(Df)_p \in V^\vee$ exists for every $p$.
In that case, one can consider $Df$ as \emph{itself} a function:
\begin{align*}
	Df : U &\to V^\vee \\
	p &\mapsto (Df)_p.
\end{align*}
This is a little crazy: to every \emph{point} in $U$
we associate a \emph{function} in $V^\vee$.
We say $Df$ is the \vocab{total derivative} of $f$,
to reflect how much information we're dealing with.
We say $(Df)_p$ is the total derivative at $p$.

Let's apply the projection principle now to $Df$.
Since we picked a basis $e_1$, \dots, $e_n$ of $V$,
there is a corresponding dual basis
$e_1^\vee$, $e_2^\vee$, \dots, $e_n^\vee$.
The Projection Principle tells us that $Df$ can thus be thought of as just $n$ functions, so we can write
\[ Df = \psi_1 e_1^\vee + \dots + \psi_n e_n^\vee.  \]
In fact, we can even describe what the $\psi_i$ are.
\begin{definition}
	The \vocab{$i^{\text{th}}$ partial derivative} of $f : U \to \RR$, denoted
	\[ \fpartial{f}{e_i}: U \to \RR \]
	is defined by
	\[
		\fpartial{f}{e_i} (p)
		\defeq \lim_{t \to 0} \frac{f(p + te_i) - f(p)}{t}.
	\]
\end{definition}
You can think of it as ``$f'$ along $e_i$''.
\begin{ques}
	Check that if $Df$ exists, then \[ (Df)_p(e_i) = \fpartial{f}{e_i}(p). \]
\end{ques}
\begin{remark}
	Of course you can write down a definition of $\fpartial{f}{v}$
	for any $v$ (rather than just the $e_i$).
\end{remark}

From the above remarks, we can derive that
\[
	\boxed{
	Df =
	\frac{\partial f}{\partial e_1} \cdot e_1^\vee
	+ \dots +
	\frac{\partial f}{\partial e_n} \cdot e_n^\vee .
	}
\]
and so given a basis of $V$, we can think of $Df$ as just
the $n$ partials.
\begin{remark}
Keep in mind that each $\frac{\partial f}{\partial e_i}$ is a function from $U$ to the \emph{reals}.
That is to say,
\[
	(Df)_p =
	\underbrace{\frac{\partial f}{\partial e_1}(p)}_{\in \RR} \cdot e_1^\vee
	+ \dots +
	\underbrace{\frac{\partial f}{\partial e_n}(p)}_{\in \RR} \cdot e_n^\vee
	\in V^\vee.
\]
\end{remark}

\begin{example}[Partial derivatives of $f(x,y) = x^2+y^2$]
	Let $f : \RR^2 \to \RR$ by $(x,y) \mapsto x^2+y^2$.
	Then in our new language,
	\[ Df : (x,y) \mapsto 2x \cdot \ee_1^\vee + 2y \cdot \ee_2^\vee. \]
	Thus the partials are
	\[
		\frac{\partial f}{\partial x} : (x,y) \mapsto 2x \in \RR
		\quad\text{and}\quad
		\frac{\partial f}{\partial y} : (x,y) \mapsto 2y \in \RR
	\]
\end{example}

With all that said, I haven't really said much about how to
find the total derivative itself.
For example, if I told you
\[ f(x,y) = x \sin y + x^2y^4 \]
you might want to be able to compute $Df$ without going through
that horrible limit definition I told you about earlier.

Fortunately, it turns out you already know how to compute partial derivatives,
because you had to take AP Calculus at some point in your life.
It turns out for most reasonable functions, this is all you'll ever need.
\begin{theorem}[Continuous partials implies differentiable]
	\label{thm:apcalc_partials}
	Let $U \subseteq V$ be open and pick any basis $e_1, \dots, e_n$.
	Let $f : U \to \RR$ and suppose that $\fpartial{f}{e_i}$ is defined
	for each $i$ and moreover is \emph{continuous}.
	Then $f$ is differentiable and $Df$ is given by
	\[ Df = \sum_{i=1}^n \fpartial{f}{e_i} \cdot e_i^\vee. \]
\end{theorem}
\begin{proof}
	Not going to write out the details, but\dots
	given $v = t_1e_1 + \dots + t_ne_n$,
	the idea is to just walk from $p$ to $p+t_1e_1$, $p+t_1e_1+t_2e_2$, \dots,
	up to $p+t_1e_1+t_2e_2+\dots+t_ne_n = p+v$,
	picking up the partial derivatives on the way.
	Do some calculation.
\end{proof}

\begin{remark}
	The continuous condition cannot be dropped. The function
	\[
		f(x,y)
		= \begin{cases}
			\frac{xy}{x^2+y^2} & (x,y) \neq (0,0) \\
			0 & (x,y) = (0,0).
		\end{cases}
	\]
	is the classic counterexample -- the total derivative $Df$ does not exist at zero,
	even though both partials do.
\end{remark}

\begin{example}
	[Actually computing a total derivative]
	Let $f(x,y) = x \sin y + x^2y^4$. Then
	\begin{align*}
		\fpartial fx (x,y) &= \sin y + y^4 \cdot 2x \\
		\fpartial fy (x,y) &= x \cos y + x^2 \cdot 4y^3.
	\end{align*}
	So \Cref{thm:apcalc_partials} applies,
	and $Df = \fpartial fx \ee_1^\vee + \fpartial fy \ee_2^\vee$,
	which I won't bother to write out.
\end{example}

The example $f(x,y) = x^2+y^2$ is the same thing.
That being said, who cares about $x \sin y + x^2y^4$ anyways?

\section{(Optional) A word on higher derivatives}
Let $U \subseteq V$ be open, and take $f : U \to W$, so that $Df : U \to \Hom(V,W)$.

Well, $\Hom(V,W)$ can also be thought of as a normed vector space in its own right:
it turns out that one can define an operator norm on it by setting
\[ \norm{T} \defeq \sup \left\{ \frac{\norm{T(v)}_W}{\norm{v}_V} \mid v \neq 0_V \right\}. \]
So $\Hom(V,W)$ can be thought of as a normed vector space as well.
Thus it makes sense to write
\[ D(Df) : U \to \Hom(V,\Hom(V,W)) \]
which we abbreviate as $D^2 f$. Dropping all doubt and plunging on,
\[ D^3f : U \to \Hom(V, \Hom(V,\Hom(V,W))). \]
I'm sorry.
As consolation, we at least know that $\Hom(V,W) \cong V^\vee \otimes W$ in a natural way,
so we can at least condense this to
\[ D^kf : V \to (V^\vee)^{\otimes k} \otimes W \]
rather than writing a bunch of $\Hom$'s.
\begin{remark}
	If $k=2$, $W = \RR$, then $D^2f(v) \in (V^\vee)^{\otimes 2}$,
	so it can be represented as an $n \times n$ matrix,
	which for some reason is called a \vocab{Hessian}.
\end{remark}
The most important property of the second derivative is that
\begin{theorem}
	[Symmetry of $D^2 f$]
	Let $f : U \to W$ with $U \subseteq V$ open.
	If $(D^2f)_p$ exists at some $p \in U$, then it is symmetric, meaning
	\[ (D^2f)_p(v_1, v_2) = (D^2f)_p(v_2, v_1). \]
\end{theorem}
I'll just quote this without proof (see e.g. \cite[\S5, theorem 16]{ref:pugh}),
because double derivatives make my head spin.
An important corollary of this theorem:
\begin{corollary}
	[Clairaut's theorem: mixed partials are symmetric]
	Let $f : U \to \RR$ with $U \subseteq V$ open be twice differentiable.
	Then for any point $p$ such that the quantities are defined,
	\[
		\frac{\partial}{\partial e_i}
		\frac{\partial}{\partial e_j}
		f(p)
		=
		\frac{\partial}{\partial e_j}
		\frac{\partial}{\partial e_i}
		f(p).
	\]
\end{corollary}

\section{Towards differential forms}
This concludes the exposition of what the derivative really is:
the key idea I want to communicate in this chapter is that $Df$
should be thought of as a map from $U \to V^\vee$.

The next natural thing to do is talk about \emph{integration}.
The correct way to do this is through a so-called \emph{differential form}:
you'll finally know what all those stupid $dx$'s and $dy$'s really mean.
(They weren't just there for decoration!)

\section\problemhead
\begin{sproblem}[Chain rule]
	Let $U_1 \taking f U_2 \taking g U_3$ be differentiable maps
	between open sets of normed vector spaces $V_i$, and let $h = g \circ f$.
	Prove the Chain Rule: for any point $p \in U_1$, we have
	\[ (Dh)_p = (Dg)_{f(p)} \circ (Df)_p. \]
\end{sproblem}

\begin{problem}
	Let $U \subseteq V$ be open, and $f : U \to \RR$ be differentiable $k$ times.
	Show that $(D^kf)_p$ is symmetric in its $k$ arguments, meaning for any $v_1, \dots, v_k \in V$
	and any permutation $\sigma$ on $\left\{ 1, \dots, k \right\}$ we have
	\[ (D^kf)_p(v_1, \dots, v_k) = (D^kf)_p(v_{\sigma(1)}, \dots, v_{\sigma(k)}). \]
	\begin{hint}
		Simply induct, with the work having been done on the $k=2$ case.
	\end{hint}
\end{problem}


\section{Our technique}

\subsection{Basic technique}
\label{sec-basic-technique}

To illustrate our technique, we first show a very simple version of it
in the form of the following code:%
\footnote{Throughout this paper, we rely on the left-to-right
  evaluation order mandated by the \commonlisp{} standard.}

{\small
\begin{verbatim}
(defun count-from-end (x list)
  (labels ((aux (x list n)
             (cond ((= n 0) 0)
                   ((= n 1)
                    (if (eq x (car list)) 1 0))
                   (t (let* ((n/2 (ash n -1))
                             (half (nthcdr n/2 list)))
                        (+ (aux x half (- n n/2))
                           (aux x list n/2)))))))
    (aux x list (length list))))))
\end{verbatim}
}

This function starts by computing the length of the list and then
calling the auxiliary function with the original arguments and the
length.  The auxiliary function calls \texttt{nthcdr} in order to get
a reference to about half the list it was passed.  Then it makes two
recursive calls, first with the second half of the list and then with
the first half of the list.  The recursion terminates when the list has
a single element or no element in it.  When it has no element in it,
clearly the count is $0$.  When it has a single element in it, the
element is compared to the argument \texttt{x} and if they are the
same, the value $1$ is returned, otherwise $0$ is returned.

The main feature of our technique is that it trades fewer recursive
calls for multiple traversals of the list.  The maximum number of
simultaneous active invocations of this simple function is around
$\mathsf{lb}\thinspace n$, where $n$ is the length of the list.  The
maximum value of this number is quite modest.  On a 64-bit processor,
it can never exceed $60$ and it is significantly smaller in practice
of course.
% not relevant for count but for find
% The number of times this function computes the
% \texttt{cdr} of a list depends on where in the list the item to be
% found is located.  If it is the \emph{last} element of the list (best
% case), each \texttt{cons} cell is processed twice; once to compute the
% length of the list, and once again as part of the recursive traversal.
% When the item to be found is the \emph{first} element of the list
% (worst case),
The number of \texttt{cdr} operations can be
approximately expressed as $n\thinspace (1 +
\frac{1}{2}\mathsf{lb}\thinspace n)$.  In \refSec{sec-analyses} we
analyze this result in greater depth.

The best case for this function is very efficient indeed.
The worst case is unacceptably slow.  Even for a list of some
reasonable length such as a million elements, the execution time is a
factor $6$ slower than for the best case.

The remainder of this section is dedicated to ways of improving on the
performance of the basic technique.

\subsection{Using more stack space}
\label{sec-more-stack}

By far the most important improvement to the basic technique is to
take advantage of the available stack space to decrease the number of
multiple list traversals required by the basic technique.

The following example illustrates this technique by using the simple
recursive traversal if there are fewer than $10000$ elements in the
list.%
\footnote{The number $10000$ was chosen to be a significant part of a
  typical per-thread default stack while still leaving room for stack
  space required by callers and callees of this function.  In a real
  production implementation, the number would be chosen based on the
  remaining space left on the stack when the function is called.}
If there are more elements, then it divides the list in two,
just like the basic technique shown in \refSec{sec-basic-technique}.

{\small
\begin{verbatim}
(defun count-from-end-2 (x list)
  (labels ((recursive (x list n)
             (if (zerop n)
                 0
                 (+ (recursive x (cdr list) (1- n))
                    (if (eq x (car list)) 1 0))))
           (aux (x list n)
             (if (<= n 10000)
                 (recursive x list n)
                 (let* ((n/2 (ash n -1))
                        (half (nthcdr n/2 list)))
                   (+
                    (aux x half (- n n/2))
                    (aux x list n/2))))))
    (aux x list (length list))))
\end{verbatim}
}

With this improvement, the number of \texttt{cdr} operations required
can now be expressed as ap\-proximately
\[
n\thinspace (1 +
\frac{1}{2}\mathsf{lb}\thinspace \frac{n}{10000})
\]
which is significantly better than the corresponding value for the
basic technique.

However, there is no particular reason to divide the list into $2$
equal-sized parts when there are too many elements for the basic
technique. \refSec{sec-benchmarks} gives a more complete explanation
of the parameters involved and how they influence the execution time
of the resulting code.

\subsection{Analyses}
\label{sec-analyses}

In this section we give approximate formulas for the performance of
our technique.
The basic measure we are interested in is the number
of \texttt{cdr} operations that must be performed as a function of the
number of elements of the list.  We will denote the number of elements
of the list by $N$ and the number of \texttt{cdr} operations required
by $F(N)$.  Since our technique always starts by traversing the entire
list in order to compute $N$, we can always write $F(N)$ as $N +
f(N)$, were $f(N)$ is the number of \texttt{cdr} operations required
in the subsequent step.

For the basic technique where the list is divided into two equal-size
sublists, we obtain the following recursive relation:

\label{analyse1}
\[ f(N) = \left\{ \begin{array}{ll}
                    0 & \mbox{if $N = 1$} \\
                    \left\lfloor\frac{N}{2}\right\rfloor
                    + f(\left\lfloor\frac{N}{2}\right\rfloor)
                    + f(\left\lceil\frac{N}{2}\right\rceil) &
                    \mbox{otherwise}
                  \end{array} \right. \]

In order to obtain an approximate solution to this relation, we can
solve for $N$ being a power of $2$, i.e., $N = 2^n$.  In that case,
for $N>1$ we obtain:

\[ f(N) = \frac{N}{2} + 2f(\frac{N}{2}) \]

The details of the approximate resolution of this recursive equation
is given in the appendix.
This solution yields

\[ f(N) = \frac{N}{2}\mathsf{lb}~N + Nf(1) = \frac{N}{2}\mathsf{lb}~N\]

Including the traversal to compute the number of elements of the list,
we obtain:

\[ F(N) = \frac{N}{2}\mathsf{lb}~N + N = N(1 + \frac{1}{2}\mathsf{lb}~N)\]

which is clearly $O(N\mathsf{log}~N)$.  More importantly, for a list
with around $16$ million elements (which fills the default heap of most
implementations we have tested), we have $N \approx 2^{24}$ which
gives $F(N) \approx 13N$ which is probably unacceptably slow.

Let us now consider what happens when we are able to handle more than
a single element with the basic recursive technique, as shown in
\refSec{sec-more-stack}.  We denote the number of elements that the
basic recursive technique can handle by $K$, and again, in order to
simplify the analysis, we assume that both $N$ and $K$ are powers of
$2$, i.e., $N = 2^n$ $K = 2^k$, and also that $N \ge K$.  The
recursion relation now looks as follows:

\label{analyse2}
\[ f(N) = \left\{ \begin{array}{ll}
                    N-1 & \mbox{if $N \le K$} \\
                    \frac{N}{2} + 2f(\frac{N}{2}) &\mbox{otherwise}
                  \end{array} \right. \]

The resolution of this equation is given in the appendix (Part B).
It yields:

\[ f(N) \approx N(1 + \frac{1}{2}\mathsf{lb}~\frac{N}{K})\]

With the best portable version of our technique and a typical stack
being able to handle $K = 2^{14}$ we are now looking at a
performance for $N = 2^{24}$ of $F(N) \approx 6N$.  Comparing this
result to the technique of reversing the list, it is fair to say that
the overhead of allocating and subsequently garbage-collecting a
\texttt{cons} cell can very well be comparable to $6$ times the time
taken by the \texttt{cdr} operation.  In other words, the performance
of our portable version is already comparable to an implementation
based on first creating a reversed copy of the list and then
traversing that reversed copy.

Finally, instead of using more stack space for the base case, let us
analyze what happens if we divide the original list into more than two
parts.  For this analysis, let us assume that we divide the list into
$M$ equal parts, and that $M$ also is a power of $2$ so that $M =
2^m$.  We then obtain the following relation:

\label{analyse3}
\[ f(N) = \left\{ \begin{array}{ll}
                    0 & \mbox{if $N = 1$} \\
                    N - \frac{N}{M} + Mf(\frac{N}{M}) &\mbox{otherwise}
                  \end{array} \right. \]

The resolution of this equation is given in the appendix (Part C).
It yields:

\[ F(N) \approx N(1 + \frac{\mathsf{lb}~N}{\mathsf{lb}~M}) \]

While it may appear that we can get very good performance when $M$ is
chosen to be large, in practice, using large values of $M$ introduces
a different kind of overhead, namely large stack frames, making the
gain less than what the formula suggests.

\subsection{Implementation-specific solutions}

So far, we have explored techniques that can mostly be implemented in
portable \commonlisp{}.  In this section, we explore a variation on
our technique that requires access to the control stack of the
implementation.

Recall that at the lowest level of our technique, there is a recursive
function that is used for traversing the list when the number of
elements is small compared to the stack size.  At each invocation,
this function does very little work.

With direct access to the control stack, we can convert the recursive
function to an iterative function that pushes the elements of the list
on the control stack, and then processes them in reverse order.  This
technique has several advantages:

\begin{itemize}
\item A single word is used for each element, whereas the recursive
  function requires space for a return address, a frame pointer,
  saved registers, etc.  As a result, this technique can be used for
  lists with more elements than would be possible with the recursive
  technique, thereby further decreasing the number of times a list is
  traversed.
\item There is no function-call overhead involved.  The only
  processing that is needed for an element is to store it on the stack
  and then comparing it to the item.
\end{itemize}

We illustrate this technique in a notation similar to \commonlisp{}:

{\small
\begin{verbatim}
(defun low-level-reverse-count (item list length)
  (loop for rest = list then (cdr rest)
        repeat length
        do (push-on-stack (car rest)))
  (loop repeat length
        count (eq item (pop-from-stack))))
\end{verbatim}
}

We implemented this technique in \sbcl{}.  In order not to have to
recompile \sbcl{} with our additional function, we used the
implementation-specific foreign-function interface and wrote the
function using the language C.  Rather than pushing and popping the
control stack, we used the built-in C function \texttt{alloca} to
allocate a one-dimensional C array on the control stack to hold the
list elements.

In \sbcl{}, the default stack size is $2$MBytes, or around $250$k
words on a 64-bit processor.  We tested our technique using $100000$
words on the stack.  The result is that for a list with $10$ million
elements, our technique processes the list in reverse order as fast as
an ordinary loop from the beginning of the list.

This surprising result can be explained by a few factors:

\begin{itemize}
\item Presumably in order to speed up the functions \texttt{car} and
  \texttt{cdr}, SBCL uses the same tag for \texttt{cons} cells and for
  the symbol \texttt{nil}.  As a result, in order to traverse a list,
  SBCL must make \emph{two} tests for each element, namely one to
  check whether the putative list is something other than a list
  altogether, and another to check whether it is a \texttt{cons}
  cell.  When our technique traverses a list for which the number of
  elements is known, there is no need to make any additional tests,
  simply because when the length of the list is positive, the first
  element must be a \texttt{cons} cell.
\item The \sbcl{} compiler can not determine that the return value of
  \texttt{count} must always be a \texttt{fixnum}.%
  \footnote{On a byte-addressed processor where $n$ word-aligned bytes
    are needed to represent a \texttt{cons} cell, the number of
    elements in a list can be at most $N/n$ where $N$ is the maximum
    number of possible addresses.  In a system that uses at most
    $\mathsf{lb}~n$ tag bits for a fixnum, the value that
    \texttt{count} returns must be a fixnum.  While some systems might
    use $8$ tag bits, \sbcl{} on a $64$-bit platform uses a single tag
    bit for fixnums.  As a consequence, \texttt{count} must then
    return a fixnum.}
  When the function is implemented in C, this problem disappears.
\end{itemize}

If we put this technique in the perspective of the analyses in
\refSec{sec-analyses}, we can also see that the number of \texttt{cdr}
operations remains quite modest, even for lists with a very large
number of elements.

There are several variations on this implementation-specific
technique.  Some implementations might allocate a vector or a list
declared to be \texttt{dynamic-extent} on the stack, thus giving
essentially the same advantage as the version we implemented in C.
However, such a technique would still be implementation specific,
given that it is permitted for the compiler to ignore
\texttt{dynamic-extent} declarations.  In the case of \sbcl{}, using
such a declaration, we were able to obtain performance almost as good
as our C version.  However, as it turns out, \sbcl{} only allocates a
vector on the stack under certain circumstances thereby making this
technique impossible to apply in general.


\input{../../style/preamble}
\input{../../latex-math/basic-math}
\input{../../latex-math/basic-ml}
\input{../../latex-math/ml-gp}

\newcommand{\titlefigure}{figure_man/post-mean.png} % does not fit
\newcommand{\learninggoals}{
  \item \textcolor{blue}{XXX}
  \item \textcolor{blue}{XXX}
}

\title{Introduction to Machine Learning}
\date{}

\begin{document}

\lecturechapter{Gaussian Proccesses: Additional Material}
\lecture{Introduction to Machine Learning}

%http://www.gaussianprocess.org/gpml

\begin{vbframe}{Notation}
% We would like to model a function
%
% $$
% f: \mathcal{X} \to \Yspace
% $$
%
% where
%
% \begin{itemize}
% \item $\Xspace$ is a p-dimensional input space (here: $\Xspace = \R^n$)
% \item $\Yspace$ is the target space (usually $\Yspace = \R$ for regression and $\Yspace = \{0, 1\}$ for binary classification)
% \item $\bm{x} \in \mathcal{X}$ is called independent / predictor variable
% \item $y \in \mathcal{Y}$ is called dependent variable (target, label, output)
% \end{itemize}
%
% \framebreak
%

In this chapter

\begin{itemize}
\item $(\xv_*, y_*)$ denotes one single test observation, excluded from training
\item $\Xmat_* \in \R^{n_* \times p}$ contains a set of $n_*$ test observations and
\item $\yv_* \in \R^{n_* \times p}$ the corresponding outcomes, excluded from training.
\end{itemize}

% \framebreak

% In the context of Gaussian processes

% \begin{itemize}
% \item the function $m: \Xspace \to \R$ is called \textbf{mean function}. We define the \textbf{mean vector}

% \vspace*{-0.3cm}
% $$
% m(\Xmat):= \biggl(m\left(\bm{x}^{(1)}\right), m\left(\bm{x}^{(2)}\right), ..., m\left(\bm{x}^{(n)}\right)\biggr)^T
% $$
% \item  the bivariate, positive-definite function $k: \Xspace \times \Xspace \to \R$ is called \textbf{covariance function} or \textbf{kernel}; $k(\Xmat, \Xmat)$ denotes the $n\times n$ matrix that is obtained by plugging in all pairs $\bm{x}^{(i)}, \bm{x}^{(j)}$ and is called \textbf{kernel matrix} or \textbf{covariance matrix}

% $$
% k(\Xmat, \Xmat) := k(\bm{x}^{(i)}, \bm{x}^{(j)})_{i, j = 1, ..., n}
% $$
% \item We sometimes use the abbreviations $\bm{K} := k(\Xmat, \Xmat)$, $\bm{K}_* := k(\Xmat_*, \Xmat)$, $\bm{K}_{**} := k(\Xmat_*, \Xmat_*)$.

% \end{itemize}

\end{vbframe}

\section{Noisy Gaussian Processes}

\begin{vbframe}{Noisy Gaussian Process}

In the above equations we implicitly assumed that we had access to the true function value $\fx$. In many cases, we only have access to a noisy version thereof
$$
y = \fx + \eps.$$

Assuming additive i.i.d. Gaussian noise, the covariance function becomes

$$
\cov(y^{(i)}, y^{(j)}) = k(\bm{x}^{(i)}, \bm{x}^{(j)}) + \sigma_n^2 \delta_{ij}
$$

where $\delta_{ij} = 1$ if $i = j$. In matrix notation, this becomes

$$
\cov(\yv) = \Kmat + \sigma_n^2\id =: \Kmat_y.
$$

The $\sigma_n^2$ is also called \textbf{nugget}.

\end{vbframe}

\begin{vbframe}{GP vs. kernelized Ridge regression}

The predictive function is then

\begin{eqnarray*}
\bm{f}_* | \Xmat_*, \Xmat, \yv \sim \mathcal{N}(\bm{\bar f}_*, \cov(\bm{\bar f}_*)).
\end{eqnarray*}

with

\begin{itemize}
\item $\bm{\bar f}_* = \Kmat_{*}^{T} \Kmat_y^{-1}\yv$ and
\item $\cov(\bm{\bar f}_*) = \Kmat_{**}- \Kmat_{*}^{T}\Kmat_y^{-1}\Kmat_*$.
\end{itemize}

The predicted mean values at the training points $\bm{\bar f} = \bm{K}\Kmat_y^{-1}\bm{y}$ are a \textbf{linear combination} of the $\bm{y}$ values.

\lz

\textbf{Note:} Predicting the posterior mean corresponds exactly to the predictions obtained by kernelized Ridge regression. However, a GP (as a Bayesian model) gives us much more information, namely a posterior distribution, whilst kernelized Ridge regression does not.

\end{vbframe}

\section{Bayesian Linear Regression as a GP}

\begin{vbframe}{Bayesian linear regression as a GP}

One example for a Gaussian process is the Bayesian linear regression model covered earlier. For  $\thetab \sim \mathcal{N}(\bm{0}, \tau^2 \id)$, the joint distribution of any set of function values

$$
f(\xi) = \thetab^T \xi + \epsi
$$

is Gaussian.

\vspace*{0.3cm}

The corresponding mean function is $m(\bm{x}) = \bm{0}$ and the covariance function is

\vspace*{-0.5cm}

\begin{eqnarray*}
\cov(f(\bm{x}), f(\bm{x}^\prime)) &=& \E[f(\bm{x}) f(\bm{x}^\prime)] - \underbrace{\E[f(\bm{x})] \E[f(\bm{x}^\prime]}_{= 0} \\ &=& \E[(\thetab^T \bm{x} + \epsi)^T(\thetab^T \bm{x}^\prime + \epsi)] \\ &=&  \tau^2 \bm{x}^T\bm{x}^\prime + \sigma^2 =: k(\bm{x}, \bm{x}^\prime).
\end{eqnarray*}

% As we have just described, the predictive distribution assuming a Gaussian process Prior for one single test point $\bm{x}^*$  is normal with mean
%
% $$
% (\bm{x}^*)^T \bm{X}^T (\Xmat\Xmat^T + \id)^{-1} \yv.
% $$
%
% Remember that we derived also a normal predictive distribution for a Bayesian linear regression case - the predictive mean was
%
% $$
% \mu_{\text{post}} = (\bm{x}^*)^T(\Xmat^T\Xmat + \sigma^2 \id)^{-1}\Xmat^T\yv.
% $$
%
% Using the matrix identity $(\bm{AB} + \id)
% ^{-1}\Amat = \Amat(\bm{BA} + \id)^{-1}$^*$, it can be seen that the predictive distributions are identical.
%
% \vfill
% \begin{footnotesize}
% $^*$ Searl Set of Identities, see \emph{http://matrixcookbook.com], 3.2}
% \end{footnotesize}

\end{vbframe}

\begin{vbframe}{Feature Spaces and the Kernel Trick}

If one relaxes the linearity assumption by first projecting features into a higher dimensional feature space $\mathcal{Z}$ using a basis function $\phi: \Xspace \to \mathcal{Z}$, the corresponding covariance function is

$$
k(\bm{x}, \bm{x}^\prime) = \tau^2 \phi(\bm{x})^T\phi(\bm{x}^\prime) + \sigma^2.
$$

To get arbitrarily complicated functions, we would have to handle high-dimensional feature vectors $\phi(\bm{x})$.

\lz

Fortunately, all we need to know are the inner products $\phi(\bm{x})^T\phi(\bm{x}^\prime)$ - the feature vector itself never occurs in calculations.

\framebreak

If we can get the inner product directly \textbf{without} calculating the infinite feature vectors, we can infer an infinitely complicated model with a \textbf{finite amount} of computation. This idea is known as \textbf{kernel trick}.

\lz

 A Gaussian process can be defined by either

\begin{itemize}
\item deriving the covariance function explicitly via inner products of evaluations of basis functions or
\item choosing a positive definite kernel function (Mercer Kernel) directly, which  corresponds - according to Mercer's theorem - to taking inner products in some (possibly infinite) feature space
\end{itemize}

\end{vbframe}

\begin{vbframe}{Summary: Gaussian process regression}

\begin{itemize}
\item Gaussian process regression is equivalent to \textbf{kernelized} Bayesian linear regression
\item The covariance function describes the shape of the Gaussian process
\item With the right choice of covariance function, remarkably flexible models can be built
\item But: naive implementations of Gaussian process models scale poorly with large datasets as
\begin{itemize}
\item the kernel matrix has to be inverted / factorized, which is $\order(n^3)$,
\item computing the kernel matrix uses $\order(n^2)$ memory - running out of memory places a hard limit on problem sizes
\item generating predictions is $\order(n)$ for the mean, but $\order(n^2)$ for the variance.
\end{itemize}
(...so we need special tricks)
\end{itemize}

\end{vbframe}

\endlecture
\end{document}