\section{Deep Neural Net - RegularNet}
The regular neural network is composed exclusively of regular and strided convolutional layers. While this architecture works well for relatively shallow networks, it becomes increasingly more difficult to train as the network depth increases.

Regular networks are a combination of the two standard forms of neural networks, the fully connected neural network and the convolution neural network.

\subsection{Fully Connected Neural Networks (FCNN)}
A neural network consist of an input layer, some hidden layers and an output layer. The input layer is an image, and it is targeted for feature extraction and classification in the hidden layers and output layer. Every pixel in the image is connected to every neuron in the first hidden layer of the neural network as seen in figure \ref{fig:RegularFCNN}.
Each hidden layer is an array of neurons, with each neuron normally consisting of a weight, some activation function and a regulation function. Each pixel value is weighted, activated in an activation function and regularized. Most often ReLU or Leaky ReLU are used as activation function to zero out negative values. The activation function provides a non-linear relationship within the data points to provide better feature extractions. Each neuron in the second hidden layer is supplied with the output of all the neurons in the previous layer and the procedure is repeated until the last hidden layer. The output layer consist of a loss function, which is often either a Softmax loss function or a support vector machine loss function. The amount of loss functions in the output layer is equal to the amount of classification categories the image can be classified as.
\\
An image is forwarded through the neural network, and the loss functions provides the misclassification percentage of the image. This error, or loss in accuracy, is send back through the neural network and a gradient for each neuron is found. This process is repeated for an amount of images, batch size, and for each backward propagation the gradient is saved. After a batch of images all the gradients saved for an individual neuron are average and from this value the neural network response to the back propagation. The weights are updated depending on the averaged gradients and this procedure can be done in different ways. The most modern update method is called ADAM, and it is trying to reduce the loss result by changing the weights regarding to the gradients. ADAM will decide the amount of change the weights must have in relation to the gradients, while the regularization step decides the relationship of the change between the weights. ADAM defines the maximum change to a single weight, while the regularization defines the distribution of change over all weights related to the maximum defined change.

\myFigure{RegularNet_FCNN.png}{Fully Connected Neural Network Architecture}{fig:RegularFCNN}{0.5}
\FloatBarrier

\subsection{Convolutional Neural Networks (CNN)}

The main difference between the FCNN and CNN is the FCNN is providing every input pixel to every neuron between each layer, while the CNN is only connecting the neurons to regions of interest of the image, also known as receptive fields. Each receptive field is dotted together with a feature map which provides a single pixel output.

The input layer is an image, while the next layer consist of weighted feature maps, an activation function and a regulation function. Each feature map consist of three spatial dimensions. One spatial dimension of the feature map is convolved with one spatial dimension of the image. The result of dot products for the image pixels, at one receptive field, with the feature map provides a value for the receptive field. Convolving the input image of, M rows and N columns $MxNx3$ with X feature maps of 3x3x3 provides X new images with a dimension determine by the striding distance for the convolution as seen in figure \ref{fig:RegularStrinding}.

\myFigure{RegularNET_Strinding.png}{Convolutional Neural Network feature map striding example}{fig:RegularStrinding}{0.5}
\FloatBarrier

Pooling layers are placed between layers at convenient places, to minimize variable count and training time for the neural network. This is typical done by increasing filter strides or by max pooling. Max pooling is done by choosing the highest pixel value in a spatial dimension of the image and compressing the images to only the chosen max pooling values, as seen in figure \ref{fig:RegularCVNN}. It will not always be possible to have the desired stride unit distance in the image, so an extra layer of zeros are padded on to the image to make sure the image is filtered with the correct feature map size and stride unit distance. One zero padding layer and an 3x3 feature map will return an image with the same size as the input image.

\myFigure{RegularNet_CVNN.png}{Convolutional Neural Network Architecture}{fig:RegularCVNN}{0.5}
\FloatBarrier


\section{Geometric interpretation}

\begin{outcome}
  \begin{enumerate}
  \item View complex numbers as points in the plane.
  \item Understand the geometric meaning of addition, subtraction,
    multiplication, and the complex conjugate.
  \item Understand the geometric meaning of the magnitude and argument
    of a complex number.
  \end{enumerate}
\end{outcome}

Just as a real number can be considered as a point on the line, a
complex number $z = a + bi$ can be considered as a point $(a,b)$ in
the plane whose $x$ coordinate is $a$ and whose $y$ coordinate is
$b$. For example, in the following picture, the complex number
$z = 3+2i$ can be represented as the point in the plane with
coordinates $(3,2)$%
\index{complex number!geometric interpretation}.
\begin{equation*}
  \begin{tikzpicture}
    \draw[help lines, dotted, fill=red!10] (0,0) -- (3,0) -- (3,2) -- cycle;
    \draw[help lines, dotted] (0,2) -- (3,2);
    \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (1.2,0) arc (0:33.69:1.2) -- cycle;
    \node at (16.845:9mm){$\theta$};
    \draw[->](-1.5,0) -- (3.5,0);
    \draw[->](0,0) -- (0,2.5);
    \draw(0,1) -- +(-0.2,0) node[left] {$i$};
    \draw(0,2) -- +(-0.2,0) node[left] {$2i$};
    \draw(-1,0) -- +(0,-0.2) node[below] {$-1$};
    \draw(0,0) -- +(0,-0.2) node[below] {$0$};
    \draw(1,0) -- +(0,-0.2) node[below] {$1$};
    \draw(2,0) -- +(0,-0.2) node[below] {$2$};
    \draw(3,0) -- +(0,-0.2) node[below] {$3$};
    \draw[thick, blue] (0,0) -- node[above, black] {$r$} (3,2);
    \draw[fill, black] (3,2) circle [radius=1.8pt] node [right] {$z = 3+2i$};
  \end{tikzpicture}
\end{equation*}
The \textbf{magnitude}%
\index{complex number!magnitude}%
\index{magnitude!of a complex number} $r=\abs{z}$ of a complex number
is its distance from the origin. We also define the \textbf{argument}
of $z$ to be the angle $\theta$ between the $x$-axis and the line from
the origin to $z$, counted positively in the counterclockwise
direction. The magnitude $r$ and argument $\theta$ are shown in the
above picture.

Addition of complex numbers is like vector addition. The effect of
multiplying two complex numbers is to multiply their magnitudes and
add their arguments. For example, the following picture illustrates
the multiplication $(3+2i)(1+i)=1+5i$.
\begin{equation*}
  \begin{tikzpicture}[scale=0.8]
    \begin{scope}
      \draw[help lines, dotted, fill=red!10] (0,0) -- (3,0) -- (3,2) -- cycle;
      \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (0.8,0) arc (0:33.69:0.8) -- cycle;
      \draw[->](-1.5,0) -- (3.5,0);
      \draw[->](0,0) -- (0,2.5);
      \draw(0,1) -- +(-0.2,0) node[left] {$i$};
      \draw(0,2) -- +(-0.2,0) node[left] {$2i$};
      \draw(-1,0) -- +(0,-0.2) node[below] {$-1$};
      \draw(0,0) -- +(0,-0.2) node[below] {$0$};
      \draw(1,0) -- +(0,-0.2) node[below] {$1$};
      \draw(2,0) -- +(0,-0.2) node[below] {$2$};
      \draw(3,0) -- +(0,-0.2) node[below] {$3$};
      \draw[thick, blue] (0,0) -- (3,2);
      \draw[fill, black] (3,2) circle [radius=2.25pt] node [right] {$3+2i$};
      \draw (1,-2) node {The number $3+2i$};
    \end{scope}
    \begin{scope}[xshift=7cm]
      \draw[help lines, dotted, fill=yellow!30] (0,0) -- (1,0) -- (1,1) -- cycle;
      \filldraw[fill=blue!10,draw=blue!50!black] (0,0) -- (0.5,0) arc (0:45:0.5) -- cycle;
      \draw[->](-1.5,0) -- (3.5,0);
      \draw[->](0,0) -- (0,2.5);
      \draw(0,1) -- +(-0.2,0) node[left] {$i$};
      \draw(0,2) -- +(-0.2,0) node[left] {$2i$};
      \draw(-1,0) -- +(0,-0.2) node[below] {$-1$};
      \draw(0,0) -- +(0,-0.2) node[below] {$0$};
      \draw(1,0) -- +(0,-0.2) node[below] {$1$};
      \draw(2,0) -- +(0,-0.2) node[below] {$2$};
      \draw(3,0) -- +(0,-0.2) node[below] {$3$};
      \draw[thick, blue] (0,0) -- (1,1);
      \draw[fill, black] (1,1) circle [radius=2.25pt] node [right] {$1+i$};
      \draw (1,-2) node {The number $1+i$};
    \end{scope}
    \begin{scope}[xshift=14cm]
      \draw[help lines, dotted, fill=red!10] (0,0) -- (3,0) -- (3,2) -- cycle;
      \draw[help lines, dotted, fill=yellow!30] (0,0) -- (3,2) -- (1,5) -- cycle;
      \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (0.8,0) arc (0:33.69:0.8) -- cycle;
      \filldraw[fill=blue!10,draw=blue!50!black] (0,0) -- (33.69:0.8) arc (33:78.69:0.8) -- cycle;
      \draw[->](-1.5,0) -- (3.5,0);
      \draw[->](0,0) -- (0,5.5);
      \draw(0,1) -- +(-0.2,0) node[left] {$i$};
      \draw(0,2) -- +(-0.2,0) node[left] {$2i$};
      \draw(0,3) -- +(-0.2,0) node[left] {$3i$};
      \draw(0,4) -- +(-0.2,0) node[left] {$4i$};
      \draw(0,5) -- +(-0.2,0) node[left] {$5i$};
      \draw(-1,0) -- +(0,-0.2) node[below] {$-1$};
      \draw(0,0) -- +(0,-0.2) node[below] {$0$};
      \draw(1,0) -- +(0,-0.2) node[below] {$1$};
      \draw(2,0) -- +(0,-0.2) node[below] {$2$};
      \draw(3,0) -- +(0,-0.2) node[below] {$3$};
      \draw[thick, blue] (0,0) -- (3,2);
      \draw[fill, black] (3,2) circle [radius=2.25pt];
      \draw[thick, blue] (0,0) -- (1,5);
      \draw[fill, black] (1,5) circle [radius=2.25pt] node [right] {$1+5i$};
      \draw (1,-2) node {The product $(3+2i)(1+i)=1+5i$};
    \end{scope}
  \end{tikzpicture}
\end{equation*}
To take the multiplicative inverse of a complex number, we take the
reciprocal of the magnitude and negate the argument.
\begin{equation*}
  \begin{tikzpicture}
    \draw[help lines, dotted] (30:4.5) -- (30:4.5 |- 0,0);
    \draw[help lines, dotted] (-30:2) -- (-30:2 |- 0,0);
    \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (1.2,0) arc (0:30:1.2) -- cycle;
    \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (1.2,0) arc (0:-30:1.2) -- cycle;
    \node at (15:9mm){$~\theta$};
    \node at (-15:9mm){$-\theta~$};
    \draw[->](-1.5,0) -- (5.5,0);
    \draw[->](0,0) -- (0,3.5);
    \draw(0,3) -- +(-0.2,0) node[left] {$i$};
    \draw(3,0) -- +(0,-0.2) node[below] {$1$};
    \draw[thick, blue] (0,0) -- node[above, black] {$r$} (30:4.5);
    \draw[fill, black] (30:4.5) circle [radius=1.8pt] node [right] {$z$};
    \draw[thick, blue] (0,0) -- node[below, black] {$\frac{1}{r}$} (-30:2);
    \draw[fill, black] (-30:2) circle [radius=1.8pt] node [right] {$z^{-1}$};
  \end{tikzpicture}
\end{equation*}
The effect of taking the complex conjugate is to reflect the given
complex number about the $x$ axis (or equivalently, keep the magnitude
unchanged and negate the argument).
\begin{equation*}
  \begin{tikzpicture}
    \draw[help lines, dotted] (30:4.5) -- (30:4.5 |- 0,0);
    \draw[help lines, dotted] (-30:4.5) -- (-30:4.5 |- 0,0);
    \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (1.2,0) arc (0:30:1.2) -- cycle;
    \filldraw[fill=green!20,draw=green!50!black] (0,0) -- (1.2,0) arc (0:-30:1.2) -- cycle;
    \node at (15:9mm){$~\theta$};
    \node at (-15:9mm){$-\theta~$};
    \draw[->](-1.5,0) -- (5.5,0);
    \draw[->](0,0) -- (0,3.5);
    \draw(0,3) -- +(-0.2,0) node[left] {$i$};
    \draw(3,0) -- +(0,-0.2) node[below] {$1$};
    \draw[thick, blue] (0,0) -- node[above, black] {$r$} (30:4.5);
    \draw[fill, black] (30:4.5) circle [radius=1.8pt] node [right] {$z$};
    \draw[thick, blue] (0,0) -- node[below, black] {$r$} (-30:4.5);
    \draw[fill, black] (-30:4.5) circle [radius=1.8pt] node [right] {$\bar{z}$};
  \end{tikzpicture}
\end{equation*}


\subsection{Stratified Bivariate Statistics}

\noindent{\bf Description}
\smallskip

The {\tt stratstats.dml} script computes common bivariate statistics, such
as correlation, slope, and their p-value, in parallel for many pairs of input
variables in the presence of a confounding categorical variable.  The values
of this confounding variable group the records into strata (subpopulations),
in which all bivariate pairs are assumed free of confounding.  The script
uses the same data model as in one-way analysis of covariance (ANCOVA), with
strata representing population samples.  It also outputs univariate stratified
and bivariate unstratified statistics.

\begin{table}[t]\hfil
\begin{tabular}{|l|ll|ll|ll||ll|}
\hline
Month of the year & \multicolumn{2}{l|}{October} & \multicolumn{2}{l|}{November} &
    \multicolumn{2}{l||}{December} & \multicolumn{2}{c|}{Oct$\,$--$\,$Dec} \\
Customers, millions    & 0.6 & 1.4 & 1.4 & 0.6 & 3.0 & 1.0 & 5.0 & 3.0 \\
\hline
Promotion (0 or 1)     & 0   & 1   & 0   & 1   & 0   & 1   & 0   & 1   \\
Avg.\ sales per 1000   & 0.4 & 0.5 & 0.9 & 1.0 & 2.5 & 2.6 & 1.8 & 1.3 \\
\hline
\end{tabular}\hfil
\caption{Stratification example: the effect of the promotion on average sales
becomes reversed and amplified (from $+0.1$ to $-0.5$) if we ignore the months.}
\label{table:stratexample}
\end{table}

To see how data stratification mitigates confounding, consider an (artificial)
example in Table~\ref{table:stratexample}.  A highly seasonal retail item
was marketed with and without a promotion over the final 3~months of the year.
In each month the sale was more likely with the promotion than without it.
But during the peak holiday season, when shoppers came in greater numbers and
bought the item more often, the promotion was less frequently used.  As a result,
if the 4-th quarter data is pooled together, the promotion's effect becomes
reversed and magnified.  Stratifying by month restores the positive correlation.

The script computes its statistics in parallel over all possible pairs from two
specified sets of covariates.  The 1-st covariate is a column in input matrix~$X$
and the 2-nd covariate is a column in input matrix~$Y$; matrices $X$ and~$Y$ may
be the same or different.  The columns of interest are given by their index numbers
in special matrices.  The stratum column, specified in its own matrix, is the same
for all covariate pairs.

Both covariates in each pair must be numerical, with the 2-nd covariate normally
distributed given the 1-st covariate (see~Details).  Missing covariate values or
strata are represented by~``NaN''.  Records with NaN's are selectively omitted
wherever their NaN's are material to the output statistic.

\smallskip
\pagebreak[3]

\noindent{\bf Usage}
\smallskip

{\hangindent=\parindent\noindent\it%
{\tt{}-f }path/\/{\tt{}stratstats.dml}
{\tt{} -nvargs}
{\tt{} X=}path/file
{\tt{} Xcid=}path/file
{\tt{} Y=}path/file
{\tt{} Ycid=}path/file
{\tt{} S=}path/file
{\tt{} Scid=}int
{\tt{} O=}path/file
{\tt{} fmt=}format

}

\smallskip
\noindent{\bf Arguments}
\begin{Description}
\item[{\tt X}:]
Location (on HDFS) to read matrix $X$ whose columns we want to use as
the 1-st covariate (i.e.~as the feature variable)
\item[{\tt Xcid}:] (default:\mbox{ }{\tt " "})
Location to read the single-row matrix that lists all index numbers
of the $X$-columns used as the 1-st covariate; the default value means
``use all $X$-columns''
\item[{\tt Y}:] (default:\mbox{ }{\tt " "})
Location to read matrix $Y$ whose columns we want to use as the 2-nd
covariate (i.e.~as the response variable); the default value means
``use $X$ in place of~$Y$''
\item[{\tt Ycid}:] (default:\mbox{ }{\tt " "})
Location to read the single-row matrix that lists all index numbers
of the $Y$-columns used as the 2-nd covariate; the default value means
``use all $Y$-columns''
\item[{\tt S}:] (default:\mbox{ }{\tt " "})
Location to read matrix $S$ that has the stratum column.
Note: the stratum column must contain small positive integers; all fractional
values are rounded; stratum IDs of value ${\leq}\,0$ or NaN are treated as
missing.  The default value for {\tt S} means ``use $X$ in place of~$S$''
\item[{\tt Scid}:] (default:\mbox{ }{\tt 1})
The index number of the stratum column in~$S$
\item[{\tt O}:]
Location to store the output matrix defined in Table~\ref{table:stratoutput}
\item[{\tt fmt}:] (default:\mbox{ }{\tt "text"})
Matrix file output format, such as {\tt text}, {\tt mm}, or {\tt csv};
see read/write functions in SystemML Language Reference for details.
\end{Description}

\begin{table}[t]\small\hfil
\begin{tabular}{|rcl|rcl|}
\hline
& Col.\# & Meaning & & Col.\# & Meaning \\
\hline
\multirow{9}{*}{\begin{sideways}1-st covariate\end{sideways}}\hspace{-1em}
& 01     & $X$-column number                &
\multirow{9}{*}{\begin{sideways}2-nd covariate\end{sideways}}\hspace{-1em}
& 11     & $Y$-column number                \\
& 02     & presence count for $x$           &
& 12     & presence count for $y$           \\
& 03     & global mean $(x)$                &
& 13     & global mean $(y)$                \\
& 04     & global std.\ dev. $(x)$          &
& 14     & global std.\ dev. $(y)$          \\
& 05     & stratified std.\ dev. $(x)$      &
& 15     & stratified std.\ dev. $(y)$      \\
& 06     & $R^2$ for $x \sim {}$strata      &
& 16     & $R^2$ for $y \sim {}$strata      \\
& 07     & adjusted $R^2$ for $x \sim {}$strata      &
& 17     & adjusted $R^2$ for $y \sim {}$strata      \\
& 08     & p-value, $x \sim {}$strata       &
& 18     & p-value, $y \sim {}$strata       \\
& 09--10 & reserved                         &
& 19--20 & reserved                         \\
\hline
\multirow{9}{*}{\begin{sideways}$y\sim x$, NO strata\end{sideways}}\hspace{-1.15em}
& 21     & presence count $(x, y)$          &
\multirow{10}{*}{\begin{sideways}$y\sim x$ AND strata$\!\!\!\!$\end{sideways}}\hspace{-1.15em}
& 31     & presence count $(x, y, s)$       \\
& 22     & regression slope                 &
& 32     & regression slope                 \\
& 23     & regres.\ slope std.\ dev.        &
& 33     & regres.\ slope std.\ dev.        \\
& 24     & correlation${} = \pm\sqrt{R^2}$  &
& 34     & correlation${} = \pm\sqrt{R^2}$  \\
& 25     & residual std.\ dev.              &
& 35     & residual std.\ dev.              \\
& 26     & $R^2$ in $y$ due to $x$          &
& 36     & $R^2$ in $y$ due to $x$          \\
& 27     & adjusted $R^2$ in $y$ due to $x$ &
& 37     & adjusted $R^2$ in $y$ due to $x$ \\
& 28     & p-value for ``slope = 0''        &
& 38     & p-value for ``slope = 0''        \\
& 29     & reserved                         &
& 39     & \# strata with ${\geq}\,2$ count \\
& 30     & reserved                         &
& 40     & reserved                         \\
\hline
\end{tabular}\hfil
\caption{The {\tt stratstats.dml} output matrix has one row per each distinct
pair of 1-st and 2-nd covariates, and 40 columns with the statistics described
here.}
\label{table:stratoutput}
\end{table}

\noindent{\bf Details}
\smallskip

Suppose we have $n$ records of format $(i, x, y)$, where $i\in\{1,\ldots, k\}$ is
a stratum number and $(x, y)$ are two numerical covariates.  We want to analyze
conditional linear relationship between $y$ and $x$ conditioned by~$i$.
Note that $x$, but not~$y$, may represent a categorical variable if we assign a
numerical value to each category, for example 0 and 1 for two categories.

We assume a linear regression model for~$y$:
\begin{equation}
y_{i,j} \,=\, \alpha_i + \beta x_{i,j} + \eps_{i,j}\,, \quad\textrm{where}\,\,\,\,
\eps_{i,j} \sim \Normal(0, \sigma^2)
\label{eqn:stratlinmodel}
\end{equation}
Here $i = 1\ldots k$ is a stratum number and $j = 1\ldots n_i$ is a record number
in stratum~$i$; by $n_i$ we denote the number of records available in stratum~$i$.
The noise term~$\eps_{i,j}$ is assumed to have the same variance in all strata.
When $n_i\,{>}\,0$, we can estimate the means of $x_{i, j}$ and $y_{i, j}$ in
stratum~$i$ as
\begin{equation*}
\bar{x}_i \,= \Big(\sum\nolimits_{j=1}^{n_i} \,x_{i, j}\Big) / n_i\,;\quad
\bar{y}_i \,= \Big(\sum\nolimits_{j=1}^{n_i} \,y_{i, j}\Big) / n_i
\end{equation*}
If $\beta$ is known, the best estimate for $\alpha_i$ is $\bar{y}_i - \beta \bar{x}_i$,
which gives the prediction error sum-of-squares of
\begin{equation}
\sum\nolimits_{i=1}^k \sum\nolimits_{j=1}^{n_i} \big(y_{i,j} - \beta x_{i,j} - (\bar{y}_i - \beta \bar{x}_i)\big)^2
\,\,=\,\, \beta^{2\,}V_x \,-\, 2\beta \,V_{x,y} \,+\, V_y
\label{eqn:stratsumsq}
\end{equation}
where $V_x$, $V_y$, and $V_{x, y}$ are, correspondingly, the ``stratified'' sample
estimates of variance $\Var(x)$ and $\Var(y)$ and covariance $\Cov(x,y)$ computed as
\begin{align*}
V_x     \,&=\, \sum\nolimits_{i=1}^k \sum\nolimits_{j=1}^{n_i} \big(x_{i,j} - \bar{x}_i\big)^2; \quad
V_y     \,=\, \sum\nolimits_{i=1}^k \sum\nolimits_{j=1}^{n_i} \big(y_{i,j} - \bar{y}_i\big)^2;\\
V_{x,y} \,&=\, \sum\nolimits_{i=1}^k \sum\nolimits_{j=1}^{n_i} \big(x_{i,j} - \bar{x}_i\big)\big(y_{i,j} - \bar{y}_i\big)
\end{align*}
They are stratified because we compute the sample (co-)variances in each stratum~$i$
separately, then combine by summation.  The stratified estimates for $\Var(X)$ and $\Var(Y)$
tend to be smaller than the non-stratified ones (with the global mean instead of $\bar{x}_i$
and~$\bar{y}_i$) since $\bar{x}_i$ and $\bar{y}_i$ fit closer to $x_{i,j}$ and $y_{i,j}$
than the global means.  The stratified variance estimates the uncertainty in $x_{i,j}$
and~$y_{i,j}$ given their stratum~$i$.

Minimizing over~$\beta$ the error sum-of-squares~(\ref{eqn:stratsumsq})
gives us the regression slope estimate \mbox{$\hat{\beta} = V_{x,y} / V_x$},
with~(\ref{eqn:stratsumsq}) becoming the residual sum-of-squares~(RSS):
\begin{equation*}
\mathrm{RSS} \,\,=\, \,
\sum\nolimits_{i=1}^k \sum\nolimits_{j=1}^{n_i} \big(y_{i,j} -
\hat{\beta} x_{i,j} - (\bar{y}_i - \hat{\beta} \bar{x}_i)\big)^2
\,\,=\,\,  V_y \,\big(1 \,-\, V_{x,y}^2 / (V_x V_y)\big)
\end{equation*}
The quantity $\hat{R}^2 = V_{x,y}^2 / (V_x V_y)$, called \emph{$R$-squared}, estimates the fraction
of stratified variance in~$y_{i,j}$ explained by covariate $x_{i, j}$ in the linear
regression model~(\ref{eqn:stratlinmodel}).  We define \emph{stratified correlation} as the
square root of~$\hat{R}^2$ taken with the sign of~$V_{x,y}$.  We also use RSS to estimate
the residual standard deviation $\sigma$ in~(\ref{eqn:stratlinmodel}) that models the prediction error
of $y_{i,j}$ given $x_{i,j}$ and the stratum:
\begin{equation*}
\hat{\beta}\, =\, \frac{V_{x,y}}{V_x}; \,\,\,\, \hat{R} \,=\, \frac{V_{x,y}}{\sqrt{V_x V_y}};
\,\,\,\, \hat{R}^2 \,=\, \frac{V_{x,y}^2}{V_x V_y};
\,\,\,\, \hat{\sigma} \,=\, \sqrt{\frac{\mathrm{RSS}}{n - k - 1}}\,\,\,\,
\Big(n = \sum_{i=1}^k n_i\Big)
\end{equation*}

The $t$-test and the $F$-test for the null-hypothesis of ``$\beta = 0$'' are
obtained by considering the effect of $\hat{\beta}$ on the residual sum-of-squares,
measured by the decrease from $V_y$ to~RSS.
The $F$-statistic is the ratio of the ``explained'' sum-of-squares
to the residual sum-of-squares, divided by their corresponding degrees of freedom.
There are $n\,{-}\,k$ degrees of freedom for~$V_y$, parameter $\beta$ reduces that
to $n\,{-}\,k\,{-}\,1$ for~RSS, and their difference $V_y - {}$RSS has just 1 degree
of freedom:
\begin{equation*}
F \,=\, \frac{(V_y - \mathrm{RSS})/1}{\mathrm{RSS}/(n\,{-}\,k\,{-}\,1)}
\,=\, \frac{\hat{R}^2\,(n\,{-}\,k\,{-}\,1)}{1-\hat{R}^2}; \quad
t \,=\, \hat{R}\, \sqrt{\frac{n\,{-}\,k\,{-}\,1}{1-\hat{R}^2}}.
\end{equation*}
The $t$-statistic is simply the square root of the $F$-statistic with the appropriate
choice of sign.  If the null hypothesis and the linear model are both true, the $t$-statistic
has Student $t$-distribution with $n\,{-}\,k\,{-}\,1$ degrees of freedom.  We can
also compute it if we divide $\hat{\beta}$ by its estimated standard deviation:
\begin{equation*}
\stdev(\hat{\beta})_{\mathrm{est}} \,=\, \hat{\sigma}\,/\sqrt{V_x} \quad\Longrightarrow\quad
t \,=\, \hat{R}\sqrt{V_y} \,/\, \hat{\sigma} \,=\, \beta \,/\, \stdev(\hat{\beta})_{\mathrm{est}}
\end{equation*}
The standard deviation estimate for~$\beta$ is included in {\tt stratstats.dml} output.

\smallskip
\noindent{\bf Returns}
\smallskip

The output matrix format is defined in Table~\ref{table:stratoutput}.

\smallskip
\noindent{\bf Examples}
\smallskip

{\hangindent=\parindent\noindent\tt
\hml -f stratstats.dml -nvargs X=/user/biadmin/X.mtx Xcid=/user/biadmin/Xcid.mtx
  Y=/user/biadmin/Y.mtx Ycid=/user/biadmin/Ycid.mtx S=/user/biadmin/S.mtx Scid=2
  O=/user/biadmin/Out.mtx fmt=csv

}
{\hangindent=\parindent\noindent\tt
\hml -f stratstats.dml -nvargs X=/user/biadmin/Data.mtx Xcid=/user/biadmin/Xcid.mtx
  Ycid=/user/biadmin/Ycid.mtx Scid=7 O=/user/biadmin/Out.mtx

}

%\smallskip
%\noindent{\bf See Also}
%\smallskip
%
%For non-stratified bivariate statistics with a wider variety of input data types
%and statistical tests, see \ldots.  For general linear regression, see
%{\tt LinearRegDS.dml} and {\tt LinearRegCG.dml}.  For logistic regression, appropriate
%when the response variable is categorical, see {\tt MultiLogReg.dml}.


\documentclass{article}
\usepackage{amsmath, tikz, enumerate, sfmath, bm, multicol, tcolorbox}
\renewcommand{\familydefault}{\sfdefault}
\usepackage[top = 0.5in, bottom = 0.5in, left = 1in, right = 1in]{geometry}
\pagestyle{empty}
\raggedright
\tikzset{>=stealth}
\usetikzlibrary{calc}
\begin{document}
\section*{Matrix Multiplication}

\begin{tcolorbox}[colframe=orange!70!white, coltitle=black, title=\textbf{Summary}]
\begin{enumerate}
    \item Matrix multiplication creates new basis vectors $\hat{\imath}$ and $\hat{\jmath}$.
    \item An $m \times n$ matrix can be multiplied by an $n \times r$ matrix to make an $m \times r$ matrix.
    \item Matrix multiplication transforms the coordinate plane itself as a series of compositions of functions.
\end{enumerate}
\end{tcolorbox}

\subsection*{Matrix Times a Vector}

Matrix multiplication does not work the way you might think. For instance, if
\[ A = \begin{bmatrix} 3 & -2 \\ 1 & 0 \end{bmatrix} \quad \text{and} \quad B = \begin{bmatrix} -1 & 4 \\ 5 & 6 \end{bmatrix}
\]
\vspace{1in}

Then $AB$ is not $\begin{bmatrix} -3 & -8 \\ 5 & 0 \end{bmatrix}$. In other words, we don't multiply corresponding elements.	\vspace{1in}

In order to even multiply a matrix by a vector or another matrix, the number of columns in the first matrix \emph{must} equal the number of rows (for the vector or matrix) in the second.	\vspace{1in}

\newpage

Suppose we have the basis vector $\hat{\imath} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$. What happened to $\hat{\imath}$ when we multiply it by the matrix $\begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}$?
\bigskip

\[
\begin{bmatrix}
2 & 0 \\ 0 & 2
\end{bmatrix}
\begin{bmatrix}
1 \\ 0
\end{bmatrix}
\]
\vfill

\begin{center}
\begin{tikzpicture}[scale=0.75]
\draw[<->] (-5.5,0) -- (5.5,0) node [right] {$x$};
\draw[<->] (0,-5.5) -- (0,5.5) node [above] {$y$};
\draw[gray!70] (-5,-5) grid (5,5);
\end{tikzpicture}
\end{center}

\vfill

\newpage

{\color{red}\textbf{Example 1.}} What are the coordinates of $\hat{\jmath} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$ when you multiply it by
$
\begin{bmatrix}
3 & 0 \\
0 & 3
\end{bmatrix}
$
\bigskip

\[
\begin{bmatrix}
3 & 0 \\
0 & 3
\end{bmatrix}
\begin{bmatrix} 0 \\ 1 \end{bmatrix}
\]

\vfill

\begin{center}
\begin{tikzpicture}[scale=0.6]
\draw[<->] (-5.5,0) -- (5.5,0) node [right] {$x$};
\draw[<->] (0,-5.5) -- (0,5.5) node [above] {$y$};
\draw[gray!70] (-5,-5) grid (5,5);
\end{tikzpicture}
\end{center}

\vfill

\newpage

Suppose we have the vector $\vec{v} = \begin{bmatrix} 2 \\ 1 \end{bmatrix}$. This can be written using the basis vectors $\hat{\imath}$ and $\hat{\jmath}$ as $\vec{v} = 2\hat{\imath} + 1\hat{\jmath}$.
\begin{center}
\begin{tikzpicture}[scale=0.6]
\draw[<->] (-5.5,0) -- (5.5,0) node [right] {$x$};
\draw[<->] (0,-5.5) -- (0,5.5) node [above] {$y$};
\draw[gray!70] (-5,-5) grid (5,5);
\draw[color=red, very thick, ->] (0,0) -- (2,1) node [above right] {$\vec{v}$};
\end{tikzpicture}
\end{center}

What happens when we multiply $\vec{v}$ by the matrix
$
\begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}
$
\bigskip

\[
\begin{bmatrix} 2 & 0 \\ 0 & 2 \end{bmatrix}
\begin{bmatrix} 2 \\ 1  \end{bmatrix}
\]

\vfill

\begin{center}
\begin{tikzpicture}[scale=0.6]
\draw[<->] (-5.5,0) -- (5.5,0) node [right] {$x$};
\draw[<->] (0,-5.5) -- (0,5.5) node [above] {$y$};
\draw[gray!70] (-5,-5) grid (5,5);
\draw[color=red, very thick, ->] (0,0) -- (2,1) node [above right] {$\vec{v}$};
\end{tikzpicture}
\end{center}

\newpage

{\color{red}\textbf{Example 2.}} Find the product and graph the result.
\[
\begin{bmatrix}
3 & -2 \\
1 & 1
\end{bmatrix}
\begin{bmatrix}
2 \\ 1
\end{bmatrix}
\]

\vfill

\begin{center}
\begin{tikzpicture}[scale=0.7]
\draw[<->] (-6.5,0) -- (6.5,0) node [right] {$x$};
\draw[<->] (0,-6.5) -- (0,6.5) node [above] {$y$};
\draw[gray!70] (-6,-6) grid (6,6);
\end{tikzpicture}
\end{center}

\vspace{1in}

\newpage

\subsection*{Matrix Times a Matrix}

We can look at multiplying matrices as where $\hat{\imath}$ and $\hat{\jmath}$ end up in the coordinate plane. \newline\\

{\color{red}\textbf{Example 3.}} Explain the following problem visually
\[
\begin{bmatrix}
0 & -1 \\
1 & 0
\end{bmatrix}
\begin{bmatrix}
1 & 0 \\
0 & 1
\end{bmatrix}
\]

\vfill

\begin{center}
\begin{tikzpicture}[scale=0.8]
\draw [gray!70] (-5,-5) grid (5,5);
\draw[<->] (-5.5,0) -- (5.5,0) node [right] {$x$};
\draw[<->] (0,-5.5) -- (0,5.5) node [above] {$y$};
\end{tikzpicture}
\end{center}

\vspace{1in}

\newpage

{\color{red}\textbf{Example 4.}} Given $A = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$, determine the effect of each product.

\begin{enumerate}[(a)]
	\item $\begin{bmatrix} 0 & -1 \\ 1 & 0 \end{bmatrix}  \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$.	\newline\\
	\begin{flushright}
	\begin{tikzpicture}
	\draw[gray!70] (-3,-3) grid (3,3);
	\draw[<->] (-3.5,0) -- (3.5,0) node [right] {$x$};
	\draw[<->] (0,-3.5) -- (0,3.5) node [above] {$y$};
	\end{tikzpicture}
	\end{flushright}
    \vfill

	\item $\begin{bmatrix} 1 & 0 \\ 1 & 1
 \end{bmatrix}  \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}$  \label{ex4b} \newline\\

    \begin{flushright}
	\begin{tikzpicture}
	\draw[gray!70] (-3,-3) grid (3,3);
	\draw[<->] (-3.5,0) -- (3.5,0) node [right] {$x$};
	\draw[<->] (0,-3.5) -- (0,3.5) node [above] {$y$};
	\end{tikzpicture}
	\end{flushright}
\end{enumerate}

\newpage

In general, we can think of matrix multiplication like a composition of functions. \newline\\

Each multiplication leads to a new transformation of coordinates $\hat{\imath}$ and $\hat{\jmath}$:

\begin{itemize}
    \item Scaling $\hat{\imath}$ and/or $\hat{\jmath}$
    \item Rotating $\hat{\imath}$ and/or $\hat{\jmath}$
    \item Shearing $\hat{\imath}$ and/or $\hat{\jmath}$
    \item Changing the orientation of $\hat{\imath}$ and $\hat{\jmath}$
\end{itemize}

\vfill

For instance, in Example 4\ref{ex4b}, when we multiplied by the matrix   \newline\\ $\begin{bmatrix} 1 & 0 \\ 1 & 1 \end{bmatrix}$   \newline\\ it transformed the coordinate plane by shearing $\hat{\imath}$:   \newline\\

\begin{tikzpicture}
\draw[<->, thick] (-3.5,0) -- (3.5,0);
\draw[<->, thick] (0,-3.5) -- (0,3.5);
\draw[->, ultra thick, red] (0,0) -- (1,1) node [below right] {$\hat{\imath}'$};
\draw[->, ultra thick, blue] (0,0) -- (0,1) node [above left] {$\hat{\jmath}'$};
\draw [gray!70] (-3, -3) grid (3, 3);
\pgftransformcm{1}{1}{0}{1}{\pgfpoint{0cm}{0cm}}
\draw[dashed, violet] (-3,-3) grid (3,3);
\end{tikzpicture}

\newpage

{\color{red}\textbf{Example 5.}} Find each product.

 \begin{enumerate}[(a)]
 \item \quad $\begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix} \begin{bmatrix} 1 &- 1 \\ 0 & 3 \end{bmatrix}$

    \begin{flushright}
	\begin{tikzpicture}[scale=0.6]
	\draw[gray!70] (-5,-5) grid (5,5);
	\draw[<->] (-5.5,0) -- (5.5,0) node [right] {$x$};
	\draw[<->] (0,-5.5) -- (0,5.5) node [above] {$y$};
	\end{tikzpicture}
	\end{flushright}
\vfill
 \item \quad $\begin{bmatrix} 1 &- 1 \\ 0 & 3 \end{bmatrix} \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$
 \begin{flushright}
	\begin{tikzpicture}[scale=0.6]
	\draw[gray!70] (-5,-5) grid (5,5);
	\draw[<->] (-5.5,0) -- (5.5,0) node [right] {$x$};
	\draw[<->] (0,-5.5) -- (0,5.5) node [above] {$y$};
	\end{tikzpicture}
	\end{flushright}
\end{enumerate}

\newpage

\begin{enumerate}[(a)] \addtocounter{enumi}{2}
 \item \quad $\begin{bmatrix} 2 & 4 \\ -2 & 3 \\ 5 & 1 \end{bmatrix}$ $\begin{bmatrix}
 -3 & 6 \\ 0 & -2
 \end{bmatrix}$
 \vfill

 \item \quad $\begin{bmatrix}
 -3 & 6 \\ 0 & -2
 \end{bmatrix}$ $\begin{bmatrix} 2 & 4 \\ -2 & 3 \\ 5 & 1 \end{bmatrix}$
 \vfill
\end{enumerate}

\newpage

\begin{enumerate}[(a)] \addtocounter{enumi}{4}
\item \quad $\begin{bmatrix} 1 & -2 \end{bmatrix} \begin{bmatrix}
4 & -1 \\ 5 & 1
\end{bmatrix}$
\vfill

\item \quad $\begin{bmatrix}
0 & 2 & -1 \\ 4 & 1 & 0 \\ 0 & -1 & 2
\end{bmatrix}$
$\begin{bmatrix}
4 & 3 & 0 \\ -1 & 0 & 2 \\ 1 & 0 & -2
\end{bmatrix}$
\end{enumerate}
\vfill

\end{document}


\section{Hyper-parameter Optimization}%
\label{sec:hyper_parameter_optimization}

Before a machine learning model can be trained, certain decisions regarding the
architecture of the network have to be made. This involves parameters such as
the number of layers in feedforward networks, kernel sizes in
convolutional nets or, as in the case of the ESN, sparsity and spectral radius
of the reservoir.  These parameters are called \emph{hyper-parameters} (HP) and
they fill a compact space $\mathbf{X}$.  The goal of HP optimization is to find
a configuration $\mathbf{x_i}$ that maximizes the performance of the network:
\begin{equation}
  \arg \max_{\mathbf{x_i} \in \mathbf{X}} f(\mathbf{x_i})
\end{equation}
The function $f$ is not known and very expensive to evaluate as one evaluation
amounts to the creation of a trained NN.  Often this optimization problem is
solved by handily tuning each HP until a \emph{reasonably} good
$\mathbf{x_{\text{opt}}}$ is found, but there are a few methods that can be
applied to automate this process.  The two simplest methods are \emph{grid
searches} and \emph{random searches} of the HP space.  First, the space of
valid HPs is defined.  For example, a number of units $N = (10, 20, 30,...,
100)$ and a learning rate $\eta = (0.1, 0.01, 0.001, ...)$. Grid search then
performs an exhaustive search of all parameters, which is very simple to
implement but suffers from the curse of dimensionality.  Random search randomly
samples $\mathbf{x_i}$ from the HP space, which does not have the problem of
needing to perform an exhaustive search and is widely applied in practice, as
it is still very simple to implement.

The next section describes an algorithm called \emph{Bayesian Optimization},
which also samples the next $\mathbf{x_i}$ but incorporates the knowledge of
already evaluated points in the HP space.  It utilizes Bayes' Theorem which,
adjusted to this problem, states that the \emph{posterior} probability
distribution of a model $M$ (over the HP space $\mathbf{X}$) given a number of
already evaluated samples $A \in \mathbf{X}$ is:
\begin{equation}
  P(M|A) \propto P(A|M) P(M),
\end{equation}
where $P(M)$ is the \emph{prior} probability distribution over $X$, and
$P(A|M)$ the \emph{likelihood} of the samples given $M$.

\subsection{Bayesian Optimization}%

The following description is largely based on article~[\cite{brochu2010bayesopt}]
and will briefly introduce the concept of Gaussian
Processes and how they are used in Bayesian Optimization as introduced
by~[\cite{williams1996gaussian}].  In summary, the Bayesian Optimization
algorithm tries to maximize an objective function $f$ by balancing exploration
(evaluating areas where the true values of $f$ are very uncertain) and
exploitation (which tries to evaluate $f$ where it is expected to be high). It
rests upon the concept of Gaussian Processes (GP), which can be used to define
a distribution over $f$.  From the GP it is possible to calculate an
acquisition function, which is used to efficiently sample the next
$\mathbf{x}_i$ that should be evaluated.  There are different acquisition
functions that emphasize either exploration or exploitation. The method of
Bayesian Optimization yields good results with only few evaluations and is very
likely to perform well on objective functions with local maxima.\\

\subsubsection{Gaussian Processes}%
\label{ssub:gaussian_processes}

A GP is defined by its mean function $m(\mathbf{x}_i)$ and its covariance
function $k(\mathbf{x}_i, \mathbf{x}_j)$.  This means that to each argument
$\mathbf{x}_i$ we assign a random variable $f(\mathbf{x}_i)$.  In analogy to
the Normal distribution, a GP is formally written as:
\begin{equation}
  f \sim {GP}(m,k)
\end{equation}
where $m(\mathbf{x}_i)$ can be any function but typically is just zero, and a
common covariance matrix is created with a Gaussian kernel:
\begin{equation} \label{eq:cov}
  \mathbf{K}_{ij} = k(\mathbf{x}_i, \mathbf{x}_j) =
      \exp \bigg(\frac{- ||\mathbf{x}_i - \mathbf{x}_j||^2}{2}\bigg).
\end{equation}
This kernel is close to one for values close to each other and approaches zero
as the values grow further apart, which implies that close values are highly
correlated.
With this definition, a GP is equivalent to a multivariate Gaussian
$\mathcal{N}(\mu, \Sigma)$ with mean vector $\mu$ and a covariance matrix
$\Sigma$:
\begin{align}
  \mu_i &= m(\mathbf{x}_i) \\
  \Sigma_{ij} &= k(\mathbf{x}_i, \mathbf{x}_j)
\end{align}

Given mean and covariance we can use the GP as a prior for the function $f$ of
which we want to find the maximum. By choosing a zero mean, the only actual
prior information is a smoothly operating $f$, implied by the Gaussian kernel.
The interesting part is how to update this prior with information that is
gained by evaluating $f$ at a certain point to obtain a posterior.  Assuming that
we already have $n$ observations $\{\mathbf{x}_{1:n} ; f(\mathbf{x}_{1:n})\}$
we can find the covariance matrix $\mathbf{K}$ with Eq.~\ref{eq:cov}.  To find
the probability distribution of an unobserved point $\mathbf{x}_{n+1}$, the
conditional probability of $f_{n+1}$ given the previous observations (also
called predictive distribution) is needed:
\begin{equation}
  P(f_{n+1} | \mathbf{x}_{1:n} ; f(\mathbf{x}_{1:n}))
    = \mathcal{N}(\mu_p(\mathbf{x}_{n+1}) , \sigma_p^2(\mathbf{x}_{n+1})),
\end{equation}

where $\mu_p$ and $\sigma_p$ are the predicted mean and standard deviation at
an unknown point.  They can be calculated thanks to the properties of the GP,
which state that the observations and the arbitrary point $\mathbf{x}_{n+1}$
are jointly Gaussian:
\begin{align}
  &\begin{bmatrix} f(\mathbf{x}_{1:n}) \\ f(\mathbf{x}_{n+1}) \end{bmatrix}
  \sim \mathcal{N} \bigg(
    0,  \begin{bmatrix}
           \mathbf{K} & \mathbf{k}^T \\
           \mathbf{k} & k(\mathbf{x}_{n+1}, \mathbf{x}_{n+1})
        \end{bmatrix}
  \bigg) \\
  &\mathbf{k} = \begin{bmatrix}
      k(\mathbf{x}_1, \mathbf{x}_{n+1}) &
      \dots &
      k(\mathbf{x}_n, \mathbf{x}_{n+1})
  \end{bmatrix}
\end{align}

From this it is possible to derive that:
\begin{align}
  \mu_p (\mathbf{x}_{n+1}) &= \mathbf{k}^T \mathbf{K}^{-1} f(\mathbf{x}_{1:n}) \\
  \sigma_p (\mathbf{x}_{n+1}) &= k(\mathbf{x}_{n+1}, \mathbf{x}_{n+1})
                               - \mathbf{k}^T \mathbf{K}^{-1} \mathbf{k}
\end{align}

The resulting predictive distribution is typically very cheap to evaluate as
the number of observations is low. With this distribution it is possible
to find the so called \emph{acquisition function}, which enables an educated
guess of the next best point of evaluation of the objective function.

\label{sub:bayesian_optimization}
\begin{figure}
  \centering
  \includegraphics[width=\linewidth]{gpopt_01.pdf}
  \includegraphics[width=\linewidth]{gpopt_02.pdf}
  \includegraphics[width=\linewidth]{gpopt_04.pdf}
  \includegraphics[width=\linewidth]{gpopt_08.pdf}
  \caption{A timeline of Bayesian Optimization. The plots show how the GP
    approximates the objective function better and better after each iteration.
    The maximum of the green acquisition function indicates where $f$ will be
    sampled next.  The uncertainty of already observed points is zero in this
    case, as there is no noise incorporated in the example. However, noisy
    objective functions can be dealt with by only slight additions to the
    described algorithm. An in depth description of Gaussian Processes and
    how they can be applied to ML can be found in~[\cite{rasmussen2004gaussian}].
  }
  \label{fig:gpopt_01}
\end{figure}

\subsubsection{Acquisition function}%
\label{ssub:acquisition_function}

The acquisition function is obtained from the predictive distribution of the
GP and is defined such that it is high where the objective function $f$ is
\emph{potentially} high.  The probability of large objective values is high
where either the uncertainty or the mean (or both) of the GP are large.
Maximizing the acquisition function amounts to sampling $f$ at
\begin{equation}
  \mathbf{x}_{n+1} = \arg \max_{\mathbf{x_i} \in \mathbf{X}}
                     \text{ PI}(\mathbf{x}_i).
\end{equation}
The function PI is an example of an acquisition function called the
\emph{probability of improvement}:
\begin{align}
  \text{PI}(\mathbf{x}_i) &= P(f(\mathbf{x}_i) > f(\mathbf{x}^+)) \\
  &= \Phi\bigg( \frac{\mu_p(\mathbf{x}_i) - \mu_p(\mathbf{x}^+)}
                     {\sigma_p(\mathbf{x}_i)}  \bigg) \nonumber
\end{align}
Here $\Phi$ is the cumulative distribution function. The emphasis of PI clearly
lies on exploitation, as samples with a mean lower than the current maximal
mean will never reach PI values over 0.5.  More sophisticated acquisition
functions than PI, which are not purely driven by exploitation include
\emph{expected improvement}. A description of various acquisition functions
and Bayesian Optimization as a whole can be found in~[\cite{brochu2010bayesopt}].


\section{Physics Background}

\subsection{Resonant Cavities}
We begin our discussion with a description of the physics problem at hand, as well as a derivation for the mathematics used. Implicitly in the title of this project we are looking at \emph{resonant cavities}, which are volumes used to store standing waves. In the case of Electromagnetic waves that means that the walls of this cavity are perfect conductors. Most physics students will be able to draw parallels to a vibrating string as it is a common wave PDE problem. A one dimensional parallel for electromagnetic waves would be two parallel mirrors separated by a distance L. In this case, the normal modes are electromagnetic waves that bounce between the mirrors such that the total trip is an integer number over wavelengths\cite{Zangwill}. This would be analogous to the \emph{standing wave} on the string where the nodes and antinodes are stationary.

The approach we will use to find the standing waves of a conducting cavity focuses on time-harmonic and divergence-free solutions of the homogeneous wave equation in a volume $V$ with surface $S$\footnote{At this point I am showing my future aspirations of being a theorist by using natural units $c=\hbar=1$. This should make little difference to the physics we are studying, and it makes the equations slightly nicer.}.
\ea{
\qty(\laplacian + \epsilon\mu\omega^2)\va{E} &= 0 \ \text{ in V} \label{eq:Helmholtz}\\
\div \va{E} &= 0 \ \text{ in V}\\
\value{n} \cross \va{E} &= 0 \ \text{ on S}
}
Here $\epsilon$ is the electric permutivity of the volume, $\mu$ is the magnetic permutivity, and $\va{E}$ is the electric field which has the following spacial and temporal components:
\eq{
\va{E} = \va{E}(\va{r})e^{-i\omega t}
}
Solving the Helmholtz equation \eqref{eq:Helmholtz} is analytically possible for simple geometries using separation of variables. These geometries include variations on cylinders, rectangular prisms, and spheres. For further reading on the solutions to the Helmholtz equation you are welcome to suffer like every other Physics graduate student and read \cite{Jackson}.

\subsection{Derivation for a Spherical Cavity}
Armed with a deeper understanding of resonant cavities, we can now turn our attention to the problem at hand, spherical cavities. The electromagnetic normal modes of a spherical resonant cavity are time-harmonic, vector spherical waves that satisfy the perfect-conductor boundary condition $\vu{r}\cross \va{E}\big{|}_s=0$ at the cavity’s walls \cite{Zangwill}. If we take $u(r,\theta,\phi)$ to be the solution to \eqref{eq:Helmholtz} then our normal modes would take the form of transverse magnetic and transverse electric waves.
\ea{
\va{E}_E = -i\omega\va{r}\cross\grad u &\hlw{.05} \va{E}_M = \curl(\va{r}\cross\grad u)\\
\va{B}_E = -\curl(\va{r}\cross \grad u) &\hlw{.05} \va{B}_M = -i\omega\va{r}\cross\grad u
}
Most readers will find a parallel to other areas of Electrodynamics in that our solution to the Helmholtz equation will be a linear combinations of radial equations multiplying the spherical harmonics. In this case our radial function will be a combination of spherical Bessel functions ($j_l(kr)$) and spherical Neumann functions ($n_l(kr)$.
\eq{
u(\va{r}) = \sum_{lm}\qty(A_l j_l(kr) + B_l n_l(kr))Y_{lm}(\theta,\phi)
}
Here $k=\frac{\omega}{c}$. It will be related to the zeros of these Bessel functions, which is how we will obtain our Eigenvalues or Eigenfrequencies. In this case we are only concerned with behavior inside the cavity. Since $n_l$ diverges are the origin we are able to force all $B_l$ to be zero\cite{Jackson}. Thus the form of $u$ inside the cavity is the following:
\eqb{
u_{lm}(\va{r}) = A_l j_l(kr))Y_{lm}(\theta,\phi)
}
Computing the various transverse waves above will require some vector calculations on this solution. The following identity will prove to be quite useful due to the nature of spherical harmonics.
\eq{
\curl(\va{r}\cross\grad u) = \va{r}\laplacian u - 2\grad u - r\pdv{r}\grad u
}
This allows us to write the form of the transverse electric and magnetic waves\cite{Zangwill}.
\ea{
E_E &= i\omega j_l(kr)\qty(\frac{1}{\sin\theta}\pdv{Y_{lm}}{\phi}\vu{\theta}-\pdv{Y_{lm}}{\theta}\vu{\phi}) \label{eq:wave1}\\
\nonumber\\
E_M &= -\qty(\frac{l(l+1)}{r^2}\va{r}+\qty[\vu{\theta}\pdv{\theta}+\vu{\phi}\frac{1}{\sin\theta}\pdv{\phi}]\frac{1}{r}\pdv{r})rj_l(kr)Y_{lm}(\theta,\phi) \label{eq:wave2}
}

\subsection{Analytic Solutions}

In order to make sure that we have a complete understanding of the Bessel functions we will do a slight review. Bessel functions are the solutions to Bessel's differential equation:
\eq{
x^2\dv[2]{y}{x} + x\dv{y}{x}+(x^2-m^2)y = 0
}
Most readers will be familiar with the form of cylindrical Bessel functions as the solution to the Laplace equation in cylindrical coordinates. These functions occur in the case where $m$ is an integer value. The solutions to the Helmholtz equation are the spherical Bessel functions. These functions occur when $m$ is a half integer. In such a case we make the transformation $m^2\to l(l+1)$ so that $l$ can be used as an integer index for the functions. It is customary to represent spherical Bessel functions will lowercase letters and cylindrical Bessel functions with uppercase\cite{handbook}. While these functions can be related to cylindrical Bessel functions, they are unique with respect to the fact that there is a much simpler way of writing them. Specifically we will use Rayleigh's formulas and then push forward.
\ea{
j_l(x) &= (-1)^l \qty(\dv{x})^l\frac{\sin(x)}{x}\\
n_l(x) &= (-1)^l+1 \qty(\dv{x})^l\frac{\cos(x)}{x}\\
}
\begin{figure}
    \centering
    \includegraphics[width=.8\linewidth]{Bessel.png}
    \caption{Spherical Bessel functions of various order}
    \label{fig:bessel}
\end{figure}
While we could go down an endless rabbit hole when it comes to Bessel functions, I believe this will be enough for the problem at hand.

We're most interested in finding the Eigenfrequencies of the spherical cavity. To do so we look back at the boundary condition $\vu{r}\cross \va{E}\big{|}_s=0$. When applied to equations \eqref{eq:wave1} and \eqref{eq:wave2} this means that the angular comonents of those waves must be zero for any value all values of $\theta$ and $\phi$ when $r=R$ (the outer radius of the vanity). The only way to do so is to set either the Bessel function of the derivative of the Bessel function to zero at the walls.
\ea{
j_l(k_{n,l}R) &= 0\\
\dv{r}\qty(rj_l(k_{n,l}r))\big{|}_{r=R} &= 0 \label{eq:annoying}
}
%The magnetic wave equation requires us to make use of the derivative relation
%\eq{\qty(\frac{1}{z}\dv{z})^m \qty(z^{l+1}J_l(z)) = z^{l-m+1}J_{l-m}(z)}
%Thus simplifying equation \eqref{eq:annoying} to the following:
%\eq{\dv{r}\qty(rj_l(kr)) = \frac{1}{k}\dv{kr}\qty(rj_l(kr))}

This allows us to write down the Eigenfrequencies in terms of the zeros of the bessel functions ($x_n,l)$.
\eqb{
\omega = x_{n,l}\frac{c}{R}
}
The zeros have multiple frequencies as the zeros of the $l$ bessel function are not the same as the zeros of the $l+1$ function. Moreover, due to the periodic nature of $\sin$ and $\cos$ each order of Bessel function has many zeros.

At this point it is clear that our goal is calculate the zeros of the various Bessel functions. Doing so by hand is rather straight forward, but it takes time, especially for higher order Bessel functions. Since these functions have been around for so long most people consult a table of zeros when they are needed. In our case

\section{Computational Solutions}
%Be sure to mention what you are using and why
During lecture we looked at multiple methods for root finding numerically. These methods have a direct parallel to a sort of graphical search in that they were easy to animate. Specifically we looked at four methods:
\begin{enumerate}
    \item Simple Search: Where we took a midpoint of a specific range and cut the step size in half each iteration
    \item Bisection: This method utilizes the intermediate value theorem to find one root in a given interval
    \item Secant Line: Draws a line connecting two points and uses the intersection at zero to start the next integration.
    \item Utilizes the derivative of the function to draw a tangent line to begin next iteration.
\end{enumerate}
Since our Bessel functions are analytic the tangent method should be the most efficient. The scipy library holds Bessel functions and derivatives of Bessel functions for integer and half integer values. I used that alon with the algorithms mentioned in class to design my implementation.

Comparing these methods proved that assumption correct. We were also able to animate the search to more intuitively see how accurate they were. In general, even the simple search eventually hit the desired accuracy of $\pm 1\cross 10^{-6}$ within the max iterations.
\begin{figure}
    \centering
    \includegraphics[width=.7\linewidth]{simple.png}
    \caption{The final plot of the simple search's attempt to find a zero of a Bessel function of the first kind}
    \label{fig:simple}
\end{figure}

While the animations may look nice I have a larger issue at hand. There are different algorithms in existence that can calculate the zeros of spherical Bessel functions quite exactly using the relation to cylindrical Bessel functions. In particular I think of the \href{https://scipy-cookbook.readthedocs.io/items/SphericalBesselZeros.html}{provided link} as my direct competition. This method also makes use of an algorithm that is able to use the previous zero to calculate the next one. Ideally I would like to be able to do something that these methods can't do.

These methods do have one fatal flaw, they require you to calculate every zero previous to the one that you are looking for. Since our method is more \emph{graphical} than a series, I do not have the same restraint. The hard part I need to overcome is I need a way to estimate a start point for my tangent search.

If we ignore the area right around zero (\emph{I'll never beat the other algorithms at that small of a value of x anyway}) a spherical Bessel function of order 0 behaves like a $\sin$ function of decreasing amplitude.
\eq{
j_0 = \frac{\sin(x)}{x}
}
The zeros of a sin function are integer values of $\pi$ so I am already able to know exactly where these zeros are. Now if we look at a Bessel function of order 1 we essentially just subtract $\cos$.
\eq{
j_1 = \frac{1}{x}(\frac{\sin(x)}{x} - \cos(x))
}
Once again it is pretty easy to guess these roots, they occur at $z\pi-\frac{\pi}{2}$. At the risk of being too general I thought I could generalize this process. For a Bessel function of order $l$ the n-th zero should occur at:
\eqb{
\text{Estimated Zero } = (n+1)\pi-\frac{l\pi}{2}
}
Armed with this knowledge I might actually stand a chance against those other algorithms for higher values of $x$, which is the same as saying higher Eigenfrequencies. Testing this out when beautifully even at order 16 we were able to calculate the 30th root with only 7 iterations.\\
\\
Thus I believe I can make the following conclusion. Whilst it is more accurate to analytically calculate or use the series method \href{https://scipy-cookbook.readthedocs.io/items/SphericalBesselZeros.html}{outlined here} to calculate the zeros of spherical Bessel functions. Our iteration excels at higher orders and highers Eigenfrequencies.

\section{Applications and Current Research}

Whilst I was searching for information on resonant cavities I came across some interesting research currently happening with regards to this topic. First I found \cite{research1} which used radial basis functions to solve the Helmholtz equation in rectangular coordinates as opposed to spherical ones. While I did not completely understand the math, the paper did have a very interesting visualization of the normal modes inside the chamber. I also found \cite{research2} which technically used an overall cylindrical geometry, however it did make use of perturbation theory which I found to be interesting.


\chapter{Unsupervised Learning}
\label{chp:unsuplea}
In unsupervised learning there is no response variable Y. There is only a set of features $X_1, X_2,..., X_p$ measured on n observations.
Supervised learning compared to unsupervised learning comes with some challenges. The tasks tends to be more subjective and there is no simple goal such as prediction of a response as there is in supervised learning. Furthermore there is no accepted method for determining the quality of the result, due to the fact that there is no Y to check the results up against.
In this section the focus will be on the  technique called \emph{clustering} which is used to discover subgroups or clusters in a data set.
This is done by dividing the data set into clusters so that the observations in each of the clusters are similar to each other, and the observations belonging to clusters are different compared to the observations in other clusters.
The problem is to determine if observations are similar or different to each other.

\section{K-Means Clustering}
\label{chp:clus}
K-means clustering is a technique to divide observations in a data set into a prespecified k number of clusters. Each observation is assigned to one of the k clusters. In figure \ref{fig:K-meanex} a data set is assigned to three clusters. This means that \emph{k} is equal to 3. The data set consists of 50 observations. Each cluster is marked with a color and the cross is the center point of a cluster. This point is also called a \emph{centroid}.

\myFigure{K-mean.PNG}{Example of the K-means algorithm where k is 3}{fig:K-meanex}{0.55}

K-means clustering needs to satisfy two properties.
\begin{itemize}
	\item Each observation belongs to one of the clusters
	\item Clusters cannot overlap, which means that no observation can belong to more than one cluster
\end{itemize}

To get a good clustering the within-cluster variation needs to be as small as possible. So the algorithm tries to minimize this. The within-cluster variation is commonly calculated with the sum of squared pairwise Euclidean distance between observations and the centroid they belong to, divided by the number of observations belonging to that centroid.
The K-means clustering algorithm can then be written as in algorithm \ref{algo:KMeansClustering}. Other initialization algorithms than the one described in line 1 exists such as K-means++ \citep{kplusplus}.

\begin{algorithm}
	\caption{K-Means Clustering}
	\label{algo:KMeansClustering}
	\begin{algorithmic}[1]
		\State A random number from 1 to k is chosen for each observation. This number will act as an initial cluster assignment for every given observation.

		\State
		Iterate until the observations stop changing between clusters.
		\begin{enumerate}[label=(\alph*)]
			\item The centroids are calculated for each of the k clusters and these centroids act as the vector for the means for each observation in the k'th cluster.

			\item For each observation the Euclidean distance to each centroid is calculated and the observation is assigned to the closest one.
		\end{enumerate}
	\end{algorithmic}
\end{algorithm}

\subsection{Lab 10.5.1 - K-Means Clustering}
First, 50 random two dimensional data points with a normal distribution are generated. Then the mean is shifted for 25 of the observations such that two clusters should exists. After this, the K-means from the sklearn library is used to perform the K-means clustering.
After the algorithm is used, the labels and the centroids are saved. The code is shown in listing \ref{lst:kmeanscode} with the number of clusters set to k = 2.
\begin{lstlisting}[language=Python, label=lst:kmeanscode, caption=The KMeans function]
k = 2
kmeans = KMeans(n_clusters=k, init="k-means++",n_init= 20).fit(zip(X,Y))
Labels = kmeans.labels_
centroids = kmeans.cluster_centers_
\end{lstlisting}

The K-means also takes two other parameters. Namely the \emph{init} and the \emph{n\_init}. Init states which approach is used, and in this case the \emph{K-means++} approach is used. The n\_init is how many times the approach should be performed, and this is set to 20. The algorithm then chooses the best initialization of the 20 that was run.

The clusters are visualized with the code in listing \ref{lst:kmeansplot}.
\newpage
\begin{lstlisting}[language=Python, label=lst:kmeansplot, caption=Plotting the two clusters]
lines = plt.plot(centroids[:,0], centroids[:,1], 'kx')
plt.setp(lines, ms=15.0)
plt.setp(lines, mew=3.0)

for i in range(k):
	dsx = X[np.where(Labels==i)]
	dsy = Y[np.where(Labels==i)]
	plt.plot(dsx, dsy, 'o')

plt.show()
\end{lstlisting}

Initially the two centroids are plotted. This is done from line 1 to 3. The \emph{kx} is written so they are labeled with a cross. For plotting the observations a for loop is used. It runs through the two possible clusters. The \emph{o} is added so the observations are plotted as a dot.
The result can be seen in figure \ref{fig:kmeansResult}.

\myFigure{KmeansResult.PNG}{The plotted result of lab 10.5.1}{fig:kmeansResult}{0.6}

The centroids are marked with crosses and each observation is either blue or green depending on which cluster it is assigned too. The observations are clusters as expected but only because k was set correctly. If the true number of clusters were not known, then choosing k would have been a more difficult task, this is often the case when clustering is needed.

\section{Hierarchical Clustering}
Hierarchical clustering does not require a prespecified k value as K-means clustering does.
Furthermore hierarchical clustering produces a dendrogram which shows the observations as a tree structure.
The most common type is called the agglomerative clustering, it is also known as the bottom-up approach.

\subsection{Dendrograms}
Figure \ref{fig:dendrogram} shows a dengrogram which is constructed of a data set with 50 observations.
One leaf is equal to one of the 50 observations. Observations that fuse into branches indicates similarity in the observations. The observations that fuse lowest in the dendrogram are more similar.

Depending on the data set and the observations, the dendrogram can help determine the number of clusters that fits the data best. Line number one shows where to cut the dendrogram to divide the observations into two clusters. Line number two shows the cut dividing the observations into three clusters. Looking at the dendogram, two clusters would yield the best result.

\myFigure{Dendrogram.PNG}{Dendrogram with 50 observations}{fig:dendrogram}{1}

\FloatBarrier
To show how this works, a simple data set containing 7 observations is shown in figure \ref{fig:dendrogramdata} to the right.
It can be seen that the observations 5 and 7 are close to each other, and observations 1 and 6 are also close to each other. These are at the bottom of the dendrogram, as it can been seen in the left side of figure \ref{fig:dendrogramdata}. Now the dendrogram is expanded, and by looking at the next observation that are closest to 5 and 7. Clearly 3 is the closest one, so this is the next leaf in the dendrogram. The same is done for observations 1 and 6 and at the end the dendrogram will look like figure \ref{fig:dendrogramdata}.

\myFigure{DendrogramData.PNG}{Illustration of a dendrogram and its data set}{fig:dendrogramdata}{1}

\FloatBarrier
\subsection{The Hierarchical Clustering Algorithm}
To produce the dendrogram the hierarchical clustering algorithm is used. First step is to find the Euclidean distance between the pair of observations and find the dissimilarities.
Then it runs iteratively trough all the observations. It starts at the bottom and finds the two observations acting as their own cluster that are closest, and fuses those two so that there now are $n-1$ clusters. This continues, so the next two clusters that are most similar are fused, making the number of clusters $n-2$. This is done until all observations belong to one big cluster.

The hierarchical clustering algorithm is shown in algorithm \ref{algo:HierarchicalClustering}.

\begin{algorithm}
	\caption{Hierarchical Clustering}
	\label{algo:HierarchicalClustering}
	\begin{algorithmic}[1]
 		\State Given n observations, measure the pairwise dissimilarities for all  $(\dfrac{n}{2}) =n(n-1)/2 $. Treat each observation as its own cluster.

 		\State for \emph{i} $= n,n-1,...,2:$
 		\begin{enumerate}[label=(\alph*)]
 			\item Examine the pairwise dissimilarities among the i clusters and identity the pair which is most similar and fuse them. The pairwise dissimilarity describes at which height the fuse should be placed in the dendogram.
 			\item Compute the new pairwise inter-cluster dissimilarity for the remaining $\emph{i}-1$ clusters.
 		\end{enumerate}
 	\end{algorithmic}
 \end{algorithm}

One issue has not been addressed. The algorithm needs to be capable of defining the dissimilarities between two clusters if one or both contains multiple observations. This is done with an extension called linkage which defines dissimilarities between two groups of observations. The most common types of linkage are complete, average, single and
centroid. These are explained in table \ref{table:linkage}

\begin{table}
	\begin{center}
		\begin{tabular}{ | l | p{12cm} |}
			\hline
			Linkage & Description \\ \hline
			Complete & Maximal intercluster dissimilarity. Record the largest of the pairwise dissimilarities between observations in cluster A and B \\ \hline
			Single & Minimal intercluster dissimilarity. Record the smallest of the pairwise dissimilarities between observation in cluster A and B \\ \hline
			Average & Mean intercluster dissimilarity. Record the average of the pairwise dissimilarities between the observations in cluster A and B\\
			\hline
			Centroid & Dissimilarity between the centroids/means for the clusters A and B
			\\
			\hline
		\end{tabular}
	\end{center}
	\caption{Different linkage methods}
	\label{table:linkage}
\end{table}

Within statistics complete and average linkage are generally the most preferred linkage as they produce more balanced dendrograms \citep[pp. 394-395]{ISLR}.

%Sometimes you need to scale your data set to obtain standard deviation. Then new problems arises, because of which dissimilarity measure should be used and which linkage should be used. This is a difficult problem and there is no standard answer.

\subsection{Lab 10.5.2 - Hierarchical Clustering}
A random data set is generated the same way as in lab 10.5.1.
The hierarchical clustering dendrogram will be produced using both complete, single and average linkage with Euclidean distance for measuring dissimilarity.
To perform the hierarchical clustering the library scipy is imported.
The code for creating the three types of linkage are shown in listing \ref{lst:Linkage}.

\begin{lstlisting}[language=Python, label=lst:Linkage, caption=Applying the three different linkage styles on the data set]
complete = linkage(X, method='complete', metric='euclidean')
single = linkage(X, method='single', metric='euclidean')
average = linkage(X, method='average', metric='euclidean')
\end{lstlisting}

Three parameters are passed: the data set, the method describing the chosen linkage algorithm and metric. The result of each of the three different dendrograms can be seen in figures \ref{fig:dendrogramsCompAvg} and \ref{fig:dendrogramsingle}.
%To create the dendrograms the function \emph{dendrogram} from the scipy library is used. The only parameter that is needed to build the dendrograms is the three linkage objects that was made in listing \ref{lst:Linkage}.

%When plotting the dendrograms the only difference is which linkage object is parsed as a parameter.

\mySubFigure{DendrogramComplete.PNG}{DendrogramAverage.PNG}{Dendrograms for linkage methods complete and Average}{Complete}{Average}{fig:dendrogramsCompAvg}{fig:denComplete}{fig:denAverage}

\myFigure{DendrogramSingle.PNG}{Dendrogram for the linkage method single}{fig:dendrogramsingle}{0.7}

As expected the dendrograms are different from each other.
Looking at figures \ref{fig:denComplete} and \ref{fig:denAverage} average and complete looks more balanced compared to figure \ref{fig:dendrogramsingle} where the method single is used. Both average and complete also placed the 25 observations correctly in each their cluster. Single instead placed one observations in its own and 49 observations in the other. By looking at the dendogram it can also be seen that the three observations on the left should probably be placed in two different clusters, making a total of four clusters. Using four clusters with the single linkage method also gave a better result. Now only four observations were placed wrongly in a cluster.


\section{Numerical experiments} \slabel{exp}

To evaluate the simple model of \sref{model}, we conduct a battery of direct numerical simulations.
The novel components of the simple model, i.e. viscous drag and mixing, are most pronounced at late times.
We primarily direct our effort at simulating higher aspect ratio domains to allow the bubble to reach a dissipative flow.

The numerical experiments simulate the incompressible Navier-Stokes equations with the Boussinesq approximation:
\begin{align}
\frac{\partial u}{\partial t} + u \cdot \nabla u &= \nu \nabla^2 u - \nabla P + A g \phi \\
\frac{\partial \phi}{\partial t} + u \cdot \nabla \phi &= D \nabla^2 \phi
\end{align}
where $u$ is the velocity,
$\nu$ is the kinematic viscosity,
$P$ is the pressure,
$\phi$ the non-dimensional density,
and $D$ is the diffusivity of $\phi$.

The initial conditions are quiescent with a horizontal interface perturbed by product of cosine functions and smeared by an error function:
\begin{equation}
\begin{split}
	\phi(x,y,z,t=0) = \\
	\text{erf}\left(\frac{z + a_0 \cos(2 \pi (x/\lambda)) \cos(2 \pi (y/\lambda))}{\delta})\right)
\end{split}
\end{equation}
where $a_0$ is the initial amplitude and $\delta$ is the initial interface thickness.
Both $a_0$ and $\delta$ are taken to be small enough to minimize their effects on the solution, $0.01$ and $1/128$, respectively.
The governing equations and initial condition have four dimensional parameters: $\nu$, $D$, $Ag$, $\lambda$.
These are combined into 2 non-dimensional numbers, the Grashof number and the Schmidt number:
\begin{equation}
\text{Gr} = \frac{A_0 g \lambda^3}{\nu^2} \quad \text{Sc} = \frac{\nu}{D}
\end{equation}
The Grashof number serves the role of a Reynolds number for instability problems without a consistent characteristic velocity.
For this reason, the root of the Grashof number is sometimes called the \textit{perturbation Reynolds number}~\cite{Wei2012}:
\begin{equation}
\text{Re}_p = \sqrt{\frac{A_0 g \lambda^3}{\nu^2}}
\end{equation}

The domain is $\left[0.5, 0.5, 64\right]$ and rotated 45 degrees in the span-wise plane to model $\lambda = \sqrt{2}$, transforming the initial condition to:
\begin{equation}
\begin{split}
	\phi(x,y,z,t=0) = \\
	\text{erf}\left(\frac{z + a_0 \cos(\pi (x+y)) \cos(\pi (x-y))}{\delta})\right).
\end{split}
\end{equation}
This is done so the span-wise boundaries at $x=\{0,0.5\}$ and $y=\{0,0.5\}$ are symmetric.
The length of the domain is $64/\sqrt{2} \approx 45.2$ wavelengths with no-slip walls at the top and bottom.
Based on a previous validation of the smRTI with no-slip boundaries, we expect the bubble to be unaffected by the top and bottom walls until it reaches 75\% of the height, or about $17\lambda$.
This provides significantly more data than the $h < 4 \lambda$ results of Ramaprabhu \etal~\cite{Ramaprabhu2012}.

The model introduced in \sref{model} assumes the bubbles and spikes are coherent structures, that is they travel at some velocity and have a well defined interface.
As the Grashof number increases and the bubbles and spikes break up, departing from the assumptions of the model.
On the other hand, at low Grashof number and finite diffusivity, diffusion moves the $\phi = 0$ interface, as opposed to simply transporting the scalar across it, which also departs from the model assumptions.
For these reasons, we restrict our study to an intermediate range of Grashof numbers: those which are large enough to sustain bubble dynamics while not being so large as to break the bubbles apart.
This range has been identified empirically to be approximately $6 \times 10^2 \le \text{Gr} \le 6 \times 10^5$ for Schmidt numbers greater than 1.

The number of spatial samples needed to resolve the advection-diffusion equation for the scalar goes with the Peclet number to the third power.
It is prohibitively expensive to perform calculations at high Schmidt numbers and high Grashof numbers.

Simulations are performed with the NekBox version~\cite{NekBox} of the Nek5000 code~\cite{argonne:nekdoc}, which has been previously validated against single-mode Rayleigh-Taylor experiments~\cite{Hutchinson2016,Wilkinson2007}.
The spectral element method implemented by NekBox has purely dispersive errors and converges exponentially with the spectral order~\cite{fischer:hom}.
The resolution parameters, the number of spectral elements, the order of the spectral elements, and the time step were chosen to achieve an accuracy of $10^{-4}$ in the bubble aspect ratio~\cite{hutchinson2016efficiency}.

\subsection{Observables}

\begin{comment}
\begin{figure}
\includegraphics[width=\columnwidth]{figs/swp_Sc_1.0_nu_8.0-t_max_z-0256.eps}
\includegraphics[width=\columnwidth]{figs/swp_Sc_1.0_nu_8.0-t_proj_z-0256.eps}
\caption{\flabel{profiles}
  Span-wise max and mean of scalar field.
}
\end{figure}
\end{comment}

\begin{figure}
\includegraphics[width=\columnwidth]{figs/comp-height-8-8}
\caption{ \flabel{heights}
  Comparison of height metrics at $\text{Gr} = 4.8\times10^4$ and $\text{Sc} = 1$.
}
\end{figure}

\paragraph{Bubble height}
For miscible RTI, the shape of the scalar field is due to a combination of advection in the bubble and diffusion across the interface.
We can assume an error function-like profile across the interface at the bubble tip, but diffusion across the bubble sidewalls results in a linear profile in both the span-wise mean and maximum of the scalar.
To separate the definition of the bubble tip from sidewall mixing, which is incorporated by the decreasing effective Atwood number, we introduce a new definition: the bubble tip is defined as the inflection point in the span-wise maximum scalar profile.
For the symmetric case, this span-wise maximum of the scalar is equivalent to the value along the bubble axis.
While mixing leads to a linear decay behind the bubble tip, the profile remains sharp near the bubble tip, decoupling the position of the inflection point from the sidewall mixing.

This definition of the bubble height is compared to two more traditional definitions, based on a cutoff in the mean or maximum profiles, in \fref{heights}.
At early times, the definition based on the mean profile grows diffusively.
At late times, the definition based on the max profile kinks as the linear part of the profile crosses zero and then stagnates.
The definition based on the inflection avoids both breakdowns while agreeing with the two traditional definitions within each of their valid ranges.

\paragraph{Mixed volume}

The scalar is normalized such that $\phi \in \left[-1, 1\right]$ and the average $\bar\phi = 0$.
The purity of the fluid is therefore $\left| \phi \right|$ and the volume of mixed fluid is given by a simple integral:
\begin{equation}
M(t) = \int \left( 1 - \left| \phi(x,y,z) \right|\right) dV
\end{equation}


\section*{Discussion}

Our method produces coherent movement in objects without explicit regularization, and is able to
converge on the same order as compared to prior work. It also does not suffer extremely significant deformations around time extremities as compared to D-NeRF, since we are analytically able to ensure that at time 0 there is no motion.

In addition, our method demonstrates extremely smooth interpolation, as enforced by the structure of our network.
We are thus able to exaggerate movements significantly, while ensuring plausibility between frames. In addition, because of the analytic nature
of splines, we are likely able to perform interesting motion deformations, such as diverging motion paths in order to construct novel motion.

One issue with our approach is that because it is forced to learn continuous representations, it has some difficulty when surfaces collide and bounce apart, as it must separate each surface but the boundary between them may be very small and hard to distinguish. We expect that in the limit these issues would vanish, but they slow down training.

Another component of our model are the computational challenges with spline evaluation. We are currently unaware of a fully vectorized method for evaluating splines at a given point, and developing an algorithm for such would benefit our model.


\hypertarget{classnumpp_1_1differentiation_1_1symbolic_1_1multiply}{}\section{numpp\+:\+:differentiation\+:\+:symbolic\+:\+:multiply$<$ Left, Right $>$ Class Template Reference}
\label{classnumpp_1_1differentiation_1_1symbolic_1_1multiply}\index{numpp\+::differentiation\+::symbolic\+::multiply$<$ Left, Right $>$@{numpp\+::differentiation\+::symbolic\+::multiply$<$ Left, Right $>$}}
\subsection*{Public Types}
\begin{DoxyCompactItemize}
\item
\mbox{\Hypertarget{classnumpp_1_1differentiation_1_1symbolic_1_1multiply_a0d483b631e2e23b2f0f316ff192213aa}\label{classnumpp_1_1differentiation_1_1symbolic_1_1multiply_a0d483b631e2e23b2f0f316ff192213aa}}
{\footnotesize template$<$std\+::size\+\_\+t Active$>$ }\\using {\bfseries derivative} = simplify\+\_\+addition$<$ simplify\+\_\+multiplication$<$ typename Left\+::template derivative$<$ Active $>$, Right $>$, simplify\+\_\+multiplication$<$ Left, typename Right\+::template derivative$<$ Active $>$ $>$ $>$
\end{DoxyCompactItemize}
\subsection*{Static Public Member Functions}
\begin{DoxyCompactItemize}
\item
\mbox{\Hypertarget{classnumpp_1_1differentiation_1_1symbolic_1_1multiply_a02603172de5069f4a519d66ca326cd53}\label{classnumpp_1_1differentiation_1_1symbolic_1_1multiply_a02603172de5069f4a519d66ca326cd53}}
static C\+O\+N\+S\+T\+E\+X\+PR auto {\bfseries calculate} (auto \&\&values)
\end{DoxyCompactItemize}

The documentation for this class was generated from the following file\+:\begin{DoxyCompactItemize}
\item
differentiation/symbolic/arithmetic.\+hpp\end{DoxyCompactItemize}


\documentclass{article}
\usepackage{tocloft}
\include{common_symbols_and_format}
\renewcommand{\cfttoctitlefont}{\Large\bfseries}
\begin{document}
\logo
\rulename{Triple Exponential Average} %Argument is name of rule
\tblofcontents

\ruledescription{The triple exponential average, known more commonly as the TRIX is a momentum indicator that is meant to filter out insignificant and unimportant price movements. Many consider it to be similar to the Moving Average Convergence/Divergence (MACD) indicator. The only difference is that the TRIX indicator provides outputs that are smoother due to triple smoothing of the exponential moving averages used to create the indicator. It is an indicator that is usually used for identifying overbought and oversold market conditions.}

\howtotrade
{The strategy is to identify the oversold and overbought markets and a momentum indicator. \\
Bullish Momentum: when the TRIX is above zero. \\
Bearish Momentum: when the TRIX is below zero.
}

\ruleparameters
{Window size for \break Exponential moving average of series i, \break i $\in \{1,2,3\}$}{14}{This is the number of time steps over which exponential contributions are sourced.}{$\lookbacklength_{i}$}
{Smoothing Factor for \break Exponential moving average of series i, \break i $\in \{1,2,3\}$}{2}{Smoothing factor represents the weighting applied to the most recent period’s value.}{$S_{i}$}
\stoptable

\newpage
\section{Equation}
\begin{equation}
    EMA_{\currenttime}^{1} = \Big(V_{\currenttime} * \Big(\frac{S_{1}}{1 + \lookbacklength_{1}}\Big)\Big) + EMA_{\currenttime-1} * \Big(1-\frac{S_{1}}{1 + \lookbacklength_{1}}\Big)
\end{equation}

\begin{equation}
    EMA_{\currenttime}^{2} = \Big(EMA_{\currenttime}^{1} * \Big(\frac{S_{2}}{1 + \lookbacklength_{2}}\Big)\Big) + EMA_{\currenttime-1}^{1} * \Big(1-\frac{S_{2}}{1 + \lookbacklength_{2}}\Big)
\end{equation}

\begin{equation}
    EMA_{\currenttime}^{3} = \Big(EMA_{\currenttime}^{2} * \Big(\frac{S_{3}}{1 + \lookbacklength_{3}}\Big)\Big) + EMA_{\currenttime-1}^{2} * \Big(1-\frac{S_{3}}{1 + \lookbacklength_{3}}\Big)
\end{equation}

\begin{equation}
    TRIX_{\currenttime} = \frac{EMA_{\currenttime}^{3} - EMA_{\currenttime-1}^{3}}{EMA_{\currenttime-1}^{3}}
\end{equation}
\\

where: \\

$TRIX$ is Triple exponentially weighted moving average. \\

$EMA$ is exponentially weighted moving average. \\

$V_{\currenttime}$ is current stock value. \\

$\lookbacklength$ \ is look back length. \\

$S$ is the smoothing factor.

\hspace{200mm}
\hspace{200mm}
\keyterms
\furtherlinks
\end{document}


\section{Complexity and performance}

\subsection{Time complexity of the Merge-SVD algorithm}
The overall complexity of the \cref{alg:svd-dist} can be expressed in
terms of functions $\func{Basecase-SVD}$ and $\func{Merge-SVD}$; but
given the former is seen as a black box over which we have little
control, and that the main contribution of \Rehurek (from SVD
perspective), is the \cref{alg:merge-svd}, we focus on the complexity
of that $\func{Merge-SVD}$ alone. \\

Let us review the cost of its main steps of
\cref{alg:merge-svd} individually, as a way to arriving to the the overall
complexity (we will not consider the possible parallelization or
vectorization of the basic kernel operations \footnote{A ``kernel'' in
the context of Numerical Linear Algebra, is a basic routine which is
heavily used by higher level algorithms; hence, its performance is
crucial and they are heavily optimized.}, which is usually
achieved by using standard libraries like BLAS or LAPACK):

\begin{itemize}
  \item The matrix multiplication that produces $Z$ in line $1$, is
    done against matrices $\trans{U_1}$ (of dimensions $k_1 \times m$)
    and matrix $U_2$ (of dimensions $m \times k_2$); hence it has a
      complexity \bigO{m k_1 k_2}. \\

  \item The second step is dominated by the QR calculation; according
    to Golub \cite{golub13}, the complexity of a QR factorization
    based on the Gram-Schmidt process for a matrix $A^{m \times n}$,
    is \bigO{m n^2}. Applying that result to the particular case of
    line $2$, give us a complexity of \bigO{m k_2^2}. \\

   \item It seems hard to
     find reported complexities for the SVD algorithms, in the
     available literature; \Rehurek mentions that the complexity of
     the full SVD calculation from line $3$ is
     \bigO{(k_1+k_2)^3}\footnote{We could find at least one reference
       that also mentions this complexity, see
       \cite{plassman05}.}. Hence, given that the truncation factors
     $k_1$ and $k_2$ are usually a few hundreds in the context of LSI;
     the cost of this step can be neglected. \\

   \item Finally, the complexity of the matrix operations in the last step
     (focusing on the products only), is \bigO{mkk_1 + mkk_2}.
\end{itemize}
\hfill

In practice the truncation factors do not vary much in LSI
applications, thus, we can simplify further. Let us assume that $k
\approx k_1 \approx k_2$, then the reported complexities in the list
above become: \bigO{m k^2}, \bigO{m k^2}, \bigO{k^3}, \bigO{m
  k^2}. Given that the number of terms $m$ will be much bigger than
the truncation factor $k$ (hundred of thousands, vs a few hundreds);
we can conclude that the overall time complexity is \bigO{m k^2}. \\

Due time constraints we did not enter into detailed memory complexity
analysis of the algorithm, but is part of our todo list (for the case
that this project evolves into a full thesis).

\subsection{Performance with a large scale corpus}

\Rehurek used 3 different corpus to test his distributed SVD
algorithm, in the context of LSI. We focused only on the large corpus,
which was the English Wikipedia. By that time, it contained $3.2$
million documents; where 100,000 terms were chosen after removing the
stop words. That resulted in an sparse matrix of dimensions
$100,000 \times 3,199,665$, with $0.5$Gb of non zero entries. Such
matrix can fit in memory of a modern personal computer, but as
explained earlier, the main objective of using the distributed
algorithm is to scale in time. The truncation factor $k$ was set to
$400$ during this experiment. \\

On his Phd thesis, \Rehurek reports the following wall times of the
distributed \cref{alg:svd-dist}, running on a single computer and on a
cluster: \\

\begin{itemize}
\item $8.5$ hours on a dual-core 2.53GHz MacBook Pro with 4GB RAM and
vecLib, a fast BLAS (\cite{blas})/ LAPACK (\cite{lapack}) library
provided by the vendor. \\

\item $2$ hours $23$ minutes on a cluster of four dual-core 2GHz Intel
  Xeons, each with 4GB of RAM, which share the same Ethernet segment
  and communicate via TCP/IP. These machines did not have any
  optimized BLAS library installed.
\end{itemize}
\hfill

The above numbers suggest what we expected: given that the
parallelization achieved by the distributed algorithm is almost
perfect (only communication needed is on the final merge), the scaling
in time is basically linear with respect to the number of computing
nodes. \\

The \href{https://radimrehurek.com/gensim/dist_lsi.html}{gensim page}
(see \cite{gensim}) has a more up to date
experiment, which reports $5$ hours 25 minutes for a single machine;
and $1$ hour with 41 minutes for a cluster with 4 nodes (this time,
the cluster nodes got ATLAS installed, an open source BLAS/LAPACK
implementation (see \cite{atlas}). \\

\Rehurek does an additional comparison for the execution on a single
machine, by contrasting with a custom implementation of the SVD
algorithm published in \cite{zha98}, named as ZMS in his Phd
thesis. The ZMS algorithm took 109 hours, which brutally contrast with
the 2 hours 23 minutes mentioned above for the \cref{alg:svd-dist}. A
probably more fair comparison, would be against the SVD algorithm
implemented by SLEPc (\cite{hernandez07}); thought this opensource
implementation does not target specifically the LSI problem, it claims
to be distributed and highly scalable. Other comparisons with more
opensource implementations are possible: together, along with
reproducing the results published by \Rehurek with more nodes in the
cluster, are planned to be performed in a further stage of this
project.