\section{Recurrent Neural Networks}
\label{sec:recurrent_neural_networks}

The two major types of neural networks are distinguished by the structure of
their internal weights.  Feedforward networks simply pipe their input through
all the layers towards the output. They have proven very useful for tasks that
require static, non-linear input-output mappings such as pattern recognition or
classification.  Processing time series is a different task because, to model a
sequence, the network needs information about the past. In a FNN this is not
the case.  An input vector $\vt{u}$ that is fed into the network $F$ does not
contain information about previous or future inputs and the prediction $\vt{y}$
has to be made solely based on the current input:
\begin{equation}
  \vt{y} = F(\vt{u})
\end{equation}

\begin{figure}
  \begin{minipage}[b]{.4\textwidth}
    \centering
    \FeedForwardNet{1}
    Feedforward Network
  \end{minipage}
  \hspace{.05\textwidth}
  \begin{minipage}[b]{.5\textwidth}
    \centering
    \RecurrentNet
    Recurrent Network
  \end{minipage}
  \caption{The traditional feedforward network on the left only exhibits
  forward connections while the recurrent network on the right has cyclic
  connections within or possibly even between layers if there is more than one
  hidden weight matrix.}
  \label{fig:fnn_rnn}
\end{figure}

In contrast, recurrent neural networks (RNN) possess cyclic weight connections.
The outputs of a layer have feedback weights that are connected to the same or
a previous layer.  This cyclic nature of RNNs mathematically makes them
dynamical systems [\cite{FUNAHASHI}] and it can be shown that they are
\emph{Turing complete} [\cite{siegelmann1991}]. A brief description on the analogies
between RNNs and dynamical systems is given in
Sec.~\ref{ssub:state_space_model}. Roughly, RNNs maintain an internal state
$\vt{x}$ at all times which acts as a memory of previous inputs. At every time
step the network receives the previous internal state along with an input
vector and returns a prediction $\vt{y}$ as well as an updated internal state
$\vt{x}$:
\begin{equation}
  \label{eq:recurrent_network_F}
  \vt{y}, \text{ } \vt{x} = F(\vt{u}, \vec{x}_{t-1}).
\end{equation}

This enables RNNs to process time series data and they are thus qualified for
tasks such as filtering, dynamic pattern recognition, and prediction.  RNNs are
most widely used in speech recognition or other language processing tasks, but
they are also highly interesting from a neurological point of view, as all
biological neural networks are recurrent.  Generally, they are highly promising
tools for non-linear time series modelling.  They can be run in a
self-monitoring fashion, which makes them very interesting for an automated
outlier detection in large scale time series data.  Especially in the field of
Natural Language Processing (NLP), RNNs have achieved impressive results while
requiring little preprocessing of datasets [\cite{sutskever2011generating}].

\subsection{State Space Model}
\label{ssub:state_space_model}

The simplest dynamical system in discrete time is defined by a state space $M$,
a set of times $T$, and an evolution function $\Phi$.  The function $\Phi$ maps
a given state $\vec{x}_{t-1} \in M$ at time $t \in T$ to a new state $\vt{x}$:
\begin{equation}
  \vt{x} = \Phi(\vec{x}_{t-1}).
\end{equation}

A dynamical system with inputs $\vt{u}$ and outputs $\vt{y}$ is defined by
the state space representation:
\begin{align}
  \vt{x} &= \Phi(\vt{u}, \vec{x}_{t-1}), \\
  \vt{y} &= \Psi(\vt{u}, \vec{x}_{t-1}).
\end{align}

In a basic RNN the two functions $\Phi$ and $\Psi$ are defined with an input
matrix $\wmatr{in}$, a recurrent weight matrix $\wmatr{}$, and an output matrix
$\wmatr{out}$:
\begin{align}
  \label{eq:state_space}
  \vt{x} &= \varphi(\wmatr{in} \vt{u} + \wmatr{}\vec{x}_{t-1}), \\
  \label{eq:readout}
  \vt{y} &= \psi(\wmatr{out} \vt{x}),
\end{align}

where $\varphi$ denotes the component-wise applied, non-linear, state
activation function. In RNNs a typical choice is the hyperbolic tangent.  The
output activation function $\psi$ is commonly chosen to be the identity
function, resulting in a linear output layer.  A flow chart of the basic RNN is
shown in Fig.~\ref{fig:rnn_flow_chart}.  The input weights $\wmatr{in}$ have
dimensions $n \times m$, hidden weights $\wmatr{}$: $n \times n$ and
output weights $\wmatr{out}$: $n \times k$. In the simple RNN the input is
not utilized by the output layer, but would be entirely possible to introduce
another matrix to do this.

An input series $\mathbf{u}$ of length $N$ that is fed to the network one by
one produces $N$ internal states.

\begin{equation}
  \mathbf{x} = (\vt{x}, \vec{x}_{t+1}, ..., \vec{x}_{t+N}),
\end{equation}
\begin{figure}
  \centering
  \RNNFlowChart
  \caption{Basic RNN cell flow chart. The recurrent weights are enclosed by the
  non-linearity $\varphi$, the output weights by the function $\psi$.}
  \label{fig:rnn_flow_chart}
\end{figure}

From Eq.~\ref{eq:state_space}, it becomes evident that the internal state acts
as a kind of memory of the network.  This memory is dynamic as opposed to the
static memory brought about by weight adjustments of GD.  The latter is called
long-term memory. The dynamic memory of the internal RNN state is termed
\emph{short-term memory} (STM). STM will be further discussed in
Sec.~\ref{sub:short_term_memory}  and brief computational analysis of the STM
capacity is given in Sec.~\ref{sec:short_term_memory}.

Every new input overwrites a part of the previous internal state, gradually
encoding the input sequence into $\vt{x}$.  The length of an input sequence
that can be encoded into $\vt{x}$ depends on the size of the internal state and
on the two matrices $\wmatr{in}$ and $\wmatr{}$.  Generally, the state size $n$
must be much larger than the input size $m$, in order to create an effective
RNN. How an RNN is trained in the light of a time dependency of the weights is
described in Sec.~\ref{sub:training_recurrent_neural_networks}.

As the internal states now contain information both about current and past
inputs, it is possible for the output matrix $\wmatr{out}$ to create an educated
prediction of the next frame.  An RNN that receives its output as the next
input is called a freely running RNN and enables predictions further into the
future as depicted in Fig.~\ref{fig:annotated_rnn}.  Of course, in this case the
number of input and output units must be the same: $k = m$.

\begin{figure}
  \centering
  \RecurrentNetAnnotated
  \caption{RNN setup that is able to predict $n$ steps into the future by
    feeding the output back into the input. The input and output layers are
    fully connected. The internal weight connections can be sparse to speed
    up the network for large reservoir sizes.
  }
  \label{fig:annotated_rnn}
\end{figure}

\subsection{Training Recurrent Neural Networks}
\label{sub:training_recurrent_neural_networks}

There exist various different methods of training recurrent networks, which all
have their own benefits and drawbacks or just perform better or worse at
different tasks. No clear favourable approach, like the mini-batch GD algorithm
for feedfoward networks, was found yet.  This is due to several obstacles that
arise specifically during RNN training, which will be discussed below.  The
three most common methods are \emph{Backpropagation Through Time} (BPTT),
\emph{Real-time Recurrent Learning} (RTRL, [\cite{williams1989}]), and
\emph{Extended Kalman Filtering} (EKF, [\cite{williams1992}]), the first of which
will be treated below.

\subsubsection{Backpropagation Through Time}
\label{ssub:backpropagation_through_time}

BPTT is an adapted form the of classic backpropagation algorithm of feedforward
networks as described in Sec.~\ref{sec:feedforward_neural_networks} and was
developed for the first time by~[\cite{mozerBPTT}].  The
cyclic connections of RNNs prevent the direct application of the
backpropagation algorithm.  One solution is to \emph{unroll} the network in
time by stacking the network on top of itself for a certain number of time
steps.  The depth of unrolling is determined by the length $N$ of the sequence
$\mathbf{u}$ that is fed into the network.

By unrolling the network in time, one practically ends up with a very deep
feedforward network with shared weights between the stacked layers of clones of
the network.  In the forward pass each clone, which now corresponds to a time
step in the sequence, receives the corresponding input $\vt{u}$ and updates its
own internal state $\vt{x}$.  The internal state of each clone depends on its
input and on the internal state of the previous layer (at time $t-1$).  Finally
the output $\vt{y}$ is computed by each clone.  The loss function that is
minimized is defined by:
\begin{equation}
  \mathcal{L} = \sum_{t=0}^{N} \mathcal{L}_t
              = \sum_{t=0}^{N} || \vt{d} - \vt{y} ||_2.
\end{equation}

The weight adjustment is now done by a typical gradient descent algorithm.  By
collecting all the weights and biases of the state space model in the variable
$\Theta$ we can write the weight adjustment as:
\begin{equation}
  \label{eq:batch_update}
  \Theta' = \Theta + \eta \sum_t \frac{\partial \mathcal{L}_t}{\partial \Theta}
\end{equation}

The expression for the gradient of the cost can by derived by applying the
chain rule:
\begin{equation}
  \newcommand{\loss}{\mathcal{L}_t}
  \dd{\loss}{\Theta} = \dd{\loss}{\vt{x}} \dd{\vt{x}}{\Theta}
  = \dd{\loss}{\vt{x}} \dd{\vt{x}}{\vec{x}_{t-1}} \dd{\vec{x}_{t-1}}{\Theta},
\end{equation}

which results in a product of derivatives, as every state depends on the
previous one.
\begin{align}
  \newcommand{\loss}{\mathcal{L}_t}
  \dd{\loss}{\Theta} = \dd{\loss}{\vt{x}} \dd{\vt{x}}{\vec{x}_{0}} \dd{\vec{x}_{0}}{\Theta} \\
  \dd{\vt{x}}{\vec{x}_{0}} = \prod_{i=1}^t \dd{\vec{x}_i}{\vec{x}_{i-1}} \label{eq:dxtdxN}
\end{align}

The last derivative of the state $\vec{x}_0$ denotes the derivative of the first
state (starting to count from the perspective of the forward pass) of the
unrolled network, which is a constant with respect to $\Theta$.  From
Eq.~\ref{eq:dxtdxN} one can see the origin of the vanishing and exploding
gradient problems.  If the individual derivatives $\dd{\vt{x}}{\vec{x}_{0}}$ are
small, the gradient, being product of these derivatives, vanishes very quickly
and explodes if they are large.  It can be shown that it `[...] is
\textit{sufficient} for the largest eigenvalue $\lambda_1$ of the recurrent
weight matrix to be smaller than one for long term components to vanish (as $t
\rightarrow \infty$) and \textit{necessary} for  it  to  be  larger  than  one
for  gradients  to explode.' -- [\cite{razvan2012}].
By bounding the spectral radius
of the recurrent weights to be smaller than one, it is thus possible to avoid
the exploding gradient problem. A solution to the vanishing gradient problem is
more complicated and involves advanced network architectures such as the long
short-term memory (LSTM) unit [\cite{lstm}].  It introduces additional input,
forget, and output layers, but the essential part is that one of the recurrent maps of
the unit is the identity function.  The derivatives in Eq.~\ref{eq:dxtdxN}
become one and the gradient can flow through many layers.  A completely
different approach is to avoid training the recurrent weights altogether, which
will be described in Sec.~\ref{sec:reservoir_computing}.

\subsection{The Bifurcating State Space}%
\label{sub:the_bifurcating_state_space}

Another problem that arises with the optimization of recurrent weights is that
the state space is not necessarily continuous, which was shown by
[\cite{doya1992}].  The points at which the state space has discontinuities are
called \emph{bifurcations} and are extensively studied in non-linear dynamics.
They can cause discontinuities in the state space and thus impair the learning
or prevent convergence to a local minimum completely.  To understand what
bifurcations are and how they affect RNN training, we will consider the
recurrent part of a single unit RNN with the hyperbolic tangent as the
activation function.  If the RNN has only one unit the state $\vt{x}$, as well
as the weights and biases become scalars:
\begin{equation}
  \label{eq:single_unit_rnn}
  x_{t+1} = \tanh(w x_t + b).
\end{equation}
The parameter $w$ denotes the scalar weight of the single unit and $b = w_{in}
u_t$ will serve as the bias of a constant input of $u_t = 1$.  In
Fig.~\ref{fig:bif_evolution} we can see the evolution of $x_t$.  Depending on
different initial values $x_0$ and network parameters, the state converges to
different values for $t$ towards infinity. These values are called \emph{fixed
points} $x^*$ and for them $x_t = x_{t+1}$ holds.  In particular, fixed points
that the state converges to are called \emph{stable} fixed points (or
\emph{attractors}).  The second kind of fixed points are \emph{unstable}.  The
slightest deviation from an unstable fixed point will result in a flow away
from the point, which is why they are also called \emph{repellers}.  In the
first three cases of Fig.~\ref{fig:bif_evolution} a fixed point is always
reached. The fourth example in the lower right shows representatives of
the oscillating fixed point, more specifically \emph{period-2 cycles}, that
repeat every second iteration.
\begin{figure}
  \centering
  \includegraphics[width=\linewidth]{bif_evolution.pdf}
  \caption{Evolution of $x_t$ over time for different parameters $w$ and $b$.
    Dashed lines show unstable fixed points. Apart from the expected fixed points
    that $x_t$ converges to over time, there are also oscillations visible in
    the last plot. Such oscillations that repeat every 2 iterations are called
    \emph{period-2 cycles} and they appear when $w<-1$.}
  \label{fig:bif_evolution}
\end{figure}
\begin{figure}
  \centering
  \includegraphics[width=\linewidth]{cobweb.pdf}
  \caption{Cobwebs for the same parameters as in Fig.~\ref{fig:bif_evolution}.
    The black dot is the initial value $x_0$. By drawing a vertical line to the
    intersection with the activation function gives the new input $x_1$. Drawing
    a horizontal line to the intersection with $y = x$ projects the point back to
    the $x$-axis.  The projection is the next input to the activation function.
    This process is repeated until a stable orbit or a fixed point is reached.}
  \label{fig:cobweb}
\end{figure}

By varying the parameters $w$ and $b$ the location and the nature of fixed
points can be changed. The blue line in the right plot of
Fig.~\ref{fig:fixed_points} splits in two as $w$ is increased. The point at
$w=1$ is called a \emph{bifurcation} point.  There are two things that are
happening here: the stable fixed point at $x=0$ becomes unstable (indicated by
the dashed line) and two new stable fixed points above and below zero are
created.

Fixed points can be found analytically by rewriting
Eq.~\ref{eq:single_unit_rnn} with the assumption that $x^*$ is a fixed point
(for which $x_t = x_{t+1}$):
\begin{equation}
  \label{eq:fp}
  x^* = \tanh(wx^* +b).
\end{equation}
Solving once for $w$ and once for $b$ results in two equations for fixed
points:
\begin{align}
  b &= \tanh^{-1}(x^*) - wx^*\\
  w &= \frac{\tanh^{-1}(x^*) - b}{x^*},
\end{align}
which can be plotted for different values of $w$ and $b$
(Fig.~\ref{fig:fixed_points}).  The period-2 cycles cannot be found by
analysing Eq.~\ref{eq:fp}.  Instead they can be found analytically by solving
\begin{equation}
  x^* = \tanh^2(wx^* + b),
\end{equation}
but also by an intuitive, graphical approach called \emph{cobwebbing}
(Fig.~\ref{fig:cobweb}).
Starting from an initial point $x_0$ a vertical line is drawn to the value of
the activation function. Now drawing a horizontal line until we intersect with
the graph of $y = x$ gives the new input $x_1$ and so forth.

\subsubsection{Effect on RNN Training}%
\label{ssub:effect_on_rnn_training}

Now that we have an understanding of what fixed points and bifurcations are we
can examine their effect on RNN learning. Suppose we initialize the network
with a constant $b=0.1$ and a $w=3$. If $x_0$ is negative, the nearest fixed
point is on the lower branch of the yellow line in the right plot of
Fig.~\ref{fig:fixed_points}.  Further assume we train the network to output
$x_\infty = - 0.25$.  In this case, $w$ will be lowered to approach $x^*= -
0.25$ until the bifurcation point is reached and the stable fixed point
vanishes (yellow line becomes dashed line). The fixed point becomes unstable
and the network output will change discontinuously as it jumps to the attractor
on the upper branch.  This will result in a discontinuity in the loss function
and an infinite gradient.  After jumping to the upper branch $w$ will grow
towards infinite values as the GD algorithm tries to approach the target value
of $x = - 0.25$.  Similar examples can be constructed in which parameters
oscillate between two bifurcation points.

The weights of RNNs that are used in practice are normally initialized to very
small values which results
in few fixed points. As the network learns some of the weights increase which
drives the RNN through bifurcations.  The discontinuities that result in very
large gradients cause large jumps of the GD algorithm which can nullify the
learning of hundreds of steps in a single iteration. Aside from the vanishing
and exploding gradient problems, bifurcations are another major reason for the
intricacy of RNN training.

\begin{figure}
  \includegraphics[width=\linewidth]{bif_fixpoints.pdf}
  \caption{Fixed points for different parameter values of $b$ and $w$.  The
    values of the fixed points $x^*$ are affected by varying the weights of the
    RNN.  If there is more than one stable fixed point $x_t$ converges to the
    attractor that is closest to the initial value $x_0$.  Dashed lines denote
    unstable fixed points, which can only be reached if $x_0 = x^*$.}
  \label{fig:fixed_points}
\end{figure}

One solution to all the headaches that are caused by the bifurcations in the
state space, and RNN training in general, is surprisingly simple.  The cause of
bifurcations are adaptions of the recurrent weights of the RNN, so keeping them
constant would eliminate all the complications at once and even come with the
additional advantage of not having to change those weights in the first place.
This might seem like a rather drastic method as the whole ML approach is based on
gradual learning of weights. However, restricting the weight optimization to the
non-recurrent weights frees us from the notorious problems of RNNs and can
perform just as well.  This approach is called \emph{reservoir computing} and
will be discussed in the next Section.


\documentclass[a4paper,12pt]{scrartcl}

\usepackage{bm,amsmath,url,graphicx}
\usepackage{palatino}
\usepackage{color, xcolor}
\usepackage{listings}

\newcommand{\n}{{\bf n}}
\newcommand{\h}{{\bf h}}
\newcommand{\x}{{\bf x}}
\newcommand{\HH}{{\bf H}}
\newcommand{\thb}{{\boldsymbol{\theta}}}
\newcommand{\python}{{\fbox{\texttt{\bfseries python}}\quad}}
\newcommand{\pen}{{\fbox{\texttt{\bfseries pen\&paper}}\quad}}

\renewcommand{\familydefault}{\rmdefault}

\begin{document}
\section*{\bf SGN-41007 Pattern Recognition and Machine Learning}
\emph{Exercise Set 2: January 18 - 20, 2017}
\bigskip
\sloppy

\lstdefinestyle{mystyle}{
  belowcaptionskip=1\baselineskip,
  breaklines=true,
  frame=single,
  xleftmargin=\parindent,
  language=Python,
  showstringspaces=false,
  basicstyle=\ttfamily,
  keywordstyle=\bfseries\color{green!40!black},
  commentstyle=\itshape\color{purple!40!black},
  identifierstyle=\color{blue},
  stringstyle=\color{orange},
  moredelim=**[is][\color{red}]{@}{@},
}

\lstset{language=Python,style=mystyle}

\noindent
Exercises consist of both pen\&paper and computer assignments.
Pen\&paper questions are solved at home before exercises, while
computer assignments are solved during exercise hours. The
computer assignments are marked by text \python and
Pen\&paper questions by text \pen

\begin{enumerate}

\item \pen \emph{Find a maximum likelihood estimator for one sample.}

Suppose we measure one sample $x$ for which we assume the model
\[
p(x, \lambda) = \begin{cases}\lambda\exp(-\lambda x) & x \ge 0\\
0, & x < 0\end{cases}
\]

\begin{enumerate}
	\item Write and simplify the expression for $\ln p(x; \lambda)$. For simplicity, you can
	omit the case $x<0$.
	\item Compute the derivative $\frac{\partial \ln p(x; \lambda)}{\partial \lambda}$.
	\item Solve $\frac{\partial \ln p(x; \lambda)}{\partial \lambda} = 0$ with respect to $\lambda$.
	This is the formula for maximum likelihood estimate of $\lambda$.
\end{enumerate}

\item \pen \emph{Find a maximum likelihood estimator for many samples.}

Suppose we measure samples $\x = (x[0], x[1],\ldots,x[N-1])$ for which we assume the model
\[
p(x[n], \lambda) = \begin{cases}\lambda\exp(-\lambda x[n]) & x[n] \ge 0\\
0, & x[n] < 0\end{cases}
\]

\begin{enumerate}
	\item Write and simplify the expression for $\ln p(\x; \lambda)$. You can assume the
	samples are independent and $p(\x; \lambda)$ is simply the product of individual
	probabilities: $p(\x; \lambda) = \prod p(x[n], \lambda)$.
	\item Compute the derivative $\frac{\partial \ln p(\x; \lambda)}{\partial \lambda}$.
	\item Solve $\frac{\partial \ln p(\x; \lambda)}{\partial \lambda} = 0$ with respect to $\lambda$.
	This is the formula for maximum likelihood estimate of $\lambda$.
\end{enumerate}

\item \python \emph{Same as last week Question 1, but without numpy.}

Download the following file and extract the contents:
{\small
\begin{verbatim}
http://www.cs.tut.fi/courses/SGN-41006/exercises/locationData.zip
\end{verbatim}
}
\label{ex1}

\begin{enumerate}
\item Read the file into memory one line at a time (in a for loop). See similar example at the end of lecture slide set 1.
\item Load the same data into another variable using \verb+numpy.loadtxt+. Check that the contents of the
two arrays are equal using \verb+numpy.all+ or \verb+numpy.any+.
\end{enumerate}

\item \python \emph{Implement two utility functions we need later.}

\begin{enumerate}
\item Implement the following function:
\begin{verbatim}
p = gaussian(x, mu, sigma)
\end{verbatim}
which returns the Gaussian density with parameters \verb+mu+ and \verb+sigma+ at point \verb+x+.
In mathematical notation, the function is:
\[
p(x; \mu,\sigma) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{1}{2\sigma^2} (x-\mu)^2\right]
\]
\item Implement the function:
\begin{verbatim}
p = log_gaussian(x, mu, sigma)
\end{verbatim}
that returns $\ln p(x; \mu,\sigma)$. Do not use the the previous function, because
the straightforward solution \verb+p = numpy.log(gaussian(x, mu, sigma))+ would be numerically inaccurate.
Instead, first manually simplify the expression
\[
\ln p(x; \mu,\sigma) = \ln\left(\frac{1}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{1}{2\sigma^2} (x-\mu)^2\right]\right)
\]
and write the corresponding code.
\item Plot both functions for \verb+mu = 0+ and \verb+sigma = 1+ in the interval
$x\in [-5,5]$ to check that the code is correct. \emph{Hint:} create a vector
of x-coordinates using \verb+numpy.linspace+ and pass that through your functions
and plot.
\end{enumerate}

\item \python \emph{Estimate sinusoidal parameters.}

\begin{enumerate}
\item Generate a 100-sample long synthetic test signal from the model:
\[
x[n] = \sin\left( 2\pi f_0 n \right) + w[n], \qquad n = 0,1,\ldots, 99
\]
with $f_0 = 0.017$ and $w[n]\sim {\cal N}(0,0.25)$. Note that $w[n]$ is generated
by \verb+w = numpy.sqrt(0.25) * numpy.random.randn(100)+. Plot the result.
\item Implement code from estimating the frequency of \verb+x+ using the maximum likelihood
estimator:
\[
\hat{f}_0 = \text{value of } f \text{ that maximizes } \left|\sum_{n=0}^{N-1} x(n)e^{-2\pi i f n}\right|.
\]
Implementation is straightforward by noting that the sum expression is in fact a dot product:
\[
\hat{f}_0 = \text{value of } f \text{ that maximizes } \left|\x \cdot {\bf e}\right|,
\]
with $\x=(x_0,x_1,\ldots, x_{N-1})$ and ${\bf e}=(e^{-2\pi i f \cdot 0}, e^{-2\pi i f \cdot 1},\ldots , e^{-2\pi i f \cdot (N-1)})$.

Use the following template and fill in the blanks.
\begin{lstlisting}
scores = []
frequencies = []

for f in numpy.linspace(0, 0.5, 1000):

    # Create vector e. Assume data is in x.

    n = numpy.arange(100)
    z = # <compute -2*pi*i*f*n. Imaginary unit is 1j>
    e = numpy.exp(z)

    score = # <compute abs of dot product of x and e>
    scores.append(score)
    frequencies.append(f)

fHat = frequencies[np.argmax(scores)]
\end{lstlisting}
\item Run parts (a) and (b) a few times. Are the results close to true $f_0 = 0.017$?
\end{enumerate}

\end{enumerate}

\end{document}


\problemname{Forest for the Trees}

You are playing hide-and-go-seek in a forest with Belle.  The forest
has one tree at each of the positive integer lattice points.  That is,
there is a tree at every point $(x,y)$ where $x$ and $y$ are both
positive integers.  You may consider each tree as a point.  A logging
company has cut down all of the trees in some axis-aligned rectangle,
including those on the boundary of the rectangle.

You are standing at $(0,0)$ and Belle is standing at $(x_b,y_b)$.
You can see Belle if and only if there is no tree blocking your line of
sight to Belle. If there is a tree at $(x_b,y_b)$, Belle will make
it easier for you to find her by standing on the side of the tree
facing your location.

For example, suppose that Belle is standing at $(2,6)$.  If the
trees in the rectangle with corners at $(1,1)$ and $(5,4)$ are cut
down (blue rectangle in figure), then you can see Belle.  However,
if the rectangle was at $(3,5)$ and $(5,7)$ (red rectangle in figure),
then the tree at $(1,3)$ would be in the way.

\begin{center}
	\includegraphics[width=0.45\textwidth]{trees.pdf}
\end{center}

Given the rectangle and Belle's location, can you see her?

\section*{Input}

The first line of input contains two integer $x_b$ and $y_b$~($1 \leq x_b,y_b \leq 10^{12}$), which are the coordinates that Belle is
standing on.

The second line of input contains four integers $x_1$, $y_1$, $x_2$ and $y_2$~($1 \leq x_1 \leq x_2 \leq 10^{12}$ and $1 \leq y_1 \leq y_2
\leq 10^{12}$), which specify two opposite corners of the rectangle at $(x_1, y_1)$ and $(x_2, y_2)$.

\section*{Output}

If you can see Belle, display \texttt{Yes}.

Otherwise, display \texttt{No} and the coordinates of the closest tree
that is blocking your view.


% !TEX root = main.tex

%-------------------------------------------------
\section{Conditional expectation}\label{sec:cond_expe}

Conditional expectations are expectations computed with respect to conditional distributions.
%PMFs (discrete case) and conditional PDFs (continuous case).

%%-----------------------------
%\subsection{Conditioning on events}
%\begin{definition}\label{defn:cond_expe_A}
%let $A$ be an event with $\prob(A)>0$, and let $X$ be a random variable. Then
%\[
%\begin{array}{lll}
%\expe(X|A) & = \displaystyle\sum_{i=1}^{\infty} x_i\,f_{X|A}(x_i)		& \quad\text{if $X$ is discrete, or} \\[4ex]
%\expe(X|A) & = \displaystyle\int_{-\infty}^{\infty} x\,f_{X|A}(x|y)\,dy	& \quad\text{if $X$ is continuous.}
%\end{array}
%\]
%\end{definition}
%
%% example: cond_expe for cts. uniform
%\begin{example}
%Let $X\sim\text{Uniform}[0,1]$ and let $A=\{1/2\leq X\leq 3/4\}$. The CDF of $X$ is $F_X(x) = x$ for $0\leq x\leq 1$ (with $F_X(x)=0$ for $x<0$ and $F_X(x)=1$ for $x>1$) so
%\[
%\prob(A) = \prob(1/2\leq x\leq 3/4) = \prob(X\leq 3/4)-\prob(X\leq 1/2) = F_X(3/4) - F_X(1/2) = 3/4 - 1/2 = 1/4.
%\]
%The PDF of $X$ is $f_X(x)=1$ for $0\leq x\leq 1$ (and zero otherwise), so the conditional PDF of $X|A$ is
%\[
%f_{X|A}(x) = \frac{f_X(x)}{\prob(A)} = 4 \text{for $1/2\leq x\leq 3/4$ (and zero otherwise),}
%\]
%which shows that $X|A\sim\text{Uniform}[1/2,3/4]$. The conditional expectation of $X|A$ is
%\[
%\expe(X|A)
%	= \int_{-\infty}{\infty} x f_{X|A}(x)\,dx
%	= \int_{1/2}{3/4} 4x \,dx
%	= 5/8.
%\]
%This is the mid-point of the interval $[1/2,3/4]$ as expected.
%\end{example}

%\begin{example}
%Recall example~\label{example:cond_pmf}: a fair coin is tossed repeatedly until a head occurs; $X$ is the number of times the coin is tossed; $A$ is the event that the coin is tossed an odd number of times. If $A$ occurs, the expcted value of $X$ is
%\[
%\expe(X|A) = \sum_{k\text{odd}} k(3/2^{k+1}) = \sum_{k\text{odd}}^{\infty} 3/2^k  = 3(1/2 + 1/8 + 1/32 + \ldots) = 2.
%\]
%The unconditional expectation is
%\[
%\expe(X) = \sum_{k=1}^{\infty} k(1/2^k) = \sum_{k=1}^{\infty} 1/2^{k-1} = 1 + 1/2 + 1/4 + 1/8 + \ldots = 2.
%\]
%*** so it doesn't change!? ***
%\end{example}

\begin{definition}\label{def:cond_expe_x}
Let $x$ be a fixed value. The \emph{conditional expectation of $Y$ given $X=x$} is
\[
\begin{array}{lll}
\expe(Y|X=x) & = \displaystyle\sum_{j=1}^{\infty} y_j\,f_{Y|X}(y_j|x)		& \quad\text{if $Y$ is discrete, or} \\[4ex]
\expe(Y|X=x) & = \displaystyle\int_{-\infty}^{\infty} y\,f_{Y|X}(y|x)\,dy	& \quad\text{if $Y$ is continuous.}
\end{array}
\]
\end{definition}

%-----------------------------
%\subsection{Conditioning on events}
%\begin{definition}\label{defn:cond_expe_A}
%let $A$ be an event with $\prob(A)>0$, and let $X$ be a random variable. Then
%\[
%\begin{array}{lll}
%\expe(X|A) & = \displaystyle\sum_{i=1}^{\infty} x_i\,f_{X|A}(x_i)		& \quad\text{if $X$ is discrete, or} \\[4ex]
%\expe(X|A) & = \displaystyle\int_{-\infty}^{\infty} x\,f_{X|A}(x|y)\,dy	& \quad\text{if $X$ is continuous.}
%\end{array}
%\]
%\end{definition}

\begin{definition}
Let $A$ be an event with $\prob(A)>0$. The \emph{conditional expectation of $Y|A$} is the conditional expectation of $Y$ given $I_A=1$, where $I_A$ is the indicator variable of $A$.
%%\[
%%\begin{array}{cccl}
%%F_{X|A}:	& \R	& \to		& [0,1] \\
%%			& x		& \mapsto	& \prob(X\leq x\,|\, A).
%%\end{array}
%%\]
%The conditional PMF/PDF of $X|A$ are the functions
%\[
%\begin{array}{lll}
%f_{X|A}(x) & = \prob(\{X=x\}\cap A)/\prob(A)& \quad\text{if $X$ is discrete, or} \\
%f_{X|A}(x) & = f_{X}(x)/\prob(A)			& \quad\text{if $X$ is continuous.}
%\end{array}
%\]
\end{definition}

% example: cond_expe for cts. uniform
\begin{example}
Let $Y\sim\text{Uniform}[0,1]$. Find the conditional expectation of $Y$ given that $1/2\leq Y\leq 3/4$.

\begin{solution}
Let $A=\{1/2\leq X\leq 3/4\}$.

The CDF of $Y$ is $F_Y(y) = y$ for $0\leq y\leq 1$ (with $F_Y(y)=0$ for $y<0$ and $F_Y(y)=1$ for $y>1$) so
\[
\prob(A) = \prob(1/2\leq Y\leq 3/4) = \prob(Y\leq 3/4)-\prob(Y\leq 1/2) = F_Y(3/4) - F_Y(1/2) = 3/4 - 1/2 = 1/4.
\]

The PDF of $Y$ is $f_Y(y)=1$ for $0\leq y\leq 1$ (and zero otherwise), so the conditional PDF of $Y|A$ is
\[
f_{Y|A}(y) = \frac{f_Y(y)}{\prob(A)} = 4 \quad\text{for $1/2\leq x\leq 3/4$ (and zero otherwise),}
\]
which shows that $Y|A\sim\text{Uniform}[1/2,3/4]$. The conditional expectation of $Y|A$ is
\[
\expe(Y|A)
	= \int_{-\infty}^{\infty} y f_{Y|A}(y)\,dy
	= \int_{1/2}^{3/4} 4y \,dy
	= 5/8.
\]
This is the mid-point of the interval $[1/2,3/4]$ as expected.
\end{solution}
\end{example}

%-----------------------------
\subsection{Conditioning on random variables}

For any fixed value of $x$, the conditional expectation $\expe(Y|X=x)$ is just a number. Let us now think of $x$ as a variable quantity, and consider the transformation
\[
\begin{array}{rccl}
	g:	& \R	& \to		& \R \\
		& x		& \mapsto	& \expe(Y|X=x)
\end{array}
\]
This transformation of $X$ yields a new random variable called the \emph{conditional expectation of $Y$ given $X$}.
\begin{definition}\label{def:cond_expe}
The \emph{conditional expectation of $Y|X$} is the random variable
\[\begin{array}{llll}
\expe(Y|X):	& \Omega 	& \to 		& \R \\
			& \omega	& \mapsto 	& \expe\big[Y|X=X(\omega)\big].
\end{array}\]
\end{definition}

The distribution of $\expe(Y|X)$ depends only on the distribution of $X$, and its expectation is given by

\[
\begin{array}{lll}
\expe\big[\expe(Y|X)\big] & = \displaystyle\sum_{i=1}^{\infty}\expe(Y|X=x_i)f_X(x_i)		& \quad\text{if $X$ is discrete, or} \\[3ex]
\expe\big[\expe(Y|X)\big] & = \displaystyle\int_{-\infty}^{\infty} \expe(Y|X=x)f_X(x)\,dx	& \quad\text{if $X$ is continuous.}
\end{array}
\]

\begin{theorem}[Law of total expectation]
Let $X$ and $Y$ be random variables defined on the same probability space. Then
\[
\expe(Y) = \expe\big[\expe(Y|X)\big].
\]
\end{theorem}
\begin{proof}
For discrete random variables (the continuous case is similar):
\begin{align*}
\expe\big[\expe(Y|X)\big]
	= \sum_x \expe(Y|X=x) f_X(x)
	& = \sum_x\left(\sum_y y\,f_{Y|X}(y|x)\right) f_X(x) \\
	& = \sum_x\left(\sum_y y\,\frac{f_{X,Y}(x,y)}{f_X(x)}\right) f_X(x) \\
	& = \sum_x\left(\sum_y y f_{X,Y}(x,y)\right) \\
	& = \sum_x y \left(\sum_y f_{X,Y}(x,y)\right) \\
	& = \sum_x y f_Y(y)
	= \expe(Y).
\end{align*}
%For continuous random variables (the discrete case is similar):
%\begin{align*}
%\expe\big[\expe(Y|X)\big]
%	& = \int_{-\infty}^{\infty} \expe(Y|X=x) f_X(x)\,dx \\
%	& = \int_{-\infty}^{\infty}\left(\int_{-\infty}^{\infty} y\,f_{Y|X}(y|x)\,dy\right) f_X(x)\,dx \\
%	& = \int_{-\infty}^{\infty}\left(\int_{-\infty}^{\infty} y\,\frac{f_{X,Y}(x,y)}{f_X(x)}\,dy\right) f_X(x)\,dx \\
%	& = \int_{-\infty}^{\infty}\left(\int_{-\infty}^{\infty} y f_{X,Y}(x,y)\,dy\right)\,dx \\
%	& = \int_{-\infty}^{\infty} y \left(\int_{-\infty}^{\infty} f_{X,Y}(x,y)\,dx\right)\,dy \\
%	& = \int_{-\infty}^{\infty} y f_Y(y) \,dy \\
%	& = \expe(Y).
%\end{align*}
\end{proof}

\begin{theorem}[Law of total variance]
If $X$ and $Y$ are random variables defined on the same probability space,
\[
\var(Y) = \expe\big[\var(Y|X)\big] + \var\big[\expe(Y|X)\big]
\]
where $\var(Y|X)=\expe(Y^2|X) - \expe(Y|X)^2$.
\end{theorem}
\begin{proof}
By the law of total expectation,
\begin{align*}
\var(Y)
	& = \expe(Y^2) - \expe(Y)^2 \\
	& = \expe\big[\expe(Y^2|X)\big] - \expe\big[\expe(Y|X)\big]^2
\end{align*}
Because $\var(Y|X)=\expe(Y^2|X) - \expe(Y|X)^2$,
\[
\var(Y) = \expe\big[\var(Y|X) + \expe(Y|X)^2\big] - \expe\big[\expe(Y|X)\big]^2
\]
Hence, by the linearity of expectation,
\begin{align*}
\var(Y)
	& = \expe\big[\var(Y|X)\big] + \Big(\expe\big[\expe(Y|X)^2\big] - \expe\big[\expe(Y|X)\big]^2\Big) \\
	& = \expe\big[\var(Y|X)\big] + \var[\expe(Y|X)\big].
\end{align*}
\end{proof}

% example
\begin{example}
Let the joint PDF of the continuous random variables $X$ and $Y$ be
\[
f_{X,Y}(x,y) = \begin{cases}
	cxy & \quad\text{for $x,y\geq 0$ with $x+y\leq 1$}, \\
	0	& \quad\text{otherwise.}
\end{cases}
\]
\ben
\it Sketch the support of $f_{X,Y}$
\it Show that $c=24$.
\it Compute the conditional expectation $\expe(Y|X)$.
\it Verify the identity $\expe\big[\expe(Y|X)\big]=\expe(Y)$.
\een

\begin{solution}
\ben

\it % << (a)
$\supp(f_{X,Y})$ is the lower-left half of the unit square.

\it % << (b)
The marginal PDF of $X$ is
\[
f_X(x) = c\int_{0}^{1-x} xy\,dy = \frac{cx(1-x)^2}{2}
\]
To find $c$,
\[
\int_{0}^{1} f_X(x)\,dx = 1, \qquad\text{so}\quad c=24.
\]
Thus
\begin{align*}
f_X(x)	& = 12x(1-x)^2 \qquad 0\leq x\leq 1 \\
f_Y(y)	& = 12y(1-y)^2 \qquad 0\leq y\leq 1 \\
\end{align*}
and
\begin{align*}
\expe(X) & = 12\int_0^1 x(1-x)^2\,dx = 2/5 \\
\expe(Y) & = 12\int_0^1 y(1-y)^2\,dy = 2/5 \\
\end{align*}

\it % << (c)
To compute $\expe(Y|X)$,
\begin{align*}
\expe(Y|X=x)
	& = \int_0^1 y\left(\frac{f_{X,Y}(x,y)}{f_X(x)}\right)\,dy \\
	& = \int_0^{1-x} y\left(\frac{24xy}{12x(1-x)^2}\right)\,dy \\
	& = \frac{24x}{12x(1-x)^2}\int_0^{1-x} y^2\,dy \\
	& =  \frac{24x}{12x(1-x)^2}\left[\frac{(1-x)^3}{3}\right] \\
	& = \frac{2}{3}(1-x)
\end{align*}
so $\expe(Y|X) = 2(1-X)/3$.

\it % << (d)
\begin{align*}
\expe\big(\expe(Y|X)\big)
	& = \int_0^1 \expe(Y|X=x) f_X(x)\,dx \\
	& = \frac{2}{3}\int_0^1 (1-x) f_X(x)\,dx \\
	& = \frac{2}{3}\big(1-\expe(X)\big) = \frac{2}{3}\left(1-\frac{2}{5}\right) = \frac{2}{5} \\
	& = \expe(Y)
\end{align*}
\een
\mbox{}
\end{solution}
\end{example}

% exercises
\begin{exercise}
\begin{questions}
\question
Let $X$ and $Y$ be jointly continuous random variables having the following joint PDF,
\[
f_{X,Y}(x,y) =
\begin{cases}
	\frac{21}{4}x^2y		& \quad x^2<y<1, \\
	0						& \quad\text{otherwise.}
\end{cases}
\]
\ben
\it Sketch the support of $f_{X,Y}$.
\it Find the marginal PDFs of $X$ and $Y$.
\it Find the mean and variance of $Y$.
\it Find the conditional PDF of $Y$ given $X=x$.
\it Are $X$ and $Y$ independent?
\it Find the conditional expectation of $Y$ given $X=x$.
\it Find the conditional expectation of $Y$ given $X$.
\it Verify that $\expe(Y)=\expe\big[\expe(Y|X)\big]$.
\een

\begin{answer}
\ben
\it % << (a)
The support of the joint PDF $f(x,y)$ is the set $\{(x,y): x^2 < y < 1\}$.  This is the region of the plane between the vertical lines $x=-1$ and $x=+1$, bounded above by the horizontal line $y=1$ and below by parabola $y=x^2$. In particular,
\bit
\it For fixed $x\in[-1,1]$, $f_{X,Y}(x,y)\neq 0$ only for $y\in[x^2,1]$.
\it For fixed $y\in[0,1]$, $f_{X,Y}(x,y)\neq 0$ only for $x\in[-\sqrt{y},+\sqrt{y}]$.
\eit

\it % << (b)
The marginal distributions are computed as follows:
\begin{align*}
f_X(x)
	& = \int_{-\infty}^{\infty} f(x,y)\,dy
	= \int_{x^2}^1 \frac{21}{4}x^2y\,dy
	= \frac{21}{4}x^2\left[\frac{y^2}{2}\right]_{x^2}^1
	= \begin{cases} \frac{21}{8}x^2(1-x^4)	& -1<x<1, \\ 0 & \text{ otherwise.}\end{cases} \\ [2ex]
f_Y(y)
	& = \int_{-\infty}^{\infty} f(x,y)\,dx
	= \int_{-\sqrt{y}}^{\sqrt{y}} \frac{21}{4}x^2y\,dx
	= \frac{21}{4}y\left[\frac{x^3}{3}\right]_{-\sqrt{y}}^{\sqrt{y}}
	= \begin{cases} \frac{7}{2}y^{5/2} & 0< y< 1, \\ 0 & \text{ otherwise.}\end{cases} \\
\end{align*}

\it % << (c)
The expected value and variance of $Y$ are computed as follows:
\begin{align*}
\expe(Y)
	& = \int_{-\infty}^{\infty} y\,f_Y(y)\,dy 	= \int_0^1 y\left(\frac{7y^{5/2}}{2}\right)\,dy = \frac{7}{9}, \\
\expe(Y^2)
	& = \int_{-\infty}^{\infty} y^2\,f_Y(y)\,dy = \int_0^1 y^2\left(\frac{7y^{5/2}}{2}\right)\,dy = \frac{7}{11}, \\
\var(Y)
	& = \expe(Y^2) - \expe(Y)^2 = \frac{7}{11} - \frac{49}{81} = \frac{28}{891}.
\end{align*}

\it % << (d)
The conditional PDF of $Y$ given $X=x$ is
\[
f_{Y|X=x}(y)
	= \frac{f_{X,Y}(x,y)}{f_X(x)}
	= \frac{(21/4)x^2y}{(21/8)x^2(1-x^4)}
	= \begin{cases}
		\displaystyle\frac{2y}{1-x^4} 	& x^2\leq y\leq 1, \\
	 	0 								& \text{ otherwise}.
	 \end{cases}
\]

\it % << (e)
$X$ and $Y$ are clearly not independent, because the support of $f_{X,Y}$ is not a rectangular region, and moreover, the conditional PDF of $Y$ given $X=x$ depends on $x$.

\it % << (f)
The conditional expected value of $Y$ given that $X=x$ is
\begin{align*}
\expe(Y|X=x)
	& = \int_{-\infty}^{\infty} y f_{Y|X=x}(y\,|\,x)\,dy \\
	& = \int_{x^2}^1 y \left(\frac{2y}{1-x^4}\right)\,dy
	= \frac{2}{1-x^4}\left[\frac{y^3}{3}\right]_{x^2}^1
	= \frac{2(1-x^6)}{3(1-x^4)}
\end{align*}

\it % << (g)
The conditional expectation of $Y$ given $X$ is the random variable
\[
\expe(Y|X)=\displaystyle\frac{2(1-X^6)}{3(1-X^4)}
\]

\it % << (h)
The expected value of $\expe(Y|X)$ is
\begin{align*}
\expe\big[\expe(Y|X)\big]
	& = \int_{-\infty}^{\infty} \expe(Y|X=x)f_X(x)\,dx \\
	& = \frac{2}{3}\int_{-1}^{1} \left(\frac{1-x^6}{1-x^4}\right)\left(\frac{21}{8}x^2(1-x^4)\right)\,dx \\
	& = \frac{7}{4}\int_{-1}^{1} x^2(1-x^6)\,dx
	= \frac{7}{9}.
\end{align*}
Thus $\expe\big[\expe(Y|X)\big]=\expe(Y)$ as required.
\een
\end{answer}

% house for sale
\question
A man puts his house for sale, and decides to accept the first offer that exceeds the reserve price of $\pounds r$. Let $X_1,X_2,\ldots$ represent the sequence of offers received, and suppose that the $X_i$ are independent and identically distributed random variables, each having exponential distribution with rate parameter $\lambda$.
\begin{parts}
\part
What is the expected number of offers received before the house is sold?% is $e^{\lambda r}$.
\begin{answer}
Let $N$ be the number of offers received before the house is sold. Then $\{N=k\}$ is the event that the first $k-1$ offers are at most $\pounds r$, each occurring independently with probability $F(r)$, and the $k$th offer exceeds $\pounds r$, which occurs with probability $1-F(r)$. Thus $N$ has \emph{geometric} distribution, with `probability of success' equal to $1-F(r)$ (where `success' corresponds to the sale of the house). Hence the expected number of offers received before the house is sold is
\[
\expe{E}(N) = \frac{1}{1-F(r)} = e^{\lambda r}.
\]
\end{answer}
\part
What is the expected selling price of the house?
\begin{answer}
Let $F_S$ be the conditional CDF of $X_i$ given that $X_i>r$:
\[
F_S(x)
	= \frac{\mathbb{P}(r < X_i \leq x)}{\mathbb{P}(X_i > r)}
	= \frac{F(x)-F(r)}{1-F(r)}
	= \begin{cases}
		1 - e^{-\lambda(x-r)} 	& x > r, \\
		0						& \text{otherwise.}
	\end{cases}
\]
A straightforward calculation yields $\mathbb{E}(X_i | X_i>r) = r + \displaystyle\frac{1}{\lambda}$.
\end{answer}
\end{parts}

\end{questions}
\end{exercise}


\documentclass[letterpaper]{article}
\title{\textbf{Tokio university's entrance exam 2017}}
\author{Acosta Domínguez Jorge}
\date{}

\usepackage{amsmath}
\usepackage{tabto}
\usepackage{tasks}
\usepackage{geometry} %this reduces the amount of margin
\newcommand{\itab}[0]{\hspace{2em}}

\begin{document}
\pagenumbering{gobble}
\maketitle
\newpage
\pagenumbering{arabic}

\section{Mathmatics exam}
\subsection{Lineal algebra}

Suppose that three-dimensional vectors $\left(
\begin{matrix}
x_n\\y_n\\z_n\\
\end{matrix}\\
\right)$
satisfy the equation
\begin{equation*}
\left(
\begin{matrix}
x_{n+1}\\y_{n+1}\\z_{n+1}
\end{matrix}
\right)
=A\left(
\begin{matrix}
x_n\\y_n\\z_n\\
\end{matrix}\\
\right),\itab n = 0, 1, 2, ...
\end{equation*}
where $x_0$, $y_0$, $z_0$ and $\alpha$ are real numbers, and
\begin{equation*}
A = \left(\begin{matrix}
1-2\alpha & \alpha & \alpha\\
\alpha & 1-\alpha & 0\\
\alpha & 0 & 1-\alpha
\end{matrix}\right),\itab0<\alpha<\frac{1}{3}.
\end{equation*}
Answer the following questions.
\settasks{
	counter-format=(tsk),
	label-width=3ex
}
\begin{tasks}
\task{Express $x_n + y_n + z_n$ using $x_0, y_0$ and $z_0$}
\task{Obtain the eigenvalues $\lambda_1, \lambda_2$ and $\lambda_3$, and their corresponding eigenvectors $v_1, v_2, v_3$ of the matrix $A$}
\task{Express the matrix $A$ using $\lambda_1, \lambda_2, \lambda_3, v_1, v_2, v_3$.}
\task{Express $\left(\begin{matrix}
x_n\\y_n\\z_n
\end{matrix}\right)$}
using $x_0, y_0, z_0$ and $\alpha$.
\task{Obtain $\lim_{n\to\infty}
\left(\begin{matrix}
x_n\\y_n\\z_n
\end{matrix}\right)$}.
\task{Regard
\begin{equation*}
f(x_0, y_0, z_0) = \frac{(x_n, y_n, z_n)\left(\begin{matrix}x_{n+1}\\y_{n+1}\\z_{n+1}\end{matrix}\right)}{(x_n, y_n, z_n)\left(\begin{matrix}x_n\\y_n\\z_n\end{matrix}\right)}
\end{equation*}
as a function of $x_0, y_0$ and $z_0$. Obtain the maximum and the minimum values of $f(x_0, y_0, z_0)$, where we assume that $x_0^2 + y_0^2 + z_0^2 \neq 0$.}
\end{tasks}\newpage

\subsubsection{Resolution}
\begin{tasks}
\task{
Using $n=0$
\begin{align*}
x_1 &= (1-2\alpha)x_0+\alpha y_0 + \alpha z_0\\
y_1 &= \alpha x_0 + (1-\alpha)y_0\\
z_1 &= \alpha x_0 + (1-\alpha)z_0
\end{align*}
adding both sides of the equations:
\begin{align*}
x_1 + y_1 + z_1 &= (1-2\alpha)x_0 + \alpha x_0 + \alpha x_0 + \alpha y_0 + (1-\alpha) y_0 + \alpha z_0 + (1-\alpha)z_0\\
x_1 + y_1 + z_1 &= x_0(\alpha + \alpha -2\alpha + 1) + y_0(\alpha + 1 -\alpha) + y_0(\alpha + 1 -\alpha)\\
x_1 + y_1 + z_1 &=x_0 + y_0 + z_0
\end{align*}
due that with $n=1$ we get $x_1 + y_1 + z_1 =x_0 + y_0 + z_0$ we can say that:
$$x_n + y_n + z_n=x_0 + y_0 + z_0$$
}
\task{
For this part we have to recall that $|A-I\lambda|=0$. So, we substract $I\lambda$ from our A to get this:
\begin{equation*}
\left|\begin{matrix}
1-2\alpha-\lambda & \alpha & \alpha\\
\alpha & 1-\alpha-\lambda & 0\\
\alpha & 0 & 1-\alpha-\lambda
\end{matrix}\right| = 0
\end{equation*}
By getting the determinant of $|A-I\lambda|$ we get $(\lambda+1)(\lambda-1+3\alpha)(\lambda-1+\alpha)=0$
So the values of $\lambda$ are:
\begin{itemize}
\item{$\lambda_3 = 1-\alpha$}
\item{$\lambda_2 = 1-3\alpha$}
\item{$\lambda_1 = 1$}
\end{itemize}
Now, substituting each of the $\lambda$ values in $A-I\lambda$ and resolving the equations system we get the following vectors:\\
For $\lambda = 1-\alpha$
\begin{equation*}
\left(\begin{matrix}
1-2\alpha-(1-\alpha) & \alpha & \alpha\\
\alpha & 1-\alpha-(1-\alpha) & 0\\
\alpha & 0 & 1-\alpha-(1-\alpha)
\end{matrix}\right)
,\itab v_1 = \left(\begin{matrix}
0\\-1\\1
\end{matrix}\right)
\end{equation*}
For $\lambda = 1-3\alpha$
\begin{equation*}
\left(\begin{matrix}
1-2\alpha-(1-3\alpha) & \alpha & \alpha\\
\alpha & 1-\alpha-(1-3\alpha) & 0\\
\alpha & 0 & 1-\alpha-(1-3\alpha)
\end{matrix}\right)
,\itab v_2 = \left(\begin{matrix}
-2\\1\\1
\end{matrix}\right)
\end{equation*}
For $\lambda = 1$
\begin{equation*}
\left(\begin{matrix}
1-2\alpha-1 & \alpha & \alpha\\
\alpha & 1-\alpha-1 & 0\\
\alpha & 0 & 1-\alpha-1
\end{matrix}\right)
,\itab \itab \itab \itab \itab v_3 = \left(\begin{matrix}
1\\1\\1
\end{matrix}\right)
\end{equation*}
}
\task{
Since we can decompose any matrix $A$ in $A=Q\Lambda Q^{-1}$ where $\Lambda$ is the diagonal matrix made of the eigenvalues that we found, and $Q$ is the the matrix made of eigenvectors that we already found. We simply have to build the Q matrix and get its inverse.\\
We have that $Q=\left(\begin{matrix}
0 & -2 & 1\\
-1 & 1 & 1\\
1 & 1 & 1
\end{matrix}\right)$, with this we only have to find $Q^{-1}$.\\
Altought there are differents ways to get the inverse of a matrix the used for this exercise is the gauss-Jordan one. Solving we have:\\
$$\frac{1}{6}\left(\begin{matrix}
0 & -3 & 3\\
-2 & 1 & 1\\
2 & 2 & 2
\end{matrix}\right)$$
So this way we could represent $A$ using its eigenvalues and its eigenvectors\\
\begin{equation*}
A =
\frac{1}{6}
\left(\begin{matrix}
0 & -2 & 1\\
-1 & 1 & 1\\
1 & 1 & 1
\end{matrix}\right)
\left(\begin{matrix}
1-\alpha & 0 & 0\\
0 & 1-3\alpha & 0\\
0 & 0 & 1
\end{matrix}\right)
\left(\begin{matrix}
0 & -3 & 3\\
-2 & 1 & 1\\
2 & 2 & 2
\end{matrix}\right)
\end{equation*}
}
\task{
For fast analisys lets rename $\left(\begin{matrix}
x_n\\y_n\\z_n
\end{matrix}\right)$ as $V_n$.\\
This way we have that \begin{equation}
V_{n+1} = AV_n
\end{equation}
If we set $n=0$ we have $V_1=AV_0$. With this we can deduce that \begin{equation}
V_n = A^nV_0
\end{equation}
If we apply it to $v_{n+1}$ we have that $V_{n+1} = AV_n=A(A^nV_0)=A^{n+1}V_0$. This way we can prove the validity of our deduction.
So, to express $x_n + y_n + z_n$ using $x_0, y_0$ and $z_0$ we first need to find the value of $A^n$\\
Fortunately we know that \begin{equation}
A^n = Q\Lambda^nQ^{-1}
\end{equation}.
$$
A^n =
\frac{1}{6}
\left(\begin{matrix}
0 & -2 & 1\\
-1 & 1 & 1\\
1 & 1 & 1
\end{matrix}\right)
\left(\begin{matrix}
1-\alpha & 0 & 0\\
0 & 1-3\alpha & 0\\
0 & 0 & 1
\end{matrix}\right)^n
\left(\begin{matrix}
0 & -3 & 3\\
-2 & 1 & 1\\
2 & 2 & 2
\end{matrix}\right)$$
$$
A^n =
\frac{1}{6}
\left(\begin{matrix}
0 & -2 & 1\\
-1 & 1 & 1\\
1 & 1 & 1
\end{matrix}\right)
\left(\begin{matrix}
(1-\alpha)^n & 0 & 0\\
0 & (1-3\alpha)^n & 0\\
0 & 0 & 1
\end{matrix}\right)
\left(\begin{matrix}
0 & -3 & 3\\
-2 & 1 & 1\\
2 & 2 & 2
\end{matrix}\right)
$$
$$
A^n = \frac{1}{6}
\left(\begin{matrix}
4(1-3\alpha)^n+2 & -2(1-3\alpha)^n+2 & -2(1-3\alpha)^n+2\\
-2(1-3\alpha)^n+2 & 3(1-\alpha)^n+(1-3\alpha)^n+2 & -3(1-\alpha)^n+(1-3\alpha)^n+2\\
-2(1-3\alpha)^n+2 & -3(1-\alpha)^n+(1-3\alpha)^n+2 & 3(1-\alpha)^n+(1-3\alpha)^n+2
\end{matrix}\right)
$$
So, if we replace this in $3$ we have:
$$
V_n = \frac{1}{6}
\left(\begin{matrix}
4(1-3\alpha)^n+2 & -2(1-3\alpha)^n+2 & -2(1-3\alpha)^n+2\\
-2(1-3\alpha)^n+2 & 3(1-\alpha)^n+(1-3\alpha)^n+2 & -3(1-\alpha)^n+(1-3\alpha)^n+2\\
-2(1-3\alpha)^n+2 & -3(1-\alpha)^n+(1-3\alpha)^n+2 & 3(1-\alpha)^n+(1-3\alpha)^n+2
\end{matrix}\right)V_0
$$
Substituying $V_n$ to its original form and multiplying we get the following:
\begin{displaymath}
\left(\begin{matrix}x_n\\y_n\\z_n\end{matrix}\right)=
\frac{1}{6} \left(\begin{matrix}
4(1-3\alpha)^n+2 & -2(1-3\alpha)^n+2 & -2(1-3\alpha)^n+2\\
-2(1-3\alpha)^n+2 & 3(1-\alpha)^n+(1-3\alpha)^n+2 & -3(1-\alpha)^n+(1-3\alpha)^n+2\\
-2(1-3\alpha)^n+2 & -3(1-\alpha)^n+(1-3\alpha)^n+2 & 3(1-\alpha)^n+(1-3\alpha)^n+2
\end{matrix}\right)\left(\begin{matrix}x_0\\y_0\\z_0\end{matrix}\right)
\end{displaymath}
\begin{align*}
x_n &= \frac{1}{3}((x_0+y_0+z_0)+(1-3\alpha)^n(x_0-y_0-z_0))\\
y_n &= \frac{1}{3}(x_0+y_0+z_0)+\frac{1}{6}(1-3\alpha)^n(-2x_0+y_0+z_0)+\frac{1}{2}(1-\alpha)^n(y_0-z_0)\\
z_n &= \frac{1}{3}(x_0+y_0+z_0)+\frac{1}{6}(1-3\alpha)^n(-2x_0+y_0+z_0)+\frac{1}{2}(1-\alpha)^n(-y_0+z_0)
\end{align*}
}
\task{
Considering the equations above, and the condition that $0<\alpha<\frac{1}{3}$, we can write the $\lim_{n\to\infty}\left( \begin{matrix}
x_n\\y_n\\z_n
\end{matrix} \right)$ as
\begin{align*}
\frac{1}{3}(x_0+y_0+z_0) &< x_n < \frac{2}{3}(2x_0-y_0-z_0)\\
\frac{5}{6}y_0 &< y_n < \frac{1}{3}(x_0+y_0+z_0)\\
\frac{5}{6}z_0 &< z_n < \frac{1}{3}(x_0+y_0+z_0)
\end{align*}
}
\task{
Mental note: "Too much work to do for only 2 and half hours!"
}
\end{tasks}

\newpage
\subsection{Differential equations}
A real valued function $u(x, t)$ is defined in $0 \leq x \leq 1$ and $t \geq 0$. Here $x$ and $t$ are independent.\\
Suppose solving the following partial differential equation:
\begin{equation}
\frac{\partial u}{\partial t} = \frac{\partial^2 u}{\partial x^2}
\end{equation}
under the following conditions:
\begin{center}
Boundary conditions:\itab $u(0, t) = u(1, t) = 0$\\
Initial condition:\itab $u(x, 0) = x-x^2$
\end{center}
Since the constant function $u(x, t) = 0$ is obvious a solution of the partial differential equation, consider the other solutions. Answer the following questions.
\settasks{
	counter-format=(tsk),
	label-width=3ex
}
\begin{tasks}
\task{
Calculate the following expression, where $n$ and $m$ are positive integers.
$$
\int_0^1{sin(n\pi x) sin(m\pi x) dx}
$$
}
\task{
Supose $u(x, t ) = \epsilon(x) \rho(t)$ where $\epsilon(x)$ is a function only of $x$ and $\rho(t)$ is a function only of $t$. Express the ordinary differential equations for $\epsilon$ and $\rho$ using an arbitrary constant $C$. You may use that $f(x)$ and $g(t)$ are constant functions where $f(x)$ and $g(t)$ satisfy $f(x) = g(t)$ for arbitrary $x$ and $t$.
}
\end{tasks}
\newpage
\subsubsection{Resolution}
\begin{tasks}
\task{
First at all, we know that $\sin(x)\sin(y) = \frac{1}{2}[\cos(x-y)-\cos(x+y)]$. So we can rewrite the expression as follows:
\begin{equation*}
=\frac{1}{2}\left[\int_0^1{\cos(\pi x(n-m))}dx -\cos(\pi x(n+m))dx \right]
\end{equation*}
Now this is easily integrable by variable change. Applying it an resolving we get:
\begin{equation*}
=\frac{1}{2}\pi\left( \right)
\end{equation*}
}
\end{tasks}
\end{document}


\documentclass{article}

\title{Regularization techniques for Deep Learning}
\author{Vladimir Feinberg}

\input{../old/defs}

\begin{document}

\maketitle

\section{Regularization}

Regularization shrinks the capacity of an ANN. In other words, it restricts the richness of the family of functions which the ANN can take the value of. As Vincent Vanhouke puts it in his \nurl{https://www.udacity.com/course/deep-learning--ud730}{Udacity course}, ANN regularization can be likened to fitting into tight jeans, where extremely expressive model classes are tightened by generic regularization techniques, which seem to find a ``right fit'' in many problems, empirically speaking.

At a high level, regularization changes the objective from usual loss $J(\vtheta)$ to loss plus weight penalty $J(\vtheta)+\Omega(\vtheta)$ or loss plus constraints. In both supervised and unsupervised models this has justification in moving overfit models of high capacity to those of appropriate capacity for the problem: lower variance for a bit higher bias, where the bias and variance are with respect to the loss $J$ on the training set as an estimator for the loss on the entire data distribution.

In the sections that follow, we recount general regularization techniques for ANNs, most of which were collected from Dr. Goodfellow's \nurl{http://www.deeplearningbook.org/contents/mlp.html}{Deep Learning Book}.

\section{Weight Decay}

For all affine transforms let the weight matrix terms (not bias), vectorized, be $\vw$. Add $\Omega(\vtheta)=\frac{\alpha}{2}\norm{\vw}_2^2$. Intuitively, bias is unrestricted because it only interacts with one variable directly (the output feature map), so it isn't prone to overfitting. For a quadratic approximation of the loss at $J(\vtheta)$, with hessian $H=Q\Lambda Q^\top$, $L^2$ regularization would result in a new optimum $Q(\Lambda+\alpha I)^{-1}\Lambda Q^\top \vw_*$ where $\vw_*$ is the unconstrained optimum. If $\vv_i$ is the $i$-th eigenvector of $H$ and $\lambda_i$ its eigenvalue, then the regularization adjusts the solution by rescaling the $\vv_i$-th component of $\vw_*$ by a factor of $\frac{\lambda_i}{\lambda_i+\alpha}$. Regularization can also be viewed as constraints by identifying that $\alpha$ is a KKT multiplier: a fixed $\alpha$ corresponds to optimizing $\min J(\vtheta)$ such that $\norm{\vw}\le k$ for some $k$, which is in practice difficult to find given $\alpha$ (Fig.~\ref{fig:reg}). $L^2$ regularization is equivalent to Bayesian MAP if weights have $N(0,\alpha^{-1})$ prior.
\begin{figure}[!h]
\centering
{\includegraphics[width=0.75\textwidth]{reg.pdf}}
  \caption{Regularization as constrained optimization: the above shows the contour plots of a quadratic objective, and $L^2,L^1$ regularizations, respectively, as constraints constructed by weighing the objective against the shape of the regularizer's ball. Image from Bishop's Pattern Recognition and Machine Learning, Chapter 3, page 146.}
\label{fig:reg}
\end{figure}

\textbf{Proximal Regularization}. Similar to weight decay, but $\Omega(\vtheta)=\alpha\norm{\vw}_1$. Assuming a (convex) diagonal Hessian approximation to $J(\vtheta)$ (more strict than above), the $i$-th component solution is pulled down to
$$w_i=\begin{cases}
\pa{\abs{w_i}-\frac{\alpha}{H_{i,i}}}\sgn w_i  & H_{i,i}>0, \abs{w_i}>\frac{\alpha}{H_{i,i}}\\
  0 & \text{otherwise}
\end{cases}$$
$L^1$ regularization is equivalent to Bayesian MAP if weights have $\Laplace(0, \alpha^{-1})$ prior.

\textbf{Early Stopping}. Stop training if validation increases; even if training is unconverged. If validation set is large, you miss out on a lot of data. More complicated algorithms are necessary to find an applicable early stopping time when re-training to include validation set (see \nurl{http://www.deeplearningbook.org/contents/regularization.html}{Dr. Goodfellow's DL book}, Algorithm 7.3). Early stopping is similar to $L^2$ regularization, and for quadratic loss it is just that (Fig.~\ref{fig:earlystop}).
\begin{figure}[!h]
\centering
{\includegraphics[width=0.75\textwidth]{early-stop.pdf}}
  \caption{From Dr. Goodfellow's DL book lecture, Chapter 7.}
\label{fig:earlystop}
\end{figure}

Early stopping can thus be viewed as an efficient way of regularizing weights just the right amount for the problem at hand, whereas we might have to cross-validate multiple settings of $\alpha$ to find an appropriate value for the $\alpha$ hyperparameter. Nonetheless, early stopping still suffers from generalization issues of overfitting to the validation set, see discussions in \nurl{http://www.tandfonline.com/doi/abs/10.1080/00207179508921605}{Sj\"{o}berg and Ljung 1995} and \nurl{http://www.mitpressjournals.org/doi/abs/10.1162/089976699300016557}{Cataltepe et al 1999}.

\section{Dataset Augmentation and Manifold Learning}

Modify supervised pairs $(\vx, y)$ to include $(T(\vx), y)$ where $T$ is a transformation we expect the classifier to be invariant to. Noise injection into inputs, too can be a form of dataset augmentation. An automatic technique for doing this is tangent propagation (\nurl{https://papers.nips.cc/paper/4409-the-manifold-tangent-classifier}{Rifai et al 2011}), which adds a penalty if the output is sensitive to changes along learned manifold vectors $\vv_i$: $\Omega(f)=\sum_i\pa{\vv_i^\top\nabla f)^2}$.

\section{Noise Robustness}

(\nurl{https://papers.nips.cc/paper/4329-practical-variational-inference-for-neural-networks}{Graves 2011}). Inject noise into weights during training, forcing learned weights to be robust to small variations.

\section{Parameter Sharing}

Reuse part of a network for another task, encoding prior knowledge that the feature maps must be the same between two tasks. CNNs, discussed in a separate document (\nurl{https://github.com/vlad17/ml-notes/tree/master/deep-learning}{root}), are a form of parameter sharing, since the linear matrix $W$ is restricted to a convolution transform.

\section{Ensemble Methods}

Bagging uses votes of several models: $k$ uncorrelated models will tend to have $\nicefrac{1}{k}$-th generalization error when bagged. Training data can be resampled to create more uncorrelated models.

\textbf{Dropout} (\nurl{https://arxiv.org/abs/1207.0580}{Hinton et al 2012}, \nurl{http://proceedings.mlr.press/v28/wang13a.html}{Wang et al 2013}, \nurl{http://jmlr.org/papers/v15/srivastava14a.html}{Srivastava 2014}, \nurl{https://arxiv.org/abs/1506.02142}{Gal et al 2015}). Dropout is a cheap, practical approximation to averaging exponentially many models (exponential in the number of dropout layers). Dropout at a certain hidden or input layer wp $p$ drops an activation by setting it to zero during training and multiplying all activations (or the weights) by $p$ with no drops during testing (the weight scaling inference rule); theoretically this does not guarantee a proper output activation averaging, only individual hidden unit output expectation is preserved, but this works well in practice (Fig.~\ref{fig:dropout}). Weight scaling actually corresponds to taking a geometric means of the votes, and has been showed to work better than Monte Carlo sub-network averaging (which is intractable). First, this prevents feature co-adaptation: since a given activation might vanish at any time during training, the net is forced to learn those activations at each layer which independently contribute to the loss minimization task. Note that this conflicts with locality-focused tasks which rely on specific co-adaptations, like convolutional layers, but might be used at higher levels in CNNs to force the networks to identify more than one underlying feature for their classification (though lower levels can be locally co-adapted). Naive Bayes is an extreme form of dropout which demonstrates the co-adaptation avoidance.
\begin{figure}[!h]
\centering
{\includegraphics[width=0.75\textwidth]{dropout.pdf}}
  \caption{From Dr. Goodfellow's DL book lecture, Chapter 7.}
\label{fig:dropout}
\end{figure}

\textbf{Dropconnect} (\nurl{http://cs.nyu.edu/~wanli/dropc/}{Li et al 2013}). Generalizes dropout by applying it to edges rather than hidden units. It's not as commonly used as dropout, but it has a more mathematically sound explanation of how its inference procedure mimics ensembles.

\section{Adversarial Training}

(\nurl{https://arxiv.org/abs/1412.6572}{Goodfellow 2014}). Tiny changes in input can result in large differences in output, showing that the learned manifold for a classification category is too sensitive to perturbations in directions that it shouldn't be. Adversarial training forces nets to overcome this.

\end{document}

% LocalWords: Vanhouke


\section*{Exercise 20.3-1}
\subsection*{Modify vEB trees to support duplicate keys}

Instead of the tree holding bits, it could instead hold integers. These integers would then act as counters for the number of instances of each key.
\\
Furthermore, to support this change, the element stored in $min$ should appear in the clusters (like $max$), and hold an integer like the rest of the tree.
\\
When inserting and deleting in the tree, the counter should then be updated, by adding to or subtracting from the counter.


%
% This section describes load balancing and the methods offered by Zoltan.
%
\chapter{Data partitioning and load balancing}
\label{cha:lb}
As problem sizes grow, parallel computing has become an important tool
in computational science. Many large-scale computing tasks are
today run on parallel computers, with multiple processors or cores.
An important issue is then how to best divide the data and work
among processes (processors). This problem is known as \emph{data partitioning}
or \emph{load balancing}. We will use these phrases interchangably.
Partitioning or load balancing can either be performed once
(static partitioning) or multiple times (dynamic load balancing).

We wish to divide work evenly but at the same time minimize
communication among processes. Communication could either be
message passing (on distributed memory systems) or memory access
(on shared-memory systems).

We give a brief overview of the different categories of partitioning methods,
and later explain how to use them in Zoltan.

\section{Geometric methods}
Geometric methods rely on each object having a set of coordinates.
Data objects are partitioned based on geometric locality.
We assume that it is beneficial to keep objects that are close together on
the same processor, but there is no explicit model of communication.
Examples of geometric methods include recursive coordinate bisection (RCB)
and space-filling curves. An advantage of geometric methods is that
they are very fast to compute, but communication volume in
the application may be high.

\section{Graph partitioning}
Graph partitioning is perhaps the most popular method. This approach is
based on representing the application
(computation) as a graph, where data objects are vertices and
data dependencies are edges. The graph partitioning problem is then
to partition the vertices into equal-sized sets, while minimizing the
number of edges with endpoints in different sets (parts).
This is an NP-hard optimization problem, but fast multilevel
algorithms and software produce good solutions in practice.
In general, graph partitioning produces better quality partitions
than geometric methods but also take longer to compute.

\section{Hypergraph partitioning}
The graph model has several deficiencies. First, only symmetric
relations between pairs of objects can be represented. Second,
the communication model is inaccurate and does not model
communication volume correctly.
Hypergraphs generalize graphs, but can represent relationships
among arbitrary sets of objects (not just pairs). Also,
communication volume is exact, so hypergraph methods
produce very high quality partitions. The main drawback of hypergraph
methods is that they take longer to run than graph algorithms.

\section{Methods in Zoltan}
Zoltan is a toolkit containing many load balancing methods. We explain how
to use Zoltan in the next chapter. The methods currently available
in Zoltan (version 3.1) are:
\begin{description}
\item[Simple:] BLOCK, RANDOM. These are intended for testing, not real use.
\item[Geometric:] RCB, RIB, HSFC.
\item[Graph:] Zoltan has a native graph partitioner, and optionlly supports ParMetis and PT-Scotch.
\item[Hypergraph:] Zoltan has a native parallel hypergraph partitioner.
\end{description}


\setlength\abovedisplayskip{2.5pt}

For relevant nuclear forensics predictions, both classification and regression
algorithms must be used.  For example, one may want to predict the reactor type
label given some measurement-based features of \gls{SNF} of an unknown source.
This would require a classification algorithm. Or perhaps the input fuel
composition is relevant to an investigation on weapons intent, so a regression
algorithm would be used to train a model based on some different set of
measured features. Since this work trains models to predict burnup of
\gls{SNF}, the algorithms are presented in a regression context.

\subsubsection{Linear Models}
\label{sec:linear}

One of the simplest and most utilized methods of prediction is a linear model
from a least-squares fit. Thus, it is the most natural place to begin a
demonstration of \gls{ML} algorithms. Since linear models must have all
linearly related parameters, there are many restrictions regarding the shape of
the model. However, this makes the resulting models stable to peturbations.
\cite{changingml}

\begin{figure}[!htb]
  \makebox[\textwidth][c]{\includegraphics[width=1.1\textwidth]{./chapters/litrev/regularization.png}}
  \caption{Effect of Regularization on Prediction}
  \label{fig:reg}
\end{figure}

An example of a linear model is in Equation \ref{eq:linreg}. The vector of
input features of size $p$, $\boldsymbol{X}$, provides a model,
$F(\boldsymbol{X})$ by determining the unknown coefficients, or weights,
$\beta_{j}$'s. A loss function can be calculated by the difference in the
model-predicted value and $\boldsymbol{Y}$, the vector of actual labels.  The
algorithm calculates $\beta_{j}$ by minimizing the value of this loss function
over all the training data.  This is usually the least squares error from
minimizing the sum of squared errors, $\sum_{i=1}^{n} (y_i - f(x_i))^2$.  But
it could instead be the least absolute deviations from minimizing the sum of
absolute error differences, $\sum_{i=1}^{n} |y_i - f(x_i)|$. These are referred
to as the $L_2$ and $L_1$ norms, respectively.
\begin{equation}
  F(\boldsymbol{X}) = \beta_{0} +  \sum_{j=1}^{p} x_{j} \beta_{j}
  \label{eq:linreg}
\end{equation}

The form of linear regression used here is called ridge regression. This
algorithm performs optimization using the $L_2$ norm, and also uses the form of
the $L_2$ norm for \textit{regularization}. Regularization, sometimes called
\textit{shrinkage}, is a term introduced into the \gls{ML} model to prevent
overfitting; it is used in many \gls{ML} algorithms. For ridge
regression, shrinkage further reduces the weights, $\beta_j$, on the input
features, $x_j$. The shrinkage term also includes a complexity parameter,
$\lambda$, which governs the strength to which regularization is performed
\cite{elements_stats}. Thus, the linear model given from ridge regression
is updated to Equation \ref{eq:ridgereg}.  Figure \ref{fig:reg} is a
visualization of how applying regularization smooths out a model.
\begin{equation}
  F(\boldsymbol{X}) = \beta_{0} +  \sum_{j=1}^{p} x_{j} \beta_{j} + \lambda \sum_{j=1}^{p} \beta_{j}^2
  \label{eq:ridgereg}
\end{equation}

\subsubsection{Nearest Neighbor Methods}
\label{sec:neighbor}

Nearest neighbor regression is a unique algorithm in that it is instance-based;
it does not actually generalize, but tracks the observations in the training
set.  The main metric for this algorithm is distance (or dissimilarity) between
the test sample and the closest training sample(s) in the vicinity.  During
prediction, the algorithm will calculate a value based on the instance that is
closest to the current test sample. Thus, there is not any learning, but
instead a direct comparison between an unknown sample and the space that the
training set populates. The predictions from nearest neighbors can be quite
accurate, but are highly unstable to peturbations \cite{elements_stats}.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.8\linewidth]{./chapters/litrev/nn-fig.png}
  \caption{Schematic of \textit{k}-Nearest Neighbors Regression}
  \label{fig:nn}
\end{figure}

An extension of nearest neighbor is \textit{k}-nearest neighbor regression.
The closest \textit{k} neighbors are averaged for an estimate of the unknown
sample, as shown in Equation \ref{eq:knn}.  Figure \ref{fig:nn} provides a
pictoral explanation of how this is done for a prediction of a single feature.
For \textit{k}-neighbors, this algorithm predicts a value, $Y$, from the input
features, $\boldsymbol{X}$, in the neighborhood, $N_k (\boldsymbol{X})$
\cite{elements_stats}.
\begin{equation}
  Y(\boldsymbol{X}) = \frac{1}{k} \sum_{x_i \in N_k(\boldsymbol{X})} y_i
  \label{eq:knn}
\end{equation}

There are two tuneable parameters in this algorithm: the distance metric and
the value of \textit{k}.  The population of the neighborhood, \textit{k},
affects the number of points being averaged together for a prediction.  The
metrics for distance differ, but in this study, the Euclidian distance was
used. In this initial work, $k = 1$ is used. This can perform very well, but
can also easily overfit the data and thus not generalize well.

\subsubsection{Support Vector Machines}
\label{sec:svm}

\Gls{SVR} is an extension of the popular classification algorithm,
\glsreset{SVM}.  This algorithm was chosen because of its ability to handle
highly dimensional data well, which in this study is a maximum of approximately
300 features.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=0.8\linewidth]{./chapters/litrev/svm.png}
  \caption{Schematic of \acrshort{SVM} Classification}
  \label{fig:svm}
\end{figure}

As seen in Figure \ref{fig:svm}\footnote{This schematic is based on the
tutorial on SVMs by Dr.\@ Saed Sayad at
http://www.saedsayad.com/support\_vector\_machine.htm}, the \gls{SVM} algorithm
separates two classes by determining an optimal hyperplane between them.  The
algorithm evaluates the quality of the line that separates the two classes by
maximizing the margin width, $m_w = \frac{2}{\lVert w \rVert}$.  The hyperplane
is defined in Equation \ref{eq:plane}, where $\boldsymbol{w}$ is the vector
that is normal to the hyperplane.
\begin{equation}
  \boldsymbol{w \cdot x} + b = 0
  \label{eq:plane}
\end{equation}

Figure \ref{fig:svm} also shows a case of soft margins.  Some problems are not
linearly separable, and thus a penalty term, $\xi_{i}$, is introduced to allow
for some misclassifications.  The algorithm then simultaneously minimizes the
misclassifications while maximizing the margin. The objective function of the
algorithm shows this as well (using quadratic programming) in Equation
\ref{eq:svm}. Here, $C$ is responsible for the margin width/misclassification
tradeoff, and the penalty term is included in the constraint. \cite{scikit,
elements_stats}
\begin{equation}
\begin{split}
  min\ & \frac{1}{2} \lVert w \rVert ^{2} + C \sum_{i} \xi_i \\
  subject\ to:\ \ & y_i (w x_i + b) > 1 - \xi_i
  \label{eq:svm}
\end{split}
\end{equation}

Figure \ref{fig:svr}\footnote{These schematics are based on a tutorial on SVR
by Dr.\@ Saed Sayad at
http://www.saedsayad.com/support\_vector\_machine\_reg.htm} demonstrates how
\gls{SVM} can be altered slightly from classification to nonlinear regression
with \gls{SVR}.  \Gls{SVR} has a similar objective function but instead
\textit{minimizes} the margin, as shown in Figure \ref{fig:svr-a}.  While the
objective function is the same as in Equation \ref{eq:svm}, the constraint is
now set to the opposite inequality sign, shown in Equation \ref{eq:svr}
\begin{equation}
\begin{split}
  min\ & \frac{1}{2} \lVert w \rVert ^{2} + C \sum_{i} \xi_{i} \\
  subject\ to:\ \ & \lvert y_i - (w x_i + b) \rvert \leq \varepsilon + \xi_i
  \label{eq:svr}
\end{split}
\end{equation}

Further, this can be extended to nonlinear analysis via what is called the
\textit{kernel trick}.  A nonlinear kernel function maps the data
into higher order feature space. The algorithm can then find a linear
separation in this space, as shown in Figure \ref{fig:svr-b}.

\begin{figure}[!hp]
  \centering
  \begin{subfigure}[h]{0.8\linewidth}
    \includegraphics[width=\linewidth]{./chapters/litrev/svr-a.png}
    \caption{Demonstration of regression with \acrshort{SVR}}
    \label{fig:svr-a}
  \end{subfigure}
  \begin{subfigure}[h]{0.8\linewidth}
    \includegraphics[width=\linewidth]{./chapters/litrev/svr-b.png}
    \caption{The kernel trick with \acrshort{SVR}}
    \label{fig:svr-b}
  \end{subfigure}
  \caption{Illustrations of \acrshort{SVR} and Nonlinear Analysis}
  \label{fig:svr}
\end{figure}

Chosen for its flexibility, the kernel in this study is the Gaussian radial
basis function, shown in Equation \ref{eq:svr-rbf}. This has two tuneable
parameters, $\gamma$ and $C$. The $\gamma$ controls the width of influence of
individual training instances, which strongly affects the fitting of the model.
Low values correspond to underfitting because the instances have too large of a
radius (low influence) and high values correspond to overfitting because the
instances have a small radius (high influence).
\begin{equation}
\begin{split}
  min\ & \frac{1}{2} \lVert w \rVert ^{2} + C \sum_{i} \xi_{i} \\
  subject\ to:\ \ & \lvert y_i - (w \phi(x_i) + b) \rvert \leq \varepsilon + \xi_i \\
  where:\ & w = \sum_{i} \alpha_i y_i \phi(x_i) \\
  and:\ & K(x_i, x_j) = \phi(x_i) \phi(x_j) = e^{\gamma \lVert x_i - x_j \rVert ^{2}}
  \label{eq:svr-rbf}
\end{split}
\end{equation}

The $C$ parameter also affects the fitting of the model by allowing more or
less support vectors, corresponding to more or less misclassification,
respectively. A lower $C$ smooths the surface of the model by allowing more
misclassifications, whereas a higher $C$ classifies more training examples by
allowing fewer misclassifications. Thus, too low or too high of a C can cause
under- or overfitting, respectively.

Since there is a tradeoff of fitting strength provided by two parameters, it
is common to run the algorithm on a logarithmic grid from $10^{-3}$ to $10^3$
for each parameter. If plotted on a heatmap of accuracies given $\gamma$ and
$C$, there will be a diagonal of ideal combinations that emerges. The element
with the lowest value of each parameter is usually chosen.

\subsubsection{Dimensionality Reduction Techniques}
\label{sec:dimreduc}

In addition to utilizing various algorithm parameters for regularization as
discussed above, dimensionality reduction can improve generalizability by
removing the noise of features that do not affect the regression task. This can
be thought of in the following way: shrinkage techniques reduce the weights of
noisy features, whereas dimensionality reduction removes them completely
\cite{elements_stats}.  Although one could use domain knowledge to manually
reduce the number of features in a data set (e.g., only including certain
nuclide subsets such as actinides), statistical feature reduction may also
prove helpful in this work.  The mathematical treatment of the methods
described below are in Ref.  \cite{elements_stats}.

%Another use for the following techniques is the visualization of one's data
%set, perhaps for data exploration or to show why certain predictions are
%difficult.

\vspace{5mm} \noindent \textbf{Principal Components Analysis} \vspace{5mm}

\Gls{PCA} is considered the most common dimensionality reduction technique.
The \gls{PCA} algorithm learns a linear transformation of a data set
$\boldsymbol{X}$ with which to construct a transformation matrix according to a
user-chosen number of variables, i.e., components.  This matrix is part of the
singular value decomposition of the data matrix $\boldsymbol{X}$.  The
decomposition step is what provides the principal components, which are the
result of maximizing the variance in the original data while minimizing the
squared reconstruction error between the original data and the transformed
data. The mathematical assumptions are that the variables are all Gaussian and
uncorrelated.

Because the principal components are obtained purely statistically with no
model assumptions, they are usually uninterpretable. However, they can still
provide clues with which to obtain new information. In the case of this work,
this could provide insight into new forensics signatures.

\vspace{5mm} \noindent \textbf{Factor Analysis} \vspace{5mm}

Factor analysis is similar to \gls{PCA} in that it also calculates linear
combinations of the data set features using the above-mentioned decomposition
and the mathematical assumptions are the same.  It is different in that the
decomposition is rearranged so that it represents \textit{latent variables}
including random error disturbances rather than principal components.  Latent
variables are constructed by maximizing the correlation/shared variance among
the variables rather than the total variance.  However, different optimizations
can be chosen so that the solutions are parameter-dependent.

Furthermore, the initial model assumptions, while enabling interpretable
results, also increase the dependence of the solutions on the algorithm inputs.
If \gls{PCA} does not perform well with the type of training data in this work,
it is possible that factor analysis will, since there are many nuclides in the
training data set that are in a decay chain together.

\vspace{5mm} \noindent \textbf{Independent Components Analysis} \vspace{5mm}

\Gls{ICA} has characteristics of both factor analysis and \gls{PCA}, but with
the goal of finding independent measurements from multiple `sources'.  It uses
the same form of decomposition as factor analysis but without the inclusion of
random error. The mathematical assumptions are a bit different: the variables
are statistically independent and non-Gaussian.  This allows the algorithm to
minimize higher-order statistics of the data set (\gls{PCA} and factor analysis
only minimize the first two orders).

Since the independent components are useful in signal processing of multiple
signals, this technique could prove useful for reprocessed nuclear materials
where multiple source streams converge.


\appendix
\section{Proof of equation \ref{eq:energy}}\label{sec:proofenergy}
The third equality \cref{eq:energy-compact} is valid since
\begin{equation*}
    trace(\bar{\mathbf{W}}) = \mathbf{S}^T diag^{-1}(\bar{\mathbf{W}})\mathbf{S}
\end{equation*}
The last equality, \cref{eq:energy}, comes from the fact that the trace of $\bar{\mathbf{W}}$ (\cref{eq:weights}) is the sum of the diagonal elements, and the diagonal elements of $\bar{\mathbf{W}}$ are given by
\begin{equation}
    \bar{w}_{ii} = \frac{1}{M} \sum_{m=1}^M (v_i^{(m)})^2 \equiv 1
\end{equation}
The trace of $\bar{\mathbf{W}}$ is therefore given by
\begin{equation}
    trace(\bar{\mathbf{W}}) = \sum_{n=1}^N 1 = N
\end{equation}

\section{Matlab Code} \label{sec:matlab_code}
\lstinputlisting[breaklines=true,language=MATLAB]{../../project2.m}
\lstinputlisting[breaklines=true,language=MATLAB]{../../runSim.m}


\section{Computation}
\label{sec:computation}

\subsection{Stationary equilibrium}

The solution algorithm used to solve for stationary equilibria consists of the following steps:
\begin{enumerate}
    \item Guess initial levels of aggregate capital and aggregate labor
    \item derive factor prices from firm FOCs
    \item calculate pension benefits from aggregate labor supply and the government budget constraint
    \item Solve for household value functions and policy functions by backward induction
        \begin{enumerate}
            \item Start with maximum age; value function equals flow utility from consuming all available resources
            \item Iterate backwards over ages and solve using continuation values from the next age
                \begin{enumerate}
                    \item Define meshgrids for current assets, next period assets, current human capital and next period human capital
                    \item Calculate consumption, human capital effort and flow utility on meshgrid
                    \item Extract continuation values by next period assets and next period human capital from value functions of time period $t+1$
                    \item Calculate sum of flow utility and expected discounted continuation value on meshgrids
                    \item Find maximal value over assets and human capital next period
                \end{enumerate}
            \item Store value function and policy for all ages
        \end{enumerate}
    \item Simulate cross-sectional distribution of households by asset holdings, human capital and age
        \begin{enumerate}
            \item Initiate with initial mass of household at initial levels of assets and human capital
            \item Iterate forward over policy functions to obtain cross-sectional distribution
        \end{enumerate}
    \item Aggregate over households to obtain aggregate variables
    \item Verify initial guess; If tolerance for deviation is exceeded, update guess and repeat.
\end{enumerate}

\subsection{Transitional dynamics}

The solution algorithm used to solve for transitional dynamics consists of the following steps:
\begin{enumerate}
    \item Guess initial path for aggregate capital and aggregate labor
    \item Derive factor prices from firm FOCs
    \item Calculate pension benefits from aggregate labor supply and the government budget constraint
    \item Solve for household value functions and policy functions by backward induction for all age and all time periods
        \begin{enumerate}
            \item Start with the final period $T$ and use as continuation values for all ages the value functions of the final stationary equilibrium
            \item Iterate backwards through time periods and derive value functions and optimal policies
                \begin{enumerate}
                    \item Define meshgrids for current assets, next period assets, current human capital and next period human capital
                    \item Calculate consumption, human capital effort and flow utility on meshgrid
                    \item Extract continuation values by next period assets and next period human capital from value functions of time period $t+1$
                    \item Calculate sum of flow utility and expected discounted continuation value on meshgrids
                    \item Find maximal value over assets and human capital next period
                \end{enumerate}
            \item Store policies and value functions for all ages and all time periods
        \end{enumerate}
    \item Simulate cross-sectional distribution of households by asset holdings, human capital and age over time
        \begin{enumerate}
            \item Start with cross-sectional distribution from initial stationary equilibrium
            \item Iterate forward over policy functions of the respective time period to obtain cross-sectional distribution of the following time period
        \end{enumerate}
    \item Aggregate over households to obtain paths aggregate variables
    \item Verify initial guess; If tolerance for deviation is exceeded, update guess and repeat.
\end{enumerate}

\subsection{parametrization}

The following parameters are used for computation:
\begin{itemize}
    \item Asset holdings are discretized on a linear grid with bounds $[0.001, 30.0]$ with 60 grid points
    \item Human capital is discretized on a logarithmic grid with bounds $[1.0, 5.0]$ with 20 grid points
    \item The tolerance level for deviations in aggregate capital and aggregate labor are set to $10^-5$ for computation of stationary equilibria and to $10^-3$ for the computation of transitional dynamics.
\end{itemize}